The couple had three children before Mary died in 1732; only one, a son named Andrew, survived to adulthood. After Mary's death Oliver traveled to England, returning to Massachusetts in 1734. On December 19, 1734, he married Mary Sanford of Newport, Rhode Island, the sister-in-law of Thomas Hutchinson. The couple had fourteen children.

Politics and the Stamp Act