farm and seek an education. With the help of his younger brother James and their father's plow horse, Brown drove a yoke of oxen on a 125-mile trek to an academy near Anderson, South Carolina. There Brown traded the oxen for eight months' board and lodging.

In 1844, Brown moved to Canton, Georgia, where he served as headmaster of the town's academy. During this time, Brown boarded in the home of local businessman and Baptist minister John W. Lewis.