and his wife Anne Fraser, who was a sister to Brigadier-General Simon Fraser of Balnain. Both his parents were from old Highland families. His mother died while he was a child, and his father was frequently abroad, mainly due to the Seven Years War - being wounded in 1761 at the Battle of Villinghausen.

James was raised by his grandmother, and schooled at Fortrose Seminary academy. At age thirteen