According to Ramsay, Lafitte, his younger brother, Jean, and their widowed mother sailed from Saint-Domingue to New Orleans, Louisiana (New Spain) in the 1780s. Approximately 1784, his mother married Pedro Aubry, a New Orleans merchant. Jean stayed with his mother while Pierre was raised by extended family elsewhere in Louisiana.

The biographer William C. Davis reports a different childhood. According to Davis's