violently abusive. Ruffin's mother died from complications of childbirth 10 months after he was born; and his father later married Earline, a schoolteacher, in 1942.

As a young child, Ruffin, along with his other siblings (older brothers Quincy and Jimmy, and sister Reada Mae) traveled with their father and their stepmother as a family gospel group, opening shows for Mahalia Jackson