"We were considered some kind of 'low' family", Smith once said, "despite my mother's feeling that she was [an incarnation of] the Czarina of Russia". Friends recall that in high school Smith carried around a camera and in his high school yearbook said that he wanted to compose symphonic music.

Physically, Smith was undersized and had a curvature of the spine which kept him from being drafted (a circumstance that later would disqualify him from benefitting from the G.I. Bill). During World War II,