Jorgensen was the second child of carpenter and contractor George William Jorgensen and his wife, Florence Davis Hansen. She was raised in the Belmont neighborhood of the Bronx, New York City, and baptized a Lutheran. She described herself as a "frail, blond, introverted little boy who ran from fistfights and rough-and-tumble games".

Jorgensen graduated from Christopher Columbus High School