"Jerry" to those close to him, Robbins was given the middle name Wilson reflecting his parents' enthusiasm for the then-president, Woodrow Wilson.

In the early 1920s, the Rabinowitz family moved to Weehawken, New Jersey. His father and uncle opened the Comfort Corset Company in nearby Union City. He graduated in 1935 from Woodrow Wilson High School (since renamed as Weehawken High School).