Although the relationship does not appear to have been close, some have suggested that Gaspar de Guzmán, Count-Duke of Olivares, his key minister, later deliberately tried to keep the two apart to maintain his influence, encouraging Philip to take mistresses instead, by whom he is known to have had at least 30 progeny.

Philip had ten children with Elisabeth, with only one being a son, Balthasar Charles,