While visiting the White House, Holt proclaimed that he was "all the way with LBJ", a remark which was poorly received at home.

In December 1967, Holt disappeared while swimming in rough conditions at Cheviot Beach, Victoria. He was presumed dead, although his body was never recovered; his disappearance spawned a number