William Wilberforce was a frequent house guest of the first Lady Barham, who is said to have inspired and supported him in his fight against slavery. He loved the place and once wrote that 'for the charm of softness and elegance I never beheld a superior to Barham Court'.

The house was very badly damaged in a fire in 1932. It was subsequently refurbished by Holloway Brothers.