Bramber was one of the poorest parts of Sussex, and while it remained a centre of administration, the Mowbrays did not live there; by the 1550s, it was recorded as 'the late castle', used for grazing.

During the First English Civil War, Bramber was held by a Parliamentary garrison, under James Temple and Simon Lentner. In December 1643, a skirmish took place nearby,