However, William quickly arranged for the building of Rougemont Castle to strengthen Norman control over the area. Properties owned by Saxon landlords were transferred into Norman hands and, on the death of Bishop Leofric in 1072, the Norman Osbern FitzOsbern was appointed his successor.

In 1136, early in the Anarchy, Rougemont Castle was held against King Stephen