This time, the castle's structure was substantially damaged. It remained derelict until Hugh de Puiset, Bishop of Durham from 1153 to 1195, had the castle rebuilt. The work was probably directed by Richard of Wolviston, who was the bishop's architect.

In 1174, Hugh de Puiset supported the rebels in a revolt against Henry II, during which the Scottish king, William the Lion, invaded Northumberland. The rebels