When the office of postmaster-general was abolished in 1797, his resident was made Count of Penafiel in compensation, thereby resulting in the new designation of the palace.

In 1859, the kingdom's last official postmaster-general to the Kingdom, Manuel José da Maternidade da Mata de Sousa Coutinho, 1st Count of Penafiel (n.1782) died in the residence. The palace passed into the hands of his only daughter, the 2nd Countess and 1st Marquess of Penafiel, D. Maria da Assunção da Mata de Sousa Coutinho (1827-1892). There was a campaign