In 1523 the castle was already described as not useful for defence purposes but with thin walls. The repair by the Italian architect Licio de Spari 1538 - 41 for the building owner, the Augsburg patrician Paumgarten, was nearly a rebuilding. The building was the main residence of the government of Schwangau. After repeated change of ownership, the Crown Prince Maximilian of Bavaria was addicted to the charm of the place and bought the meanwhile to ruin decayed castle. The restoration started in 1833. The remarkable, on one side rounded ground-plan was retained and through the tower on the south-west corner resulted a regular building front. King Maximilian II has used the castle as summer residence. As expected will be found the abundance of decorations in the interior, the innumerable paintings and the stories they narrate from the honourable persons of the German legend and history till to the heroic deeds of the then living generations of the family Wittelsbach.