The castle was part of the medieval town fortification. It contains all the styles from Romanticism to Gothic and Renaissance. At the northern entrance, a memorial sign remembers the work of the famous Nuremberg mastersinger Hans Sachs in Wels.
Presumably, Polheim Castle was established under Albero of Polheim around the middle of the 13th century, by assembling several houses that the Polheimer had acquired over time. The building formed the northern part of the medieval city wall and was therefore part of the city fortification. The entrance to the castle’s courtyard leading to today's Ringstrasse was once secured by a drawbridge and a moat. The medieval part of the castle, with its gothic tower, window frames and a gothic coat of arms, is directly connected to the renaissance part of the castle. The beautiful architecture of its windows is especially remarkable at this part of the castle. In the courtyard of the building, one can still see the former gothic chapel with the late gothic windows.
The name "Freiung" reminds us that the town had no rights in the castle and in the "Freyung". This means that the Polheim Castle was an aristocracy of its own, outside the jurisdiction of the town. It was exempted from the city law and its consequences. The Polheimer had its own jurisdiction! A panel next to the portal on the north side of the castle recalls that the well-known Nuremberg mastersinger Hans Sachs temporarily lived there in 1513, during the time when he still was a cobbler apprentice in Wels. At that time, he was already a mastersinger of the Welser Master's School. It is stated that he wrote his first poem in Pollheim Casle.