Šumber medieval castle with its lower ward was built at the place of the former pre-historical hillfort in the Old town small village. In 1260, the castle came into the possession of Schönberg - the Austrian noble family, after whom Šumber was named. Since 1367, it has been a part of the Pazin principality and has become a border fortress between the Austrian and Venetian part of Istria. Due to the extinction of the Schönberg family, Šumber became part of the Kožljak settlement at the end of the 14th century.
Kaštel Šumber has a polygonal floor plan surrounded by walls and is defended by two circular semi-towers in the southeastern and southwestern part. The walls close the courtyard, and the main entrance to the fort is on the eastern part of the wall. On the inside part of the southern bastion are the remains of a long and narrow one-story palace, and there is one other smaller building which is located in the northeast corner of the castle.