The castle was built in the 15th century with most probably a quadrangular ground plan. It certainly had two towers on the eastern wall and it is possible that two similar towers defended the western wall as well, whilst the internal courtyard included buildings attached to the walls. Holy Trinity chapel was erected beside the buildings attached to the southern wall and was decorated with the works of art by an unknown artist in 1471.
Žminj was an important fortress, primarily because as an Austrian fortress it was located in front of the Venetian St. Vinčenat. In 1612, whilst preparing for battle with Venice, the Austrian archduke had around 200 foot soldiers in Žminj. During the war from 1615 to 1618, Žminj’s army often attacked the Venetians destroying their crops, especially the hay which they transported from the Port of Fažana to St. Vinčenat and due to the lack of hay the Venetian horsemen were almost powerless. But, as often happens, in January 1617 a traitor secretly led the Venetians into the castle. For this service he received a lifelong salary of six golden coins per month, but fearing for his life he moved to Crete.