The Castle is almost certainly on the site of the original Bishop’s Palace of St. Gilbert, who founded the Cathedral in the early 13th Century. However, it is not known when the oldest surviving part of the present building was put up. In 1557, the Palace was given to the Earl of Sutherland by his brother in law, Bishop Robert Stewart. It is clear that the building was erected before that date and may well be of the late 15th Century.

The earliest surviving documentary evidence of the Palace is the Charter of 1557, by which Bishop Robert appointed the Earl of Sutherland and his heirs to be hereditary Constables for Dornoch “situated in the Irish (Gaelic speaking) Country among the wild, unbridled, untamed and savage Scots”. The appointment was made by the Bishop who was alarmed by the progress of the Lutheran doctrines and wished to safeguard the property of the Church by putting it temporarily in the care of his relations. He cannot have visualised that the Church would never own its property again.