The Castle was begun in 1435 by Sir William ap Thomas on the site of a former Roman motte with the Great Tower. His son William Herbert continued the work with further additions including the gatehouse with more changes made well into the mid-16th century. In 1646 the castle was under siege for eleven weeks before falling to Cromwell's forces. The castle (in particular the Great Tower) were then partly destroyed by his demolition team. It's still possible to climb the tower today though for great views over the castle and the surrounding hills.