You can get there, starting from Galway, on the N59, through Oughterard, Maam Cross, then Cong. But an even better way is to catch the ferry from Oughterard, (Corrib Cruises) which leave every day at noon during the summer. More info on Corrib Cruises here
Strolling through the grounds of the castle is very pleasant. There are little paths and lanes along the shores of the Lough Corrib with plenty of benches to sit and rest or even to have a picnic.