I didn't make it!
I definitely agree with the #1 choice.
The water temple in Ocarina always annoyed the hell out of me. I'm not sure if that counts as a true "castle" per-se, but it's nonetheless bothersome. Probably doesn't help I was never quite proficient at that game.