Koga is an annoying gym leader, but i didn't have much trouble with him if i'm honest. I found Lance to be a bit of a pain because i never use anything on my team that can counter dragon type pokemon so he is the most annoying boss fight for me.
i find lance difficult but not really annoying because i know the reason i can't beat him is because his pokemon are just flat out better than mine. however koga's pokemon are bad but he spams acid armor and minimize so thats why i think he is annoying
When I first played Black 2, I didn't have any electric-type Pokemon - For me the annoying battle was a gym battle with Skyla. All her bird Pokemon keep using Roost, kept dragging the battle too long, and when I finally got the health low enough, she had to go the Hyper Potion, it got me frustrated and mad at the same time.