Protection works like armor, so it has dimishing returns in that the more you have the less you gain from increased protection. So while I'm sure there's a cap somewhere you'll hit the point where it isn't worth it to increase it anymore long before you hit any sort of hard cap.
I still have my culture armor with +prot gems, that and the right rings, cloak and necklace with my AAs which works out to a little better than 40% magical damage mitigation. I think my total protection across the board is somewhere around 1200 without running Mental Barrier but I don't remember, it may be higher or lower, sorry
Unholy damage is used more often than holy damage in both PvP and PvE.