Does downvoting a comment cause the voter to lose karma, or is it in essence free? StackOverflow does this (you lose a point each time you vote something down), and it is actually a meaningful deterrent: maybe it would help here.
The voter loses karma. One downvote is one karma. One upvote is one karma. (StackOverflow is 1 upvote for 10 karma for answers, 2 karma for questions; and 1 downvote for 1 karma everywhere, I think.)