Yeah, I've done that a few times as well. I guess it could go either way. I think that's why only displaying as grey after 3+ downvotes would keep people only voting based on their own thoughts and not on how other people have acted on that comment.