What if there was simply a way to undo upvotes, or a person could not be downvoted past, say, 30 reputation, or maybe could not be downvoted if too new or more than twice a day. Getting rid of it seems a little extreme, as the system does have its uses, but the problems you mentioned about it can be fixed.