I saw Larry mention that Hacker News does that. I figured hey, why not, I can add that. That should encourage those arguing to get a one up by making actual arguments and not trying to use votes. Not that any arguments happen here anyway.
Cool. I tried to downvote you. I wasn't 100% sure how it would handle text posts. The text area of a text post is actually stored as a comment. So you can downvote people who reply to a link post but not to a text post (if it is a first level comment and you are OP).