“The right to challenge a man's judgment carries with it no automatic right to question his honor.”