Thus, in one meaningful sense, we are all heretics: we all believe things that are not true. Or, I should say, we all have the choice between heresy and agnosticism; those who choose heresy have a further choice between heresies. The second part is important, because there are better and worse heresies. But the first part is also important to consider, too. Most of us probably cannot sustain agnosticism, or at least cannot do it in enough areas for very long. (Indeed, no one can sustain agnosticism about moral or political beliefs without giving up on any moral or political action.) And those of us who do not choose agnosticism choose heresy. Many people do not know that they are choosing heresy, since they believe they are choosing truth. But what they believe is incomplete, and does not do justice to the whole of the world, and so it is heresy that they choose.
Heresy, as generally used, does not refer so much to the fact that all beliefs contain error but, rather, to beliefs that contradict those endorsed by the community which is using the word heresy. Or, to put it more pointedly, a heresy is a belief which gives a community sufficient warrant to exclude anyone who happens to hold it. Of course the community frames the definition in terms of truth and error, but practically speaking heresy in this sense—the sense of excommunication—is not about truth but is instead about boundary-marking.
I find the heretic’s faithfulness and commitment to be quite touching, rather than (just) their status as underdog or outcast. That commitment might strike the orthodox as tremendously annoying, but, hey, lots of people are annoyed when they try to severe a relationship with someone and that someone refuses to let the relationship end. Of course, many heretics have acknowledged have had to give up on the community, due to some threat or another. Eventually you cannot fulfill your commitment to the community except by stepping out of the door (or fleeing the country) and trying to build a new community in the name of the one from which you were driven. So it is in this sense that I am charmed by heretics. Whether or not the heretic is in error is not at all relevant to the description, and I think over history heretics have probably broken even on whether or not their beliefs were better or worse than those of the community with which they argued.
Note: I am not signing off on everything in The Beginning of Infinity—of what I've read so far, I find Deutsch’s argument mixed, both in terms of his overall position and his presentation of it—but I do wholeheartedly agree with his fallibilism (though I disagree with his characterization of justificationism, which I don’t think is actually at odds with fallibilism if construed in a particular legitimate way).
Note: I am aware of Kathleen Mulhern’s “What Makes a Heretic?” (link), and this post was written with her post—and the whole series from which it comes—in the back of my mind.
(EDIT: Rachel Held Evans has an interview about heresy on her blog which discusses the overuse of the word. The distinctions between kinds of deviation from orthodoxy is probably more helpful than my discussion here, which is probably worth my thought.)