It's a good way to look at it, since not only are most Muslims not terrorists, the vast majority are not. How many, for example, of the thousands of American Muslims have actually committed terrorist acts. I can only think of three and one attempt at it. If it was rational to believe that Muslims were violent reactionaries that number would be considerably higher. Thus, Islamophobia is irrational: having a reason is not the same as having a rational reason.
Which gets at a flip side: why does every group have "the crazy ones"? If a belief system is allegedly a peaceful one, how can it produce unpeaceful results?
A lot of these kumba-yah interpretations are more modern interpretations. You don't have to go far before you find examples of "kill the nonbelievers" in those ancient holy texts that these religions view as inerrant and the Word of God. And that's a problem. So long as, say, Christians believe that God at one point commanded the execution of adulterers and that was morally good, there will be excuses for the crazy ones to act out. This seems to me like a contrary belief.