Just because you call someone a racist does not mean that they are one.
I believe the ban is in place in order to reduce name-calling. People who often have either poor arguments or lack skill in articulating an argument will often resort to simply calling someone a racist in an effort to quell any dissent. In addition to being intellectually dishonest, it's dumb.
Anyone who cannot make their point without calling someone else a racist has a problem. No one else will.
Same thing when someone calls another poster a liar without proving the lie.