I thought of it more in terms of freedom of speech coming at the cost of having to tolerate speech one finds offensive.

The complaints against Toles are based on quite deliberate misinterpretation of his message.