Everybody "in the Right" ignores him as a nutcase.

It's the same pattern with every single other defection - they just pretend that he doesn't exist. As long as these defections leak slowly, instead of a massive burst, nobody notices.