That's the really scary thing - this von Neumann argument is as far as I can tell, unassailable. It doesn't even depend on something like a propagation speed and so is independent of the limitations of relativity.