23 is telegrapher's shorthand for "go away!" - thus the phrase means "let's blow this Kool-Aid stand!".
Another theory has it that young men would gather near the Flatiron Bldg. in Manhattan, on 23rd St., and howl "23 Skidoo!" when a cutie would pass by and have her skirt elevated by a passing subway train.