Older versions of Debian used runlevel 5 for managed GUI log-in. In /etc/inittab, change the "2" in "id:2:initdefault:" to a 3. That'll give the normal, non-GUI boot.
Woody no longer uses runlevel 5 to run XDM et al. XDM is now started through one of the S... symlinks in the /etc/init.d/rc?.d directories. Runlevel 2 now appears to be the default, so rename /etc/rc2.d/S99xdm (e.g. to XS99xdm) until you get X set up right.
As I skipped Potato, I'm not sure which of the two methods it uses, but it shouldn't be hard to find out.
You can bypass XDM by using Alt-F1 through Alt-F6 (that is, if the malconfigured X left you a functioning keyboard).