Refactoring without pre-building unit tests is pretty pointless.
Exactly. Refactoring without unit tests is dangerous and I would agree with him in that case.
But advocates of refactoring generally also advocate unit tests. In fact, I first learned about unit tests from reading Fowler's refactoring book. I've been doing most of my coding test-first for the past 3 years and believe it to have markedly improved the quality of my code.
I've become a strong believer in unit tests, and especially letting them drive the design and implementation.