Think about *how* your IDE makes it easy to refactor: what is involved in copying a fragment of code and then modifying it slightly for it's new location?
In an IDE such as Microsoft's Visual Studio (note that the last version I used was for MSVC++ v2), this involves lots of action on the editing keys (cursor keys, delete and insert) to highlight relevant text and modify as appropriate, accompanied with a quantity of mouse-work also for selecting or moving text and for actvitating toolbar buttons.
Vi has many tools for changing text in efficient ways that be combined in hundreds of variations. The fact that they activated by the typewriter keys helps because your hands can stay there instead of moving off to hit a cursor key (this is the same for Emacs, BTW). For example, non-vi-ers think that having 5 basic ways to switch into a text-entry mode is superfluous. It's not. They all setup your insertion differently. The extra efficiency is slight in typing o to enter a new line below the current one over pressing End then Enter, but it adds up over time.
The difference in approach is one of a complete, but predominantly not-overlapping, set of basic tools for modifying text (typified in part by more keystrokes) versus a much larger toolset that has lots of overlapping functionality (typified in part by less keystrokes). It's difficult to explain.
Wade.