If you are making a kernel-image (debian style) you need source.
If you are making "aftermarket" modules for your kernel, rather than D/L the source and configure it and re-compile it. All you would have to do is D/L the headers (the build dir) for your exact kernel (which is provided as a packageby Debian)
Most every package that needs to compile against the current kernel *FIND* this header dir. Makes the whole thing easy.
The only time you need kernel source (in debian) is a custom kernel. You will never need the configured source ever, unless you are doing a custom kernel.
If you are compiling a kernel, you do not need the headers... you make the headers with the kernel source.