I use it on my server.

There's nothing wrong with a server distro having great hardware detection and a decent installer.

Remember, Ubuntu *is* Debian. You can do what you wish with it and X is not a required component of the installation.