So based on the discussion in this thread, I've installed OpenVAS and used it to scan my servers. I've fixed the major issues it found (turned off a bunch of "weak" SSL ciphers that are enabled by default in Apache), the only remaining issue is:

Summary
The remote host implements TCP timestamps and therefore allows to compute
the uptime.

I've looked into this, and the fix for this is to run some sysctl command, but it seems that doing so might slow down the server when it's under load.

But seriously: why do I care if my server's uptime can be computed? What's the security risk here?