The kernel packages that the likes of Red Hat and SuSE provide have a degree of QA and testing that you can't provide.
You have some assurance that a particular vendor release of a kernel won't have horrendous bugs in it. Frexample, you'll never see a 2.4.11 kernel from Red Hat - but if you were quick off the mark, you could have downloaded and installed that before Linus got a chance to tag it "2.4.11-DONTUSE".
I basically take the attitude that rolling your own kernel is only a valid exercise if the vendor-provided one doesn't do what I want.
The performance advantage these days is negligible as most drivers are now provided as modules anyway.