Its a race, carnival, camping trip, unique experience all in one event. Spots like the Mulsanne Curve (full speed -> slow to a crawl -> 90 degree turn -> full speed) are a blast. Glowing brake rotors at night. Noise! I was lucky enough to go in 1998, 1999, and 2000.
Watching races on TV doesn't hold much interest for me either, but I'd recommend attending a race at least once in a lifetime just for the experience. Try something like the American LeMans series.
[link|http://www.lemans.org/course/acocourse/acocourse/main/une_unelemansseriesgb.shtml|Official LeMans Site]