I don't believe it's possible to get around the throbber problem. I'm working on a project that does something very similar (it may be exact...but I can't give you too many details...sorry).
We decided to use an applet. One of the reasons for this is the possibility of the connection being dropped (for example, if the user *does* push stop). We haven't really seen too many problems on the client side (though we don't send data back to the server after the initial GET). The other problem is that the UI that displays the data is somewhat complex and easier to render in java than in HTML.
I'm the one building the backend. It would certainly work with javascript/DOM (and the multiple frame idea). In fact, we're looking into that, as we may have multiple "applications" on the same web page that need to be updated, and don't want each one to have its own connection.
The issue we're running into is the backend...from a network perspective. You say you're handling several thousand clients? How many servers? What's the server/client ratio? Of course, I don't expect you to answer those, as I expect you can talk about your project about as much as I can about mine (our companies may even be competing...I don't know).
Another possibility: I don't understand the requirement that the user *sit* at a web page, watching data be updated. To me...that's not the web. That's a client/server app. So, I suggested they use Java Web Start. It's not much different from an applet, but there are some things that it adds that make it more interesting to do it that way. Not sure if that's a possibility for you.
Dan