On Usenet there's a natural time limit in being able to read posts because they disappear from the server after some (server dependent) time period. That time period can be group dependent (e.g. many alt.binaries.* groups expire every day on some servers).

How about the following:
1) Server has an admin-adjustable overall post "lifetime" of say 14 days. (A reasonably long period of time, but one that would not be used by a registered reader in normal circumstances.)
2) User can set a personal overall post lifetime of up to 14 days.
2a) (Optional enhancement - post lifetime can be set in a per-forum basis.)
3) A post is shown as unread if it is younger than 14 days and if user has not marked it read.

You'd still be able to mark individual posts read, but that should keep the amount of information that you need to keep track before showing the unread messages under control, I think. (I'm no expert on databases, though.) I don't think this would impact your idea of cross-posting a message to multiple forums either - you're mainly keeping track of the Post-IDs and not the Forums in marking things read.

This also would allow us to keep a particular post "Sticky" within the overall lifetime. (You'd have to have a variety of choices to mark things read though: 1) Mark All Read, 2) Mark This Post Read, 3) Mark This Thread Read, 4) Mark This Post Sticky, 5) Mark This Thread Sticky, 6) Ignore This Thread, 7) Ignore This Subject, 8) Ignore This Poster, etc. See a good Usenet news reader like [link|http://slrn.sourceforge.net/|slrn] or [link|http://www.vex.net/yarn/|yarn] for ideas. It would be nice to have a "scoring" system like slrn or yarn as well, if at all possible.)

The nice thing about this is that it should be easily tunable if the load gets too heavy - just reduce the overall post lifetime. If you have 10M users, it might get hairy, but presumably you'd have dozens of server farms in that case. :-)

The older posts would still be in the database (and thus would show up in a search), but wouldn't ordinarily be visible.

Note that this would be a change for us in very long threads - rather than having 90+ messages in a thread over a month, one would only see, at most, the newest 2 weeks of messages. I think that's fine as I never read old threads from the beginning unless I'm doing a Search and all of the posts would still be there in that case.

(If you could force the database to keep information in a reverse chronological order when it's "inserted", then you wouldn't need to search more than a fraction of the database when making the initial pass to find the messages younger than 14 days. Or perhaps every midnight you could do a quick scan to find the Message-ID of the 14-day old cut-off post in each forum and keep that number as the display limit for that day (and not have to determine the time limit again during the day). I would hope that tricks like these, and others, would lessen the load on the database under the scheme as outlined above.)

For unregistered readers, you could either do as you do now (show everything in page after page), or you could show things back to the 14 day time limit to lessen the load some. You wouldn't have to check individual sparse lists of marked-read posts.

Just a few thought. HTH! Thanks for working on this problem. I'm looking forward to the new version. :-)

Cheers,
Scott.
(Who doesn't like to reinvent the wheel if he doesn't have to.)