Notice February and March?
See anything unusual?
I specifically #1 and #2 on [link|http://www.iwethey.org/zlogs/usage_200603.html#TOPSITES|March 2006] Top "hits" per Host, then also look Just below it at the Top 5 By KBytes.
If you'll also note that the Bestestest Referrers have quite a few pR0n sites there. Reason is, referral spam and using Z somehow (I am not showing) as a proxy to generate Site hits for those places.
Now, onto [link|http://www.iwethey.org/zlogs/usage_200602.html|February 2006] Same thing as March. [link|http://www.iwethey.org/zlogs/usage_200602.html#TOPSITES|Top "Hits" per host], the top 2 are big hitters, though #1 far exceeds double the next one. The top #1 is also related to the top 2 for March. I have thus blocked the entire subnet, based on the hostile phone call I wound up having (I called the Block owner) and basically he told me to Piss-off. So, I guess even if it was legitimate traffic, it is blocked.
Then also look on the Top by KBytes agian, search engines are really getting to like us.
Now Become.com I believe has been started in a partnership with a self-exiled person from IWETHEY. For this reason, I don't believe any harm or bother was meant. Therefore Scott and I have placed more strict robots.txt in the proper places for Z and for twiki.
This so far has kept the amount of referral-proxy down quite a bit since 8:30AM. Then I went after why the DB was goin crazy on us. That is when we found that the EvUuL Threads oF Do0m were hammering the DB.
I gave the DB more breathing room, but since the Postgresql DB connection for Zope driver doesn't do multi set queries very well, it may well do them all seperately (I think that is what was mentioned earlier today), The changes only speed it up a bit. (currently 30+ seconds to render the "7|-|R34D 0|= D347|-|" #249242).
Since we got done making threads smaller (hopefully), tweaking and blocking things, Load Averages have steadily decreased and the DB is no longer gasping for air. and basically I hope we will see a return to a respectable performance level.