Post #259,023
6/15/06 8:12:59 AM
|
Update on Z's response times? Megapath dumping packets?
I'm still having troubles with Z being very pokey (taking ~ 15 s to deliver a page and often timing out). The most recent [link|http://www.gregfolkert.net/stats/uptime.html|uptime] this morning says: top - 07:57:02 up 273 days, 9:18, 1 user, load average: 0.27, 0.33, 0.35
Tasks: 333 total, 3 running, 328 sleeping, 0 stopped, 2 zombie
Cpu(s): 8.6% us, 7.6% sy, 1.4% ni, 81.8% id, 0.5% wa, 0.0% hi, 0.0% si
Mem: 2076164k total, 2067180k used, 8984k free, 111940k buffers
Swap: 3903784k total, 16088k used, 3887696k free, 968120k cached All that looks reasonable to me, and the RAID seems fine. I'm using [link|http://fasterfox.mozdev.org/|Fasterfox] (with the default settings). Since there are only 2 active users now, I wouldn't think that it would be causing problems. The [link|http://www.internettrafficreport.com/namerica.htm|North American Internet Traffic Report] looks reasonable to me too. A tracert to www.gregfolkert.net is timing out for me: C:\\Utils>tracert www.gregfolkert.net\n\nTracing route to knight.gregfolkert.net [66.80.246.91]\nover a maximum of 30 hops:\n\n 1 4 ms 1 ms 1 ms 192.168.0.1\n 2 13 ms 10 ms 14 ms [x.y.z.t]\n 3 10 ms 10 ms 8 ms [a.b.c.d]\n 4 10 ms 9 ms 12 ms [e.f.g.h]\n 5 10 ms 12 ms 11 ms [i.j.k.l]\n 6 25 ms 17 ms 11 ms [m.n.o.p]\n 7 14 ms 11 ms 12 ms [q.r.s.t]\n 8 18 ms 16 ms 17 ms ge-5-3-0.mpr2.iad10.us.above.net [64.125.13.57]\n 9 18 ms 16 ms 15 ms so-4-0-0.mpr2.iad2.us.above.net [64.125.30.122]\n 10 21 ms 116 ms 16 ms so-4-0-0.mpr1.iad1.us.above.net [64.125.28.213]\n 11 19 ms 18 ms 23 ms ge-0-3-0.core1.iad.megapath.net [64.124.229.37]\n 12 18 ms 15 ms 16 ms fe1-7.edge1.iad.megapath.net [66.80.129.30]\n 13 380 ms 479 ms * ip-207-145-38-182.iad.megapath.net [207.145.38.182]\n 14 * * * Request timed out.\n 15 * * * Request timed out.\n 16 * * * Request timed out.\n 17 * * * Request timed out.\n 18 * * * Request timed out.\n 19 * * * Request timed out.\n 20 * * * Request timed out.\n 21 * * * Request timed out.\n 22 * * * Request timed out.\n 23 * * * Request timed out.\n 24 * * * Request timed out.\n 25 * * * Request timed out.\n 26 * * * Request timed out.\n 27 * * * Request timed out.\n 28 * * * Request timed out.\n 29 * * * Request timed out.\n 30 * * * Request timed out.\n\nTrace complete. Can you give us an update on what's causing the timeouts? Is megapath dumping packets? Is there anything you can do about it? Thanks a bunch. Cheers, Scott.
|
Post #259,059
6/15/06 3:12:27 PM
|
1,546,001 Click Fraud Proxy requests since Jun 11 7:36AM
That is what was happening.
Now all they get is 404 pages and 302 pages.
As of right now, everything for them is erroring out.
So it is just ammtter of time before things cool down, that is until the next one happens.
-- [link|mailto:greg@gregfolkert.net|greg], [link|http://www.iwethey.org/ed_curry|REMEMBER ED CURRY!] @ iwetheyFreedom is not FREE. Yeah, but 10s of Trillions of US Dollars? SELECT * FROM scog WHERE ethics > 0;
0 rows returned.
|
Post #259,061
6/15/06 3:19:31 PM
|
"Click Fraud Proxy requests"
Please explain this to me.
|
Post #259,063
6/15/06 3:54:20 PM
|
Okay.
You know what PPC (Pay Per Click) advertising is.
Now, think about some one having hundreds of Pharmacy or pr0n sites... with TONS-o-Banner ads.
Now, think about some one having a pretty nice pipe that happens to have a misconfigured gloabl variable for an Apache server. that being global "ProxyRequests On" with no deny or anything keeping it from helping out.
Now imagine, using a zombie network of hundres of thousands machines to use that misconfigured Apache server as an http proxy. Along with the 30K or so other misconfugred other servers out there.
Now, imagine them using that setup to "randomly click" on those banner-ads or google-ads using a 30Kx300K matrix
Nearly impossible to see the patterns, nearly impossible to see the directions... etc.
Get paid HUGE money. We are talking about lotsa money.
Knight was a single of many proxy machines being used.
14,000 unique IP Addresses. lotsa different websites as targets.
I have to say, once I fixed the issue, the rate increase quite a bit. But then the requests atarted to look less complex and then the all changed (within a few seconds) to probing to check if it was still working. Then they started trying different methods to discover if I just changed something or fixed the problem.
Here is a site many tried to get after I fixed it: [link|http://grem-too.com/cspamitvsax/proxy.php|http://grem-too.com/...mitvsax/proxy.php]
Neato huh?
-- [link|mailto:greg@gregfolkert.net|greg], [link|http://www.iwethey.org/ed_curry|REMEMBER ED CURRY!] @ iwetheyFreedom is not FREE. Yeah, but 10s of Trillions of US Dollars? SELECT * FROM scog WHERE ethics > 0;
0 rows returned.
|
Post #259,064
6/15/06 4:11:49 PM
|
Sounds like you saw them working it in real time
So this is slightly beyond script-kiddie activity.
Pay-per-click advertising will die once enough people know it's being abused like this.</naive optimism>
===
Purveyor of Doc Hope's [link|http://DocHope.com|fresh-baked dog biscuits and pet treats].
[link|http://DocHope.com|http://DocHope.com]
|
Post #259,065
6/15/06 4:18:32 PM
|
It has been slow for me for many months
response time of 15 - 30 seconds to display a page is average, sometimes the request times out. And this includes the office via our T3 line.
Funny (not "haha" funny) thing is that Spiceware is in the same metro area, and he says that his reponse time is below 5 seconds almost always.
I must be on a different off ramp than he is on the information superhighway.
lincoln
"Chicago to my mind was the only place to be. ... I above all liked the city because it was filled with people all a-bustle, and the clatter of hooves and carriages, and with delivery wagons and drays and peddlers and the boom and clank of freight trains. And when those black clouds came sailing in from the west, pouring thunderstorms upon us so that you couldn't hear the cries or curses of humankind, I liked that best of all. Chicago could stand up to the worst God had to offer. I understood why it was built--a place for trade, of course, with railroads and ships and so on, but mostly to give all of us a magnitude of defiance that is not provided by one house on the plains. And the plains is where those storms come from." -- E.L. Doctorow
Never apply a Star Trek solution to a Babylon 5 problem.
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the United States.
[link|mailto:bconnors@ev1.net|contact me]
|
Post #259,101
6/15/06 8:47:35 PM
|
Bwahahaha.
/me fixed many machine little red-wagons
fail2ban work bootifull
Dynamically. If they try more than 3 requests that are either 404s or 302s per minute... oops there they go.
For ten minutes. Which might get increased to 20 minutes.
But only if they are trying to access my default website. (and by IP address only)
It has dropped to less than one request a second. AHHHHH. Down from 15-16 per second.
-- [link|mailto:greg@gregfolkert.net|greg], [link|http://www.iwethey.org/ed_curry|REMEMBER ED CURRY!] @ iwetheyFreedom is not FREE. Yeah, but 10s of Trillions of US Dollars? SELECT * FROM scog WHERE ethics > 0;
0 rows returned.
|
Post #259,141
6/16/06 11:45:06 AM
|
Yet another Update for uptime stats.
I have yet again upated my uptime stats for information included. Now it is updated every 2 minutes, and it shows the currently blocked hosts. These blocked hosts could be blocked for a few reasons. Its cool! [link|http://www.gregfolkert.net/stats/uptime.html|New and Improved UPTIME for Kannigette!] Hosts are listed in this order: - Trying to use a feature that isn't there (300 class of errors in apache)
- Trying to get something that doesn't exist, or won't succeed (400 class of errors in apache)
- Authorization failure, for Apache, SSH or E-mail.
Sorted by class new to oldest. Oldes t being removed after 20 minutes of on list time. If attempts continue, 20 minutes on, until 3 additional failures, on for 20 minutes, etc... forever like that. Eventually they'll realize that if they fail 3 times in a 20 minute period (after failing 5 time initially), they'll be banned. So, then I'll have to worry about the greater than 20 minute failing 3 times people. Down the line. Seeing as this stuff is automated already. I also had an interesting ICQ chat with an operator of one zombie network. Seems he is a bit agitated that I discovered him and his matrix of websites used in discovery of Open Proxies. Then the machines used in PPC Fraud and his (Pharma and pr0n) Websites he operates with tons of banner ads. Personally, I was shaking myself with adrenaline. Now... if it takes me only a short amount of research to find them with public tools available already (Google, whois, DNS resolution, redirect cleaning and other tools) I wonder why law enforcement can't do the same. Total time invested in analyzing - 15 minutes. Total time ICQ chatting with the Operator... 30 minutes. To start all I had was 20 weeks of 3M hits per week, apache logs. I found many interesting patterns once I shredded (analyzed) the data properly. Oh, sent those logs off to that guy in Alabama, that does the working with the FBI thing. Hi first response was basically: "Muahahahaha!" -- [link|mailto:greg@gregfolkert.net|greg], [link|http://www.iwethey.org/ed_curry|REMEMBER ED CURRY!] @ iwetheyFreedom is not FREE. Yeah, but 10s of Trillions of US Dollars? SELECT * FROM scog WHERE ethics > 0;
0 rows returned.
|
Post #259,146
6/16/06 12:06:09 PM
|
you might want to contact this guy as well
[link|http://www-static.cc.gatech.edu/~feamster/|http://www-static.cc...ch.edu/~feamster/] he helped the feebs nail a local bigtime spammer.
thanx,
bill
Any opinions expressed by me are mine alone, posted from my home computer, on my own time as a free american and do not reflect the opinions of any person or company that I have had professional relations with in the past 50 years. meep
|
Post #259,149
6/16/06 12:14:36 PM
|
Law "enforcement"?!? Surely you jest...
You want the whole litany, or just the basics? The Basics? Fine:
1) This isn't Terrorism\ufffd, so it ain't sexy
2) Nobody made Director busting some guy scamming websites
3) Nobody's getting killed, maimed, poisoned, raped or whacked, so it ain't sexy.
4) It's just a bunch o' Geeks and their toys; I won't make Director worrying about them.
5) The "internet"? Whazzat?
6) "Apache"? Didn't Custer wipe all of them out?
7) "Duh..."
The bottom line is that "law enforcement" just doesn't consider this high enough of a priority to bother with. And you're a whole lot smarter, and more savvy, than the vast majority of said "law enforcement"; you know that the tools to do this are already out there, and you know how to use them (and to apply them to the task at hand). Give yourself some credit, what you're doing is not something that the average person (or even the average person who works in the field) would know how to do. And you have a vested interest in doing it, while "law enforcement" could care less.
jb4
"So don't pay attention to the approval ratings that say 68% of Americans disapprove of the job this man is doing. I ask you this, does that not also logically mean that 68% approve of the job he's not doing? Think about it. I haven't."
— Stephen Colbert, at the White House Correspondent's Dinner 29Apr06
|
Post #259,154
6/16/06 1:02:32 PM
|
Thanks muchly! Write up the details!
You should write-up an article on the techniques you used and post it with a pointer to LinuxToday (or some such site) and maybe even send an e-mail to SJVN (who seems to be a reasonably on-the-ball member of the IT press). I'm sure it would help a lot of providers and even small companies that have servers on the Internet.
Cheers,
Scott.
|
