IWETHEY v. 0.3.0 | TODO
1,095 registered users | 0 active users | 0 LpH | Statistics
Login | Create New User
IWETHEY Banner

Welcome to IWETHEY!

New This server has been hacked...
...From the looks of it, it was an exim4 exploit detailed here:

http://www.unixsecur...im-cve-2010-4344/

I noticed things running rather slowly tonight, so I decided to check it out. "uptime" showed a load average above 20, which is highly unusual. "ps -eaf | grep root" showed about a gajillion exim4 processes running. "Hmm, that's odd" thought I. So I did some searching.

I found the /var/spool/exim4/s file and before I could think "take a backup" I removed it. Shortly thereafter, I rebooted the server.

Right now, I'm running through a dist-upgrade or two. Hope that'll fix the issues.

[Edit]: Expect some reboots, of course. If we're down overnight/through the morning, it means I went to bed and I'll get back to it later.

[Edit 2]:
[root@sixoftwo exim4]$ pwd
/var/spool/exim4
[root@sixoftwo exim4]$ ls
total 508
drwxr-x--- 2 Debian-exim Debian-exim 4096 2008-03-19 14:31 db/
-rw-r--r-- 1 root root 424 2009-02-11 07:49 gnutls-params
drwxr-x--- 2 Debian-exim Debian-exim 352256 2011-03-19 00:17 input/
drwxr-x--- 2 Debian-exim Debian-exim 135168 2011-03-19 00:17 msglog/
-rwsr-xr-x 1 root root 11095 2011-03-17 09:33 s*
drwxr-x--- 2 Debian-exim Debian-exim 4096 2008-08-28 21:09 scan/
[root@sixoftwo exim4]$ more s

******** s: Not a text file ********

[root@sixoftwo exim4]$ file s
s: setuid ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not stripped


[Edit 3]:
From auth.log:
Mar 17 09:33:01 sixoftwo CRON[24741]: pam_unix(cron:session): session opened for user nobody by (uid=0)
Mar 17 09:33:01 sixoftwo CRON[24741]: pam_unix(cron:session): session closed for user nobody
Mar 17 09:36:23 sixoftwo sshd[24808]: ROOT LOGIN REFUSED FROM 66.212.21.135
Mar 17 09:37:05 sixoftwo sshd[24808]: Failed password for root from 66.212.21.135 port 4942 ssh2
Mar 17 09:37:25 sixoftwo sshd[24813]: ROOT LOGIN REFUSED FROM 66.212.21.135
Mar 17 09:39:01 sixoftwo CRON[24832]: pam_unix(cron:session): session opened for user root by (uid=0)
Mar 17 09:39:01 sixoftwo CRON[24832]: pam_unix(cron:session): session closed for user root
Mar 17 09:40:01 sixoftwo CRON[24864]: pam_unix(cron:session): session opened for user root by (uid=0)
Mar 17 09:40:01 sixoftwo CRON[24864]: pam_unix(cron:session): session closed for user root
Mar 17 09:40:57 sixoftwo su[21768]: pam_unix(su:session): session closed for user root


So I know that the hacks/attempts were coming from that IP address....And that rogue "session closed" without a session appearing to be open looks odd, too.
-Mike

"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
- Benjamin Franklin, 1759 Historical Review of Pennsylvania
Expand Edited by mvitale March 19, 2011, 01:38:52 AM EDT
Expand Edited by mvitale March 19, 2011, 01:43:35 AM EDT
Expand Edited by mvitale March 19, 2011, 01:57:33 AM EDT
New Fixed the immediate issue...
...That had us down for the last day and a half. That issue was related to the RAID array not being able to find half of itself after the initial dist-upgrade I performed from Jaunty to Karmic.

Right now, I'm upgrading from Karmic to Lucid, and then will move to Maverick after that. Couple more reboots are in order, plus I also need to move the server back upstairs, since it's been downstairs in my office for the past 2 days while I've been working on it.

Beyond that, I have changed passwords all around, I'm going to disable all accounts on here but mine, and hope that I don't have to reinstall everything.

I'll keep you updated.
-Mike

"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
- Benjamin Franklin, 1759 Historical Review of Pennsylvania
New Since you have backups of everything...
What is the worst that could happen?

Wipe and Re-install?

Yeah... you go man!
New Zooks! Thank you for all your efforts!
New And er, >thanks again<
Even we dilettantes comprehend the number of intense (nee wasted) hours invested in such a contretemps..

And, btw -- would love some dirt on the perps at 66.212.21.135, after you've expunged their tracks.
Necktie Party, anyone? DDOS --> same addy via Nadir's Raiders?

Ed: opTy plus: isn't there a Honey Pot thingie? ever more sinister than mere digital brute-force vengeance?
;^>
Expand Edited by Ashton March 20, 2011, 07:34:31 PM EDT
New You think that can do more than annoy someone?
The odds that you'd even be fighting back against the true attacker are vanishingly small. These guys always have access to compromised systems, which they use to launch attacks. You're never going to see a connection from their "real" IP.

At best you'd drop one of their bots. And maybe alert its owner that it was compromised. Of course then he'd have your IP as the one that knocked him down.

Feh ... I'm going to go work on something less depressing.



PS: Thanks for the hard work, Mike.
--

Drew
Expand Edited by drook March 20, 2011, 09:11:21 PM EDT
New Sometimes it will get someone into trouble.
But that's merely the admin you annoyed laying into whoever let that machine get compromised. :-/ At best, a whole subnet of machines would be re-assesed, security improved and perhaps a few compromised bots removed. But you'd almost never know if that's what happened or not.

Wade.

Q:Is it proper to eat cheeseburgers with your fingers?
A:No, the fingers should be eaten separately.
New They were reported.
I spoke with a techie at the controlling network for that IP address yesterday morning. I submitted an abuse report with the pertinent information. He informed me that yes, in fact, that was one of their IP addresses. In a colocation facility in China.

Regardless, the administrator of that server had 24 hours to fix the problem or their ethernet cable would get yanked.

That's about the best I could do. We'll see if I ever hear back from 'em.
-Mike

"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
- Benjamin Franklin, 1759 Historical Review of Pennsylvania
New Back years ago...
I was able to track the person back through 4 machines and the trail went dry... and this was in the late 90s.

Do you honestly think its gotten easier?

My honest guess is *AT LEAST* 10 machines between... each compromised to not have trackable logs or anything usable to get to them.

its just not doable anymore.

Plus, more than likely its all scripted. machines going to machines to machines to machines to machines to machines to machines, all coordinated through IRC Chatrooms that have ZERO LOGS.
New {sigh) Pure anonymity has arrived on little pussy-cat feet
..not so colorful a picture as that line about 'the fog' coming in. :-/

New And we're back.
The server is back upstairs. We're now fully up-to-date, running the latest version of Maverick Meerkat!
-Mike

"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
- Benjamin Franklin, 1759 Historical Review of Pennsylvania
New Thank you sir. Your efforts are appreciated
Sure, understanding today's complex world of the future is a little like having bees live in your head. But...there they are.
New You are welcome...
hehehe.

He did the work... I was just a consultant on the project.
     This server has been hacked... - (mvitale) - (12)
         Fixed the immediate issue... - (mvitale) - (2)
             Since you have backups of everything... - (folkert)
             Zooks! Thank you for all your efforts! -NT - (Another Scott)
         And er, >thanks again< - (Ashton) - (5)
             You think that can do more than annoy someone? - (drook) - (2)
                 Sometimes it will get someone into trouble. - (static) - (1)
                     They were reported. - (mvitale)
             Back years ago... - (folkert) - (1)
                 {sigh) Pure anonymity has arrived on little pussy-cat feet - (Ashton)
         And we're back. - (mvitale) - (2)
             Thank you sir. Your efforts are appreciated -NT - (beepster) - (1)
                 You are welcome... - (folkert)

Did anybody else know that they kept a lake behind that big dam?
75 ms