Debain Server hacked via SSH pw login / my pw in auth.log in clear found

[horstmc]

Greetings to you,

A few days ago may server was hacked and crashed via Out of memory.
In the auth.log i discovered my password as username. So my pw was stolen.
I got a lot of successful logins from many ips in my auth.log!

Source of my pw i guess:

I had saved the password of the debain server on a Windows 10 computer in WinSCP (which of course you should not do). The Windows computer was full of malware, I later realized
The malware on my debain 12.4 system left whitecat in /usr/bin
the bash was reset and all ssh keys.
in addition a strange entry in /etc/passwd:
htop:x:0:0:root:/root:/bin/bash

My leaked password is: fjgurdk7824!
Maybe some one can find it in a database of leaked pw.

None of the files were encrypted from the malware. I removed as much as possible from the debain machine which seemed strange to me.
I think it was a crpytominer because the system was completely overloaded and crashed.

I have deactivated password login, only login via pub key.
I will gladly post further system information

Do you have any idea what kind of malware it was?

 

[Condobloke]

G'day horstmc, Welcome to Linux.org

Have you reinstalled or taken other steps to clear your pc of any further infection ?

 

[GatorsFan]

The malware on my debain 12.4 system left whitecat in /usr/bin

whitecat is not malware per se - it is a utility to automate the capture of an 802.11 PMKID to crack pre-shared key passwords

GitHub - XORMANCER/WHITECAT: Crystal utility to automate PMKID capture via HCXDUMPTOOLS

Crystal utility to automate PMKID capture via HCXDUMPTOOLS - XORMANCER/WHITECAT

github.com

technically it is a hacking tool might be on Kali or Parrot

 

[horstmc]

Thanks for your answers!

@Condobloke, no reinstall, I testet system with clamAV, rootkithunter, chkrootkit and maldetect.
Nothing found after the reboot.
I guess it was an crypto miner running or script using my server for ddos, because system was OOM when I arravied at the server physically and saw the pc monitor with the OOM message.
Server runs normally now. Of course I changed all my passwords and disabled pw login.

Now I will setup a second honeypot server with the stolen ssh pw as login. I programmed a kind of logger which will log all terminal command independant from bash history. May be the bot will come again and I can discover what happend in detail.

@GatorsFan
So what do you mean? I think it was whitecat log cleaner:

whitecat.c ≈ Packet Storm

Information Security Services, News, Files, Tools, Exploits, Advisories and Whitepapers

packetstormsecurity.com

because bash histroy and .ssh content was gone.

 

[GatorsFan]

@GatorsFan
So what do you mean? I think it was whitecat log cleaner:

the whitecat I was referring to is for Wifi hacking not Log cleaning - guess there is more then one whitecat out there

 

[blunix]

I think it was a crpytominer because the system was completely overloaded and crashed.

That sentence is incorrect: it should read one "of of the malware that was installed by one of the attackers was a cryptominer".

Do you have any idea what kind of malware it was?

One of the malwares that was installed on your system is likely to be a cryptominer.


What to do now

Mentally prepare to throw away all the data the server had as rootkits can be hidden in all files that you can not manually inspect with a text editor or where you don't have a hashsum for (keyword md5deep apt package). Forget looking for strings / hashsums of known malicious files (keyword virus scanner), as rootkits can be easily obfuscated so you can not find them with such a scanner.

Shutdown the server and boot it into a rescue system (live system). Mount the old disk with no exec mount option. Copy over the data you think you can trust or that you can not loose (that sucks but there is no option). A skilled attacker will now forever stay on your system.

Wipe the disk (bit overkill but I'd do it anyways) and install a completely fresh operating system. Make sure to follow reasonable security, or as I would call it, make sure to install a door to your house. You using such a password is either "not installing a door" or "leaving it open"
Restore the services and the data and hope all of the attackers that compromised your machine were just script kiddys running bots.

Even though this is written a bit harsh, that happens. Just inform yourself about better security and stop using passwords where you can.

A reasonably secure password looks like this:

user@host:~$ pwgen -sy $((32 + RANDOM % 64)) 1
x.em8XrbjQy>zBHVb#fX{%<'i;@)vZBteNiYQ'Pb7Sc$8g0e~&=%HTyKpL^np.G^"Dq"cW$X\%

Use such a thing for the password of your login for this forum and use a password manager to store it. Do not store them in the Browser, but use something like the pass password-store or - now much cooler with age encryption which uses your ssh keypair so you don't need to mess with this annoying gpg anymore - the passage password store

 

[f33dm3bits]

Disable password authentication on a server connected to the internet, setup a whitelist firewall for ssh access and use port-knocking for the ssh port so that you can access it temporary from another location if needed.

 

[blunix]

@f33dm3bits thats the pro version and more than required to deter the common script kiddy bot. @op if you go with this, you can feel very secure.

If you go with this check out my favorite port knocking tool: fwknop

Keep in mind that port knocking will open traffic to your source IP - if you do that from a "coffee shop", everyone in the coffee shop will be allowed traffic to your server IP.
Thats 8 billion people less than before though

 

[f33dm3bits]

Keep in mind that port knocking will open traffic to your source IP - if you do that from a "coffee shop", everyone in the coffee shop will be allowed traffic to your server IP.
Thats 8 billion people less than before though

You can configure port-knocking to start dropping connections after an x amount of minutes except for the current active connections, that way it still limits it.

 

[blunix]

You can configure port-knocking to start dropping connections

yes, that is correct. Personally at this point I think I would go with wireguard.

I feel like we're running off-topic here His server was hacked due to a weak password. Now we are discussing port knocking configuration in detail.

If it was realistic for the op to configure that, we wouldnt be posting around in this thread

Lets keep it at "you should lock the door" instead of "have you heared about the new ultra-vault-5000 carbon-fibre-reinforced-50-inch-steel-door"

 

[horstmc]

thx @blunix and @f33dm3bits

Yes, password login is a bad idea, I know. So passwd login is disabled now. Next step I will block all ips except mine.
My ISP has a changing ip, so I have to update the iptables automaticly ...
I did not know port-knocking, I will have a look at the topic. Good idea.


Even if you have a save looking pw and you are so stupid like me and save the pw on a windows 10 system full of malware in WinScp without master password, it will be to late...

Yes, I also guess my system was used for ddos remote and also at the end when it crashed as miner.
But no files got encrpyted ...


A lot of work to reinstall the complete system with all its services, I will do it.
Physically I got complete paranoid now. I am running a linux server since 20 years and got too reckless

 

[f33dm3bits]

Yes, password login is a bad idea, I know. So passwd login is disabled now. Next step I will block all ips except mine.
My ISP has a changing ip, so I have to update the iptables automaticly ...

Glad you took up my advice, you can write a script for that. Search for your ip adress in one of the log files and if it's different than the one currently in the firewall have it replace it with the new one. I don't know how often your ip changes, but if doesn't change that often you could just decide to use port-knocking and change it manually for when it does change. I have a dynamic ip at home but it actually rarely changes.

I feel like we're running off-topic here His server was hacked due to a weak password. Now we are discussing port knocking configuration in detail.

The main point was it is a bad idea to have password authentication enabled for the ssh port on a system that is connected to the internet, having an ssh keypair with passhprase would have probably prevented this even if the passphrase was weak because they would have still needed access to the key in order to log in. And if you add least privilege access principle to that the port wouldn't have been accessible by the person who was guess/brute forcing passwords and maybe having fail2ban watch the ssh port would have blocked it after 2-3 attempts as well.

 

[vs2-free-users]

Disable password authentication on a server connected to the internet, setup a whitelist firewall for ssh access and use port-knocking for the ssh port so that you can access it temporary from another location if needed.

I think this is the best solution too. Additional its a good idea to change the listen port of ssh.

 

[blunix]

I am administrating ~200 Debian Linux Servers at the moment. Combined over all of those, the worth of those servers + their software + the cost of a week of downtime or the servers being compromised would be many million usd.

Here is what they "do not get hacked", as in my strategy to get around stuff like the OPs problem:

1) I have two (redundant) wireguard gateway servers. This we use to get access to all other servers. The other servers and the gateway servers are in a wireguard mesh private network. SSH is only allowed from the mesh IPs of the gateways, as well from the public IPs from the gateways. All servers block SSH traffic from the outside world. the firewall allows only the public IPs of all other servers for the wireguard mesh port.
2) The gateways have the wireguard for humans port open to "the internet", as well as the ssh port. As in these are not restricted to any IPs. This just breaks somehow and then we can't login when something goes down.

I run Debian stable mostly. If you can break into a normally configured openssh-server on Debian stable, then there is most likely nothing I can do anyways, because you can just send me a well crafted email with a malicious .pdf (thats why I use qubes, but my customers ONLY ever get hacked this way).

Btw, all of this is documented and open source - the code we use to setup is open source and the documentation for exactly how we do that is as well. We do security by security, not obscurity.

Restricting a well configured openssh-server on a server that is run by an individual and not a multi million usd company is nonsense in my opinion. If you can get around the normally configured ssh server, you can get around if its restricted by iptables / nftables or not.

In my opinion this is just making things more complicated than they need to be. To much security can be bad as well - if he messes up this iptables thing, he can't login, then he has to boot the server into a rescue system, mount the root disk, fix iptables. Can he do that?
Imho this approach will cause more damage to the OP than the first recommendation (disable ssh pw auth).

 

[blunix]

setup a whitelist firewall for ssh access and use port-knocking for the ssh port so that you can access it temporary from another location if needed.

These are kinda two different things.

1) restricting via iptables: this means that you can only access the servers port 22 if you come from IP 1.2.3.4. ALL OTHER traffic is blocked.
2) port knocking: this means that a (udp) port is open to the internet, and if you send the right sequence of packets there, it will open port 22 (SSH) to the IP that send those packets.

Hence, if you permanently want to only allow one IP address to be able to communicate with the server, port knocking is not the correct approach. Then you need a static IP, or you need to know your next IP in advance.

All of this is total overkill for the OP, I'm reasonably sure he lost us half way along the way already, because up till now he had "foobar1234" as his SSH password

Take it from me I do Linux every day: just setup ssh key authentication and disable passwords. Do this for all linux users on your server just to be paranoid:

passwd --lock root
passwd --lock any-other-user

man passwd:

       -l, --lock
           Lock the password of the named account. This option disables a password by changing it to a value which matches no possible encrypted value (it adds a ´!´ at the beginning of the password).

           Note that this does not disable the account. The user may still be able to login using another authentication token (e.g. an SSH key). To disable the account, administrators should use usermod --expiredate 1 (this set the
           account's expire date to Jan 2, 1970).

           Users with a locked password are not allowed to change their password.

If there is no password, h4x0rs can not use a password to login.

Make sure SSH public key authentication works before you do that. Its literally what I do. For my private webserver that runs websites for friends I do exactly that and pretty much nothing else.

What you can do in install fail2ban, but its kind of useless as there is no way to guess an ssh key.

• Make sure you do this: apt install unattended-upgrades; dpkg-reconfigure unattended-upgrades select Yes here to install security upgrades automatically.
• Reboot your server once in a blue moon to get a new kernel version.
• Make sure your OS is always latest stable.
• Generate your SSH keypair like this: ssh-keygen -t ed25519 -a 100 -o

Thats literally all you need. Its Linux not Windows. It comes secure by default. You chose a bad password and made it insecure. There is no need to overkill things now.

 

[f33dm3bits]

Take it from me I do Linux every day:

I am administrating ~200 Debian Linux Servers at the moment.

I do the same except I administer RHEL servers.

2) port knocking: this means that a (udp) port is open to the internet, and if you send the right sequence of packets

There's solutions to that, if I remember correctly you can do it over an encrypted connection, I can't exactly remember the details right now.

All of this is total overkill for the OP, I'm reasonably sure he lost us half way along the way already, because up till now he had "foobar1234" as his SSH password

Agreed. Everyone that responded was just trying to improve the ssh setup of the OP.

 

[vs2-free-users]

@blunix the problem is after all. Wireguard uses a key too. So if a hacker want to f... your server he can try to crack your wireguard key. So finaly its nearly the same security if you are using ssh with key and without password. The port knocking is only to open the firewall.

 

[blunix]

No, he will not try to crack the wireguard key. He will exploit a human weakness like send me a well crafted email with a pdf (thats why i use qubes os).

That is not realistic and it does not happen. It happens if the government is after you MAYBE, and even then it doesnt make sense.

Cracking wireguard or ssh keys requires your datacenter to burn through lots of time and electricity and is totally unrealistic.

If someone will crack wireguard to get to my server I will never know. I'm not batman, Im just a sysadmin. If someone is that skilled I will not notice the attack. Even with SIEM and me eagle-eyeing the SIEM 24/7 I will not notice that.

But there is no need to. Pretty much everyone uses windows. Thats SO MUCH MORE EASY to get into than wireguard.

I use qubes, but that also has 0days that are exploited by governments. The government DOES NOT CARE about your wordpress server

 

[blunix]

There's solutions to that, if I remember correctly you can do it over an encrypted connection, I can't exactly remember the details right now.

SSH is encrypted too

How many layers of encryption do you need?

 

[blunix]

The port knocking is only to open the firewall.

I do not understand this sentence. A successful port knock modifies the firewall on the server to open a port (or open the floodgates / all ports and protocols) to a specific IP.