• We did not send an email asking for donations - please read this post.

Suspicious process: /tmp/init -c /tmp/init12.cfg

khitai

New Member
Joined
Jan 20, 2020
Messages
2
Reaction score
0
Credits
0
Soon after the 2020 new year's holiday, I found a suspicious process on my web server:

11777 ? Ssl 30038:48 /tmp/init -c /tmp/init12.cfg

It was using almost 100% cpu, and it was now shown in "netstat -plnt" output.

> ls -l /tmp/init
-rwxr-xr-x 1 root root 902084 Jul 16 2015 /tmp/init

> file /tmp/init
/tmp/init: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped

These was another file in /tmp:

> ls -l javax
-rwxr-xr-x 1 root root 3 Jan 19 01:47 javax
> file javax
javax: ASCII text, with no line terminators
> cat javax
748

And there were a bunch of immutable empty directories in /tmp:

Demon.x86
NoIr_x.86
Ouija_x.86
Yui.x86
a
a_thk
ai.x86
baby
config.json
cross.sh
efjins
evolutions
httpdz
ini
initdz
kerberods
kh
kionai
ksoftirqds
kthrotlds
kworkerds
lilpip
log_rot
mcoin
mcoin-ankit
migrations
networkservice
php
pvds
pvds2
pvds3
r.sh
racks_s
rogue_s
rzx
seasame
skfednw46d.mips
skfednw46d.x86
sqlcan
syscb
sysguard
sysnpmc
sysr.sh
sysupdate
update.sh
watchbog
watchdogs
x86
x86_64
xdsf
xfsalloc
xmrig_s

They all belonged to root user and had a mode of r-xr-xr-x. I had to run "chattr -i -a" and "chmod 755" on them before removing them.

I killed the /tmp/init process and removed all the above files and directories in /tmp, and things seemed to be ok. But today, I found that exactly the same thing happened again, a "/tmp/init -c /tmp/init12.cfg" process using 99% to 100% cpu, and seemingly the same immutable empty directories in /tmp.

Here is the "netstat -plnt" output of my nginx web server:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 6519/master
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 21043/nginx
tcp 0 0 0.0.0.0:26725 0.0.0.0:* LISTEN 17922/sshd
tcp 0 0 0.0.0.0:18983 0.0.0.0:* LISTEN 1336/java
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 2340/php-fpm.conf)
tcp 0 0 127.0.0.1:9001 0.0.0.0:* LISTEN 2340/php-fpm.conf)
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 26649/mysqld
tcp 0 0 127.0.0.1:7983 0.0.0.0:* LISTEN 1336/java
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 21043/nginx
tcp 0 0 0.0.0.0:41490 0.0.0.0:* LISTEN 1336/java
tcp 0 0 0.0.0.0:8983 0.0.0.0:* LISTEN 1336/java
tcp6 0 0 :::26725 :::* LISTEN 17922/sshd

This is Ubuntu 14.04.5 LTS x86_64 with kernel 3.13.0-45-generic, nginx 1.4.6-1ubuntu3.5, php 5.6.23, openssl 1.0.2h, openjdk-7-jre 7u111-2.6.7-0ubuntu0.14.04.3.

Any idea about the root cause or security hole, the hacking method used, measures for security hardening is apreciated, thanks!
 


Rob

Administrator
Staff member
Joined
Oct 27, 2011
Messages
926
Reaction score
1,792
Credits
1,207
Hey there @khitai - welcome to linux.org!

Well, i'd say there's something going on there.. first thing i would do is re-mount /tmp so executables can't be run from there with:
Code:
mount -o remount,noexec /tmp

Then, in your /etc/fstab, add the noexec to the /tmp line with something like:
Code:
/dev/sda4       /tmp              ext4  noexec,nosuid           0       2

Then, delete everything in /tmp

I'd look at all of the files in the documentroot of your site to make sure only your files exist - someone got on there and put their code on your system.

Next, start setting up a new server with updated software - what software does your website use? (wordpress, etc..)

You could also try something like cxs (configserver.com) to look for malicious files on your server.. or a 'rootkit hunter' type script until you have your new server set up.
 
OP
K

khitai

New Member
Joined
Jan 20, 2020
Messages
2
Reaction score
0
Credits
0
@Rob

Thank you for your advices! About /tmp, yes, you are totally right, I should remount it with noexec,nosuid. And I am now also monitoring /tmp using auditd. I checked all the sites configured in nginx, didn't find any unknown file.

Actually I may have found the cause: there is a solr service running on my webserver, and it has https://www.tenable.com/cve/CVE-2019-12409. The fix is here.
 

Rob

Administrator
Staff member
Joined
Oct 27, 2011
Messages
926
Reaction score
1,792
Credits
1,207
Ok - sounds good.. Also, i was going to say if you looked at the init12.cfg file, it'd prob give some info about what it was doing since it was used as a config looks like.
 
$100 Digital Ocean Credit
Get a free VM to test out Linux!

Linux.org Hosting Donations
Consider making a donation

Members online


Latest posts

Top