Date: 2024-11-13
Last Update: 2025-03-05
MISP UUID: 306ccb05-4274-412d-bf13-372aa4bd8a14
Email: Computer.Security@cern.ch
It was reported that a computing node was experiencing an unusually high system load and requested support for further analysis. During the forensic investigation, several signs of compromise were identified. A rootkit was found using stealth techniques to hide processes and files, making detection more difficult. Additionally, cryptomining activity was observed, with connections established to an external mining pool. Further analysis uncovered password-stealing scripts actively capturing SSH credentials, as well as command and control scripts orchestrating malicious activities across the system.
Then, it was revealed that six months prior, the site had been contacted by individuals who presented themselves as security researchers. They claimed to have discovered multiple vulnerabilities, effectively getting access to the entire datacenter. The security researchers/attackers sent a report to the site admins detailing their findings. Initially they were trying to help to secure their systems and patch the vulnerabilities. However, the situation took a dark turn when some members of the group went rogue and deployed cryptomining in the compromised infrastructure.
The site admins tried to reinstall all computing nodes, but by this time, the attackers had dug so deeply into the network that they kept persistent access. This explained the later rootkit activity, credential theft, and cryptomining activity detected during the initial investigation.
Investigation 1 refers to the analysis conducted by our team following an alert from a site administrator who detected unusually high system load. At this stage, we were unaware of the prior compromise.
Initial analysis showed no direct traces of infection. However, a suspicious SYN connection was detected, not associated with any process. It did not appear in the process list either, leading us to suspect that it was being hidden by a rootkit.
tcp 0 1 IP_VICTIN:46297 111.118.188.100:443 SYN_SENT -
Rootkit detection tools confirmed suspicious activity, revealing processes that were invisible to "ps".
./chkrootkit Checking `lkm'... You have 77 process hidden for readdir command You have 77 process hidden for ps command chkproc: Warning: Possible LKM Trojan installed 1 /usr/share
By unhiding some of these processes, we identified a cryptominer binary, "xmrig". In its configuration file, we found the domain of the mining pool.
./unhide-linux proc
Found HIDDEN PID: 1042672 Cmdline: "/usr/share/systemd-core/systemd_main" Executable: "/usr/share/systemd-core/systemd_main" Command: "systemd_main" $USER=root $PWD=/usr/share/kernel-dbus
To identify the rootkit module, we investigated hooked syscalls in the kernel. Using various tools, we confirmed kernel-level tampering.
$ ./dist/tracee-ebpf-static --events hooked_proc_fops,hooked_syscall TIME UID COMM PID TID RET EVENT ARGS 02:22:40:721324 0 0 0 0 hooked_syscall syscall: kill, address: ffffffffc09f6240, function: hacked_kill, owner: kernelmgr 02:22:40:721324 0 0 0 0 hooked_syscall syscall: getdents, address: ffffffffc09f6400, function: hacked_getdents, owner: kernelmgr 02:22:40:721324 0 0 0 0 hooked_syscall syscall: getdents64, address: ffffffffc09f6010, function: hacked_getdents64, owner: kernelmgr
$ cat /proc/kallsyms | grep
ffffffffc09f6 ffffffffc09f6010 t hacked_getdents64 [kernelmgr]
ffffffffc09f6400 t hacked_getdents [kernelmgr]
ffffffffc09f6865 t kernelmgr_cleanup [kernelmgr]
...
The function names indicated the rootkit in question was Diamorphine.
We also identified "kernelmgr" as the rootkit module responsible for the hooks. Our next step was to locate the kernel object, which required understanding further the rootkit’s behavior. We saw that the rootkit was hiding itself.
$ lsmod | grep kernelmgr
$ kill -63 123456
$ lsmod | grep kernelmgr
kernelmgr 16384 0
Upon examining the module, we found that the "MAGIC_PREFIX" was set to "kernel-dbus". This prefix caused the rootkit to hide any file beginning with this name.
$ ls /usr/share/kernel-dbus/kernel-dbus_1/main/
DAIPC
$ ls /usr/share/kernel-dbus/kernel-dbus_1/main/kernel-dbus_network
/usr/share/kernel-dbus/kernel-dbus_1/main/kernel-dbus_network
Additionally, it employed a defense evasion technique by hiding files with a specific prefix, which included the rootkit module itself. On one instance, we removed the rootkit.
$ find / -name *kernel*.ko 2>/dev/null
$ rmmod kernelmgr
$ find / -name *kernel*.ko 2>/dev/null
/usr/share/kernel-dbus/kernel-dbus.ko
With the rootkit removed, we could trace the network connections to the associated process.
Before:
tcp 0 1 IP_VICTIN:46297 111.118.188.100:443 SYN_SENT -
After:
tcp 0 1 IP_VICTIN:26611 111.118.188.100:443 SYN_SENT 1937/kernel-dbus_ne
This allowed us to identify the command triggering the binary, which turned out to be the meshagent that the admin had initially reported.
ls -l /proc/1937/exe
lrwxrwxrwx 1 root root 0 Nov 12 12:41 /proc/1937/exe -> /usr/share/kernel-dbus/kernel-dbus_1/main/kernel-dbus_network
$ rpm -qf /usr/share/kernel-dbus/kernel-dbus_1/main/kernel-dbus_network
file /usr/share/kernel-dbus/kernel-dbus_1/main/kernel-dbus_network is not owned by any package
The attacker was also stealing passwords using the "kernel-dbus_sshd.sh" script. This script identifies the PID of the SSH daemon ("sshd -D") running as the "root" user and sets up a directory ("/var/log/sshd_dbg") to store debug logs. It creates unique file paths to avoid overwriting existing files. Using "strace", it attaches to the SSHD process to trace "write" system calls, logging the output to capture plaintext passwords.
The root cause of the observed malicious behaviors (cryptomining, SSH password stealing, command and control, and the rootkit itself) is the "kernel-dbus_start.sh" script.
ps awwx | grep kernel-dbus
1896 ? Ss 24:17 /bin/bash /usr/share/kernel-dbus/kernel-dbus_start.sh
1933 ? S 0:00 /bin/bash /usr/share/kernel-dbus/kernel-dbus_network.sh
1937 ? S 3:47 /usr/share/kernel-dbus/kernel-dbus_1/main/kernel-dbus_network --installedByUser=0
1098739 pts/0 S+ 0:00 grep --color=auto kernel-dbus
This script disables SELinux enforcement, starts multiple scripts, loads a potentially malicious kernel module ("kernel-dbus.ko"), and sends custom signals to these processes and modules to trigger hidden functionalities. It also clears logs ("/var/log/messages" and "/var/log/secure") to erase traces of its activity. Furthermore, it includes a loop that monitors logged-in users ("who") and dynamically restarts the "systemd_main" process with custom behavior upon user login or logout.
The investigation uncovered additional cryptominers masquerading as legitimate QEMU processes. A service named qemu-main.service was designed to execute the miner when no user was connected to the system. Simultaneously, qemu-manager.service ran continuously, executing management.sh, a script responsible for managing and restarting the qemu-main binary. The configuration files for these binaries pointed to the same mining pool and wallet as the previously identified miner concealed in systemd-core.
Additionally, another command-and-control (C2) agent, Dwagent, was deployed. Evidence of its installation was found in the /root/.wget-hsts file, which showed that the tool was downloaded using wget. The file timestamps confirmed that it was retrieved on October 24th, just before its deployment. Although the configuration files for Dwagent, including systray and dwagupd, were deleted, they were successfully recovered using forensic tools like Photorec.
Further analysis of the dnf history logs revealed the installation of various tools used to aid in reconnaissance and lateral movement, including strace, arp-scan, nmap, and sshpass.
The attack timeline revealed a coordinated deployment strategy:
The attackers simultaneously deployed Dwagent (C2) and qemu-main (miner) across all compromised nodes.
A second wave of attacks was executed, introducing meshagent (C2), systemd-core (cryptominer), and the Diamorphine rootkit. This wave relied on advanced stealth and evasion techniques, including the rootkit's ability to hide malicious files and processes.
These operations were organized to ensure persistence and redundancy in the event of partial detection or mitigation. The attackers leveraged overlapping tools and infrastructure to maintain control over the systems and maximize the resources utilized for cryptomining.
Investigation 2 refers to the activity performed by the "security researchers" or attackers, that was reported and sent ton the system administrators 6 months prior to the Investigation 1.
Through Shodan, investigators found that a Baseboard Management Controller (BMC) was exposed through the Supermicro IPMI Web Interface. It was using default credentials, providing access to the iKVM console. Since the disk was not encrypted, the attacker leveraged CentOS rescue mode to create a root admin account. Exploiting the RAID 1 configuration, they kept one disk running the original instance while deploying Proxmox on the second disk. This allowed them to spin up additional virtual machines (VMs) that would use during their attack or so-called "research".
Also, the fact that they could deploy the virtualkized environment on a different disk allowed the attacker to use resources without disrupting the main activities of the site.
To maintain persistence, the attacker deploied a Linux VM that was used as router. It had network services such as DCHP and that was also used to establish a connection with ZeroTier VPN, granting them remote access even if the BMC credentials were rotated.
They used the open source tool flood to perform a TCP flood attack. This allowed them to move laterally across VLANs, identifying additional vulnerable hosts.
flood -I enp33s0f0 -n 32423423414 -t 256
With access to other network segments, they continued with the reconnaissance and it uncovered more exposed BMCs, switches, PDU control panels, and other critical infrastructure components, most of them using default or weak credentials.
The attacker deploied an edge router that allowed them to keep a persisten bypass of the vlans.
allow-hotplug ens18
iface ens18 inet static
address 10.0.50.240/16
post-up ip route add 172.16.0.0/16 via 10.0.225.94
post-up ip route add 192.168.2.0/24 via 10.0.225.94
gateway 10.0.0.1
One of the newly compromised hosts, again accessed via iKVM and CentOS rescue mode, contained unprotected SSH keys belonging to multiple users, including administrators. This discovery enabled the attackers to pivot and gain control over the entire infrastructure.
During their activity, the researchers/attackers updated the Supermicro BIOS/Firmware, upgrade the PDUs to the latest version, and even rotate some of the default credentials. Additionally, they provided a set of recommendations to improve the site security, including:
| IP Address | Port | Detection | Domain Resolved |
|---|---|---|---|
| 111.118.188.100 | 443 | C2 Meshagent | |
| 141.94.96.195 | 80 | Mining Pool IP | pool-fr.supportxmr.com |
| 141.94.96.144 | 80 | Mining Pool IP | pool-fr.supportxmr.com |
| 141.94.96.71 | 80 | Mining Pool IP | pool-fr.supportxmr.com |
| 5.78.111.62 | 443 | Wget of the C2 agent | site008.dwservice.net |
| File Path | Hash | Description |
|---|---|---|
/usr/share/systemd-core/systemd_main |
ea7c97294f415dc8713ac8c280b3123da62f6e56 |
Cryptominer binary |
/usr/share/kernel-dbus/kernel-dbus.ko |
6bfa1bcabf924fc61e26084f45690f2a1f032bb8 |
Malicious kernel module (rootkit) |
/usr/share/kernel-dbus/kernel-dbus_sshd.sh |
519de0365d4cc555522bfc47c8dffe2d68872ca2 |
Password stealing script |
/usr/share/kernel-dbus/kernel-dbus_start.sh |
f6133cfbcd74058a448303f18c866b9992a77df8 |
Orchestrates malicious behaviors and clears logs |
/usr/share/kernel-dbus/kernel-dbus_network |
91971157c3444b92c8944ea535c86f97c8514103 |
Linked to meshagent and cryptomining activities |
/usr/local/share/virt-qemu/qemu-main |
a7fad4de1ce0ed2c137c09d4bf9fe7276555f4a0 |
Cryptominer binary disguised as a QEMU instance |
/usr/local/share/virt-qemu/management.sh |
Script to restart cryptomining dynamically |
Monero Wallet: 47EfLGdX38yADx97fg1ogoMAM2vVrotj7Ute7nwmScTwL1wgokzGE9zX7XwAc7aQpSdWdPYLQCq5xCNesKbmg1DZTkL83bQ