It happens on QTS 4.2.5 build 20170413 on TS-253A which has an Intel N3150 or N3160 reported by esper on disk_manage.cgi hogging CPU usage and 4.3.3 reported by rf5000 on QNAP用戶注意！ QTS 4.3.3 被植入XMR挖礦木馬，建議先不要升級！
According to #29 on this thread, the program is executed as:
/mnt/HDA_ROOT/disk_manage.cgi -a cryptonight -t 3 -b 127.0.0.1:4048 -o stratum+tcp://pool.minexmr.com:4444 -u 457MwddDoYAPem9bDhkz2tGtzZCyPkuJuQmyFxnpZadXLAc3ymKo2CQ596UzTa6rYES7Dr3FhwdPM6bMMRZxe5CL9XMo7sQ -p x -D -P
According to the [Connection Details] section in [Get Started] tab on mineXMR.com, port 4444 is for Low end CPU. And also the [Mining Apps] section, this program should be CPUMiner (forked by LucasJones & Wolf) which is available on GitHub: OhGodAPet/cpuminer-multi. In the README.md file, it’s described as an x86-64 only program.
- CryptoNight works only on x86 and x86-64.
- If you don’t have AES-NI, it’s slower. A lot slower, around 1/3rd the speed. This implementation is deprecated and will not be improved.
If you are using a NAS with ARM architecture, you have less chance to get involved in this issue.
How It Hacks
On 2017/5/13, netgear54 shared his hacking forensic investigation on disk_manage.cgi hogging CPU usage. It’s a command injection through Photo Station. Read the detail in Security Vulnerability Addressed in Photo Station 5.4.1 and 5.2.7.
Command injection is an attack in which the goal is execution of arbitrary commands on the host operating system via a vulnerable application. Command injection attacks are possible when an application passes unsafe user supplied data (forms, cookies, HTTP headers etc.) to a system shell. In this attack, the attacker-supplied operating system commands are usually executed with the privileges of the vulnerable application. Command injection attacks are possible largely due to insufficient input validation.
This attack differs from Code Injection, in that code injection allows the attacker to add his own code that is then executed by the application. In Code Injection, the attacker extends the default functionality of the application without the necessity of executing system commands.
Here is an example from Testing for Command Injection (OTG-INPVAL-013) on OWASP:
Appending a semicolon to the end of a URL for a .PHP page followed by an operating system command, will execute the command. %3B is url encoded and decodes to semicolon
How to Identify if CPUMiner is Running on my NAS
See if you have high CPU utilization, any strange process from /mnt/HDA_ROOT/, and any unknown schedule tasks on /mnt/HDA_ROOT/. Let me explain the commands you need and show you some examples.
High CPU Utilization
You probably will see high CPU utilization (30% or higher) in [Control Panel] → [System Settings] → [System Status] → [Resource Monitor] → [CPU usage] tab when low network access.
But if your are using QTS 4.3.3, don’t get fooled with the [Resource Monitor] gadget in [Dashboard] which may be launched by upper right corner. It’s not always updated automatically.
Strange Running Process
Enable Allow SSH connection in [Control Panel] → [Network Services] → [Telnet/SSH] tab, ssh as admin and search for process disk_manage.cgi. If /mnt/HDA_ROOT/disk_manage.cgi is found, you probably get infected. Check schedule tasks in next section.
disk_manage.cgi is a standard QTS program but /mnt/HDA_ROOT/disk_manage.cgi isn’t. It’s a fake with the same name to fool you.
Below is QTS 4.3.3 without mining programs:
[~] # ps -ef | grep disk_manage.cgi 6345 admin 428 S grep disk_manage.cgi
Below is QTS 4.2.2 without mining programs:
[~] # ps -ef | grep disk_manage.cgi 8203 admin Z N [disk_manage.cgi] 8204 admin Z N [disk_manage.cgi] 8392 admin 428 S grep disk_manage.cgi
[~] # ps -ef | grep disk_manage.cgi 24819 admin 1008 S grep disk_manage.cgi 26472 admin 19136 S /mnt/HDA_ROOT/disk_manage.cgi -a cryptonight -t 3 -b 127.0.0.1:4048 -o stratum+tcp://pool.minexmr.com:4444 -u 457MwddDoYAPem9bDhkz2tGLzZCyPkuJuQmyFxnpZadXLAc3ymKo2CQ596UzTa6rYES7Dr3FhwdPM6bMMRZxe5CL9XMo7sQ -p x -D -P
ps is used to check for current processes. 26472 is the PID (Process ID) of /mnt/HDA_ROOT/disk_manage.cgi in this case. You may learn more detail in PROCESS STATE CODES section in Man Pages PS(1) and a great post on ps aux output meaning on superuser. Here is a list to help you understand the symbols used by ps.
- N: low-priority (nice to other users)
- S: Interruptible sleep (waiting for an event to complete)
- Z: Defunct (“zombie”) process, terminated but not reaped by its parent.
You may also use [Control Panel] → [System Settings] → [System Status] → [Resource Monitor] → [Process] tab to find /mnt/HDA_ROOT/disk_manage.cgi. Because it only shows top 15 processes, I don’t recommend this method.
There are actually 3 suspicious processes running in the background:
Strange Schedule Programs
Enable Allow SSH connection in [Control Panel] → [Network Services] → [Telnet/SSH] tab, ssh login to QTS as admin and search for schedule task rcu_shed. If /mnt/HDA_ROOT/rcu_shed is found, you probably get infected.
Below is QTS 4.2.2 without mining programs:
[~] # crontab -l | grep rcu_shed [~] #
Or you may see all schedule tasks to verify. Here is the list running on the same machine:
[~] # crontab -l # m h dom m dow cmd 0 3 * * * /usr/local/sbin/ImR_all -soft /Qmultimedia 0 2 * * * /sbin/qfstrim 0 3 * * 0 /etc/init.d/idmap.sh dump 0 4 * * * /sbin/hwclock -s 0 3 * * * /sbin/vs_refresh 0 3 * * * /sbin/clean_reset_pwd 0-59/15 * * * * /etc/init.d/nss2_dusg.sh 30 7 * * * /sbin/clean_upload_file 0-59/10 * * * * /etc/init.d/storage_usage.sh 30 3 * * * /sbin/notice_log_tool -v -R 10 15 * * * /usr/bin/power_clean -c 2>/dev/null 4 3 * * 3 /etc/init.d/backup_conf.sh 35 7 * * * /sbin/qbox_util -c > /dev/null 2>/dev/null * * * * * /mnt/ext/opt/netmgr/util/lock_timer.sh 59 12 * * * /share/CACHEDEV1_DATA/.qpkg/QcloudSSLCertificate/bin/ssl_agent_cli
[~] # crontab -l | grep rcu_shed */3 * * * * /mnt/ext/opt/apache/bin/php /mnt/HDA_ROOT/rcu_shed
Patch Photo Station
Because we know how it hacks, the first step is to upgrade to the latest Photo Station 5.4.1 (for QTS 4.3.x) and 5.2.7 (for QTS 4.2.x). Use App Center to update current version or download the latest from Photo Station page.
JarnoVanDerLinden found there is no autorun.sh and suspect an exploit related to Security Vulnerabilities Addressed in QTS 4.2.3 Builds 20170121 and 20170124 in his post on this thread. Therefore, even you have kill the process and remove the schedule task, it might happen again until you install the security patch.
Before we confirm the exploit and security patch, please remove the malware manually and monitor your NAS regularly.
Kill the Process
[~] # kill -KILL PID_OF_/mnt/HDA_ROOT/disk_manage.cgi [~] # kill -KILL PID_OF_/mnt/HDA_ROOT/qwatchdogd.cgi [~] # kill -KILL PID_OF_/mnt/HDA_ROOT/rcu_shed.cgi
I didn’t know the PID of your processes and it is different from time to time. Therefore, please use ps to check by yourself. It’s the number on the first column.
Remember, there are at least 3 different processes. You might see more PID than 3 if it is running. Kill them all.
*/3 * * * * /mnt/ext/opt/apache/bin/php /mnt/HDA_ROOT/rcu_shed
Some report crontab -e doesn’t work to remove this line which I cannot confirm and have no ideas why.
Update on 2017/5/16 about “crontab -e”
For QNAP NAS, the lack of ‘crontab -e’ is well-known. 😉
I follow the link and it leads me to QNAP Wiki: Add items to crontab. In the Modifying entries section, it explains why it doesn’t work:
However, due to the way the QNAP firmware updates crontab, it will be overwritten on the next reboot. Obviously, you want your automation to survive reboots, so edit the crontab file directly with your text editor:
Delete Mining Program and Related
Remember to delete disk_manage.cgi, rcu_shed, rcu_shed.json, and qwatchdogd in /mnt/HDA_ROOT/ at the end. There is no need to keep them. 🙂
/mnt/HDA_ROOT/qwatchdogd is also a fake program. It’s not the build-in /sbin/qwatchdogd.
[~] # ls -al /mnt/HDA_ROOT/move_tmp -rwxrwxrwx 1 admin administ 4774560 Apr 23 23:46 disk_manage.cgi* -rwxrwxrwx 1 admin administ 5544024 Apr 23 23:46 qwatchdogd* -rwxrwxrwx 1 admin administ 12299 Apr 29 01:12 rcu_shed* -rwxrwxrwx 1 admin administ 2827 May 3 20:45 rcu_shed.json*
Use QNAP Malware Remover
It will be execute every restart or turn on automatically.
[App Center] may be launch in [Main Menu] on upper left corner. [Malware Remover] can be found in [Utilities] category or just search for “malware”.
The Malware Remover is designed to protect your Turbo NAS against harmful software. QNAP strongly recommends that you install this app to avoid potential security risks. During installation, the app will scan your Turbo NAS and quarantine any detected malware. After installation: – If no warning messages appear after installation, your Turbo NAS is not infected. You can verify the scan results by going to “Control Panel” > “System Logs”. – If a warning message appears, malware was detected on your Turbo NAS and it has now been quarantined. You must now immediately reboot the system and change the administrator’s password. If you have any questions regarding the Malware Remover, please contact us at http://helpdesk.qnap.com/
There is no interactive interface for this program. It just work in the background but you may read messages from it in [Control Panel] → [System Settings] → [System Logs] like below:
Type Date Time Users Source IP Computer name Content Information 2017/05/04 09:50:54 System 127.0.0.1 localhost [App Center] Malware Remover enabled. Information 2017/05/04 09:50:54 System 127.0.0.1 localhost [App Center] Malware Remover 2.1.0 has been installed in /share/CACHEDEV1_DATA/.qpkg/MalwareRemover successfully.
Here is a screen capture contributed by an infected user who removes them successfully with Malware Remover 2.1.0:
More about this program can be found in Detail Explain of QNAP Malware Remover 2.1.0.
TS-269H User with Malware Remover 2.1.2 Issue and Solutions
The solutions is to roll back to 2.1.1. tcbyxx shared his experience in #63 of this post as below:
- Get Malware Removal 2.1.1 on QNAP.
- Open the URL to your TS-269H. You probably won’t be able to see it due to busy CPU.
- Use power button on TS-269H to turn off.
- Press power to turn on.
- Stay close with the URL or reload until you see the login page.
- Login immediately.
- Launch App Center to remove Malware Remover 2.1.2 ASAP.
- Reboot your TS-269H again immediately.
- Login and install Malware Removal 2.1.1 in App Center.
Update at 2017/5/17 12:03
It has been reported by tcbyxx in #69 of this post that Malware Removal 2.1.3 is available to download which is not available in release note of Malware Remover yet. It has been tested by stevencheuk that everything goes back normal.
Therefore, you may try to install Malware Removal 2.1.3 in step 7 or 9.
Install Security Patch
Although this attack use a command injection through Photo Station, it’s a good idea to check the latest security patch ASAP especially Security Vulnerabilities Addressed in QTS 4.2.3 Builds 20170121 and 20170124 and Security Vulnerabilities Addressed in QTS 4.2.4 Build 20170313.
It’s not necessary to upgrade your firmware to 4.3.3 if yours is 4.2.x. Just apply the patch.
Best If You May
It is recommended to follow the instruction on An Urgent Fix on the Reported Infection of a Variant of GNU Bash Environment Variable Command Injection Vulnerability on QNAP by reinitialize NAS. They might also install other backdoors for future visit.
Please read Synology Security Issue and How-to Harden your NAS if you want to know how to secure your NAS.
How to Prevent from Command Injection
I used to work with CodeIgniter and it’s said that CodeIgniter doesn’t run shell commands in CodeIgniter: OS Command Injection on StackOverflow. Actually, there is nothing to do with frameworks. it’s usually because one of the three condition:
- Not enough permission to execute with the user rights which runs the web application
- Have been disabled in disable_functions in php.ini.
- Have been installed Suhosin which is a PHP security extension.
To stop exec() and shell_exec(), just add it to disable_functions in php.ini as below. QTS‘s php.ini on can be found in /mnt/HDA_ROOT/.config/php.ini and /etc/config/php.ini according to QNAP QTS Configuration and Executable Files.
disable_functions = "exec, shell_exec"
To stop eval(), you cannot use disable_functions because it is a language construct. eval() is not a function. You need Suhosin and check Suhosin HOWTOs: eval() and other language constructs for detail.
I recommend to read PHP: How To Disable Dangerous Functions on StackOverflow for more discussion and Suhosin on wiki to understand what it can harden your php.
- CodeIgniter: function_usable($function_name)
- Detail Explain of QNAP Malware Remover 2.1.0
- FreeBAS: FreeBSD Man Pages PS(1)
- GitHub: OhGodAPet/cpuminer-multi
- HKEPC: [求救] QNAP 中咗malware
- Intel® Celeron® Processor N3150
- Intel® Celeron® Processor N3160
- Marvell: Marvell 88F6282 SoC
- Mobile01: QNAP用戶注意！ QTS 4.3.3 被植入XMR挖礦木馬，建議先不要升級！
- Mobil01: QTS 4.3.3 正式版問題討論區 #86
- Mobile01: sapow的會員資訊
- OWASP: Code Injection
- OWASP: Command Injection
- OWASP: Testing for Command Injection (OTG-INPVAL-013)
- php: disable_functions
- php: eval
- php: exec
- php: shell_exec
- phpinfo() Reports on NAS
- QNAP QTS Configuration and Executable Files
- QNAP: An Urgent Fix on the Reported Infection of a Variant of GNU Bash Environment Variable Command Injection Vulnerability
- QNAP Forum: CPUMiner being Injected through Vulnerable Photo Station on QNAP x86 Models
- QNAP Forum: disk_manage.cgi hogging CPU usage
- QNAP Wiki: Add items to crontab
- QNAP: App Center
- QNAP: Downloads and other support for TS-119P II
- QNAP: Downloads and other support for TS-251A
- QNAP: Downloads and other support for TS-253A
- QNAP: Malware Removal 2.1.0
- QNAP: Malware Removal 2.1.1
- QNAP: Malware Removal 2.1.2
- QNAP: Malware Removal 2.1.3
- QNAP: Malware Remover
- QNAP: Photo Station
- QNAP: QTS
- QNAP: Security Advisory for XMR Mining Program
- QNAP: Security Vulnerability Addressed in Photo Station 5.4.1 and 5.2.7
- QNAP: Security Vulnerabilities Addressed in QTS 4.2.3 Builds 20170121 and 20170124
- QNAP: Security Vulnerabilities Addressed in QTS 4.2.4 Build 20170313
- QNAP: TS-269H
- StackOverflow: CodeIgniter: OS Command Injection
- StackOverflow: PHP: How To Disable Dangerous Functions
- Synology Security Issue and How-to Harden your NAS
- Suhosin: The PHP security extension
- Suhosin HOWTOs: eval() and other language constructs
- superuser: ps aux output meaning
- Wiki: ARM architecture
- Wiki: Cron
- Wiki: Secure Shell
- Wiki: Suhosin
- Wiki: vi
- Wiki: x86-64