Check And Solve If Your QNAP NAS Has been Injected a CPUMiner Program


It’s been discussed from 2017/4/18 on disk_manage.cgi hogging CPU usage and 2017/4/28 on  Mobil01: QTS 4.3.3 正式版問題討論區 #86 that there is a mining program using your NAS to work for mineXMR.com

I have check my TS-119P II (QTS 4.3.3) and TS-251A (QTS 4.2.2), none of them have this program. Here is my summary of this issue from internet:

What Happen

A mining program is injected to QTS using your NAS to work for mineXMR.com.

It happens on QTS 4.2.5 build 20170413 on TS-253A which has an Intel N3150 or N3160 reported by esper on disk_manage.cgi hogging CPU usage and 4.3.3 reported by rf5000 on QNAP用戶注意! QTS 4.3.3 被植入XMR挖礦木馬,建議先不要升級!

My TS-119P II which has a Marvell 88F6282 is running QTS 4.3.3 but doesn’t have this issue.

According to #29 on this thread, the program is executed as:

/mnt/HDA_ROOT/disk_manage.cgi -a cryptonight -t 3 -b 127.0.0.1:4048 -o stratum+tcp://pool.minexmr.com:4444 -u 457MwddDoYAPem9bDhkz2tGtzZCyPkuJuQmyFxnpZadXLAc3ymKo2CQ596UzTa6rYES7Dr3FhwdPM6bMMRZxe5CL9XMo7sQ -p x -D -P

According to the [Connection Details] section in [Get Started] tab on mineXMR.com, port 4444 is for Low end CPU. And also the [Mining Apps] section, this program should be CPUMiner (forked by LucasJones & Wolf) which is available on GitHub: OhGodAPet/cpuminer-multi. In the README.md file, it’s described as an x86-64 only program.

Architecture-specific notes:

  • CryptoNight works only on x86 and x86-64.
  • If you don’t have AES-NI, it’s slower. A lot slower, around 1/3rd the speed. This implementation is deprecated and will not be improved.

If you are using a NAS with ARM architecture, you have less chance to get involved in this issue.

How It Hacks

On 2017/5/13, netgear54 shared his hacking forensic investigation on disk_manage.cgi hogging CPU usage. It’s a command injection through Photo Station. Read the detail in Security Vulnerability Addressed in Photo Station 5.4.1 and 5.2.7.

Here is a comparison of command injection to code injection:

Command injection is an attack in which the goal is execution of arbitrary commands on the host operating system via a vulnerable application. Command injection attacks are possible when an application passes unsafe user supplied data (forms, cookies, HTTP headers etc.) to a system shell. In this attack, the attacker-supplied operating system commands are usually executed with the privileges of the vulnerable application. Command injection attacks are possible largely due to insufficient input validation.

This attack differs from Code Injection, in that code injection allows the attacker to add his own code that is then executed by the application. In Code Injection, the attacker extends the default functionality of the application without the necessity of executing system commands.

Here is an example from Testing for Command Injection (OTG-INPVAL-013) on OWASP:

http://sensitive/something.php?dir=%3Bcat%20/etc/passwd

Appending a semicolon to the end of a URL for a .PHP page followed by an operating system command, will execute the command. %3B is url encoded and decodes to semicolon

How to Identify if CPUMiner is Running on my NAS

See if you have high CPU utilization, any strange process from /mnt/HDA_ROOT/, and any unknown schedule tasks on /mnt/HDA_ROOT/. Let me explain the commands you need and show you some examples.

High CPU Utilization

You probably will see high CPU utilization (30% or higher) in [Control Panel] → [System Settings] → [System Status] → [Resource Monitor] → [CPU usage] tab when low network access.

But if your are using QTS 4.3.3, don’t get fooled with the [Resource Monitor] gadget in [Dashboard] which may be launched by upper right corner. It’s not always updated automatically.

Strange Running Process

Enable Allow SSH connection in [Control Panel] → [Network Services] → [Telnet/SSH] tab, ssh as admin and search for process disk_manage.cgi. If /mnt/HDA_ROOT/disk_manage.cgi is found, you probably get infected. Check schedule tasks in next section.

disk_manage.cgi is a standard QTS program but /mnt/HDA_ROOT/disk_manage.cgi isn’t. It’s a fake with the same name to fool you.

Below is QTS 4.3.3 without mining programs:

[~] # ps -ef | grep disk_manage.cgi
 6345 admin       428 S   grep disk_manage.cgi

Below is QTS 4.2.2 without mining programs:

[~] # ps -ef | grep disk_manage.cgi
 8203 admin Z N [disk_manage.cgi]
 8204 admin Z N [disk_manage.cgi]
 8392 admin 428 S grep disk_manage.cgi 

Below is QTS with mining programs according to #29 on this thread, you will see how the program is executed:

[~] # ps -ef | grep disk_manage.cgi
24819 admin  1008 S grep disk_manage.cgi
26472 admin 19136 S /mnt/HDA_ROOT/disk_manage.cgi -a cryptonight -t 3 -b 127.0.0.1:4048 -o stratum+tcp://pool.minexmr.com:4444 -u 457MwddDoYAPem9bDhkz2tGLzZCyPkuJuQmyFxnpZadXLAc3ymKo2CQ596UzTa6rYES7Dr3FhwdPM6bMMRZxe5CL9XMo7sQ -p x -D -P

ps is used to check for current processes. 26472 is the PID (Process ID) of /mnt/HDA_ROOT/disk_manage.cgi in this case. You may learn more detail in PROCESS STATE CODES section in Man Pages PS(1) and a great post on ps aux output meaning on superuser. Here is a list to help you understand the symbols used by ps.

  1. N: low-priority (nice to other users)
  2. S: Interruptible sleep (waiting for an event to complete)
  3. Z: Defunct (“zombie”) process, terminated but not reaped by its parent.

You may also use [Control Panel] → [System Settings] → [System Status] → [Resource Monitor] → [Process] tab to find /mnt/HDA_ROOT/disk_manage.cgi. Because it only shows top 15 processes, I don’t recommend this method.

There are actually 3 suspicious processes running in the background:

  1. /mnt/HDA_ROOT/disk_manage.cgi
  2. /mnt/HDA_ROOT/qwatchdogd.cgi
  3. /mnt/HDA_ROOT/rcu_shed.cgi

Strange Schedule Programs

Enable Allow SSH connection in [Control Panel] → [Network Services] → [Telnet/SSH] tab, ssh login to QTS as admin and search for schedule task rcu_shed. If /mnt/HDA_ROOT/rcu_shed is found, you probably get infected.

Below is QTS 4.2.2 without mining programs:

[~] # crontab -l | grep rcu_shed 
[~] # 

Or you may see all schedule tasks to verify. Here is the list running on the same machine:

[~] # crontab -l
# m h dom m dow cmd
0 3 * * * /usr/local/sbin/ImR_all -soft /Qmultimedia
0 2 * * * /sbin/qfstrim
0 3 * * 0 /etc/init.d/idmap.sh dump
0 4 * * * /sbin/hwclock -s
0 3 * * * /sbin/vs_refresh
0 3 * * * /sbin/clean_reset_pwd
0-59/15 * * * * /etc/init.d/nss2_dusg.sh
30 7 * * * /sbin/clean_upload_file
0-59/10 * * * * /etc/init.d/storage_usage.sh
30 3 * * * /sbin/notice_log_tool -v -R
10 15 * * * /usr/bin/power_clean -c 2>/dev/null
4 3 * * 3 /etc/init.d/backup_conf.sh
35 7 * * * /sbin/qbox_util -c  > /dev/null 2>/dev/null
* * * * * /mnt/ext/opt/netmgr/util/lock_timer.sh
59 12 * * * /share/CACHEDEV1_DATA/.qpkg/QcloudSSLCertificate/bin/ssl_agent_cli

Below is QTS with mining programs according to #17 on this thread and JarnoVanDerLinden’s post on this thread, you will see how the program is scheduled:

[~] # crontab -l | grep rcu_shed
*/3 * * * * /mnt/ext/opt/apache/bin/php /mnt/HDA_ROOT/rcu_shed

Patch Photo Station

Because we know how it hacks, the first step is to upgrade to the latest Photo Station 5.4.1 (for QTS 4.3.x) and 5.2.7 (for QTS 4.2.x). Use App Center to update current version or download the latest from Photo Station page.

Although it focus on x86-64 systems, I suggest everybody to upgrade even you have an ARM model. Next command injection may go through Photo Station again if you choose to stick with existing version.

Remove Malware

JarnoVanDerLinden found there is no autorun.sh and suspect an exploit related to Security Vulnerabilities Addressed in QTS 4.2.3 Builds 20170121 and 20170124 in his post on this thread. Therefore, even you have kill the process and remove the schedule task, it might happen again until you install the security patch.

Before we confirm the exploit and security patch, please remove the malware manually and monitor your NAS regularly.

Kill the Process

To stop continue the mining program, ssh login to QTS as admin and run following commands:

[~] # kill -KILL PID_OF_/mnt/HDA_ROOT/disk_manage.cgi
[~] # kill -KILL PID_OF_/mnt/HDA_ROOT/qwatchdogd.cgi
[~] # kill -KILL PID_OF_/mnt/HDA_ROOT/rcu_shed.cgi

I didn’t know the PID of your processes and it is different from time to time. Therefore, please use ps to check by yourself. It’s the number on the first column.

Remember, there are at least 3 different processes. You might see more PID than 3 if it is running. Kill them all.

  1. /mnt/HDA_ROOT/disk_manage.cgi
  2. /mnt/HDA_ROOT/qwatchdogd.cgi
  3. /mnt/HDA_ROOT/rcu_shed.cgi

Stop Auto-reload

To stop reload the mining program, remove following line in your cron configuration file. Use vi to load /mnt/HDA_ROOT/.config/crontab, delete following line, and overwrite existing file.

*/3 * * * * /mnt/ext/opt/apache/bin/php /mnt/HDA_ROOT/rcu_shed

Some report crontab -e doesn’t work to remove this line which I cannot confirm and have no ideas why.

Update on 2017/5/16 about “crontab -e”

OneCD reply’s in CPUMiner being Injected through Vulnerable Photo Station on QNAP x86 Models on QNAP Forum:

For QNAP NAS, the lack of ‘crontab -e’ is well-known. 😉

https://wiki.qnap.com/wiki/Add_items_to_crontab

I follow the link and it leads me to QNAP Wiki: Add items to crontab. In the Modifying entries section, it explains why it doesn’t work:

However, due to the way the QNAP firmware updates crontab, it will be overwritten on the next reboot. Obviously, you want your automation to survive reboots, so edit the crontab file directly with your text editor:

Delete Mining Program and Related

Remember to delete disk_manage.cgi, rcu_shed, rcu_shed.json, and qwatchdogd in /mnt/HDA_ROOT/ at the end. There is no need to keep them. 🙂

/mnt/HDA_ROOT/qwatchdogd is also a fake program. It’s not the build-in /sbin/qwatchdogd.

Below is a list of this mining programs and related files. It is an modified version report by sapow on mobile01 in #62 of this thread:

[~] # ls -al /mnt/HDA_ROOT/move_tmp 
-rwxrwxrwx 1 admin administ 4774560 Apr 23 23:46 disk_manage.cgi*
-rwxrwxrwx 1 admin administ 5544024 Apr 23 23:46 qwatchdogd*
-rwxrwxrwx 1 admin administ   12299 Apr 29 01:12 rcu_shed*
-rwxrwxrwx 1 admin administ    2827 May  3 20:45 rcu_shed.json*

Use QNAP Malware Remover

Malware Remover in App Center on QTS
Malware Remover in App Center on QTS

There is a [Malware Remover] in [App Center] in your QTS but not available on QNAP App Center page yet. The latest version 2.1.0 may remove this mining program and related files completely.

It will be execute every restart or turn on automatically.

You may also download from here which is a direct link to QNAP. You need to unzip the download file and upload the QDK_2.2.14.qpkg in [App Center] in QTS.

[App Center] may be launch in [Main Menu] on upper left corner. [Malware Remover] can be found in [Utilities] category or just search for “malware”.

The Malware Remover is designed to protect your Turbo NAS against harmful software. QNAP strongly recommends that you install this app to avoid potential security risks. During installation, the app will scan your Turbo NAS and quarantine any detected malware. After installation: – If no warning messages appear after installation, your Turbo NAS is not infected. You can verify the scan results by going to “Control Panel” > “System Logs”. – If a warning message appears, malware was detected on your Turbo NAS and it has now been quarantined. You must now immediately reboot the system and change the administrator’s password. If you have any questions regarding the Malware Remover, please contact us at http://helpdesk.qnap.com/

There is no interactive interface for this program. It just work in the background but you may read messages from it in [Control Panel] → [System Settings] → [System Logs] like below:

Type    Date    Time    Users    Source IP    Computer name    Content    
Information    2017/05/04    09:50:54    System    127.0.0.1    localhost    [App Center] Malware Remover enabled.    
Information    2017/05/04    09:50:54    System    127.0.0.1    localhost    [App Center] Malware Remover 2.1.0 has been installed in /share/CACHEDEV1_DATA/.qpkg/MalwareRemover successfully.    

Here is a screen capture contributed by an infected user who removes them successfully with Malware Remover 2.1.0:

QNAP Malware Remover removes the XMR mining program and related files successfully and report to System Logs
QNAP Malware Remover removes the XMR mining program and related files successfully and report to System Logs

More about this program can be found in Detail Explain of QNAP Malware Remover 2.1.0.

This program is also available in Security Advisory for XMR Mining Program. History release notes may be found on Malware Remover.

TS-269H User with Malware Remover 2.1.2 Issue and Solutions

In [求救] QNAP 中咗malware on HKEPC , TS-269H users reports high CPU utilization after installing/upgrading to the latest Malware Remover 2.1.2. Because of this, everything becomes extremely slow.

The solutions is to roll back to 2.1.1. tcbyxx shared his experience in #63 of this post as below:

  1. Get Malware Removal 2.1.1 on QNAP.
  2. Open the URL to your TS-269H. You probably won’t be able to see it due to busy CPU.
  3. Use power button on TS-269H to turn off.
  4. Press power to turn on.
  5. Stay close with the URL or reload until you see the login page.
  6. Login immediately.
  7. Launch App Center to remove Malware Remover 2.1.2 ASAP.
  8. Reboot your TS-269H again immediately.
  9. Login and install Malware Removal 2.1.1 in App Center.

By the time you read this section, QNAP should restore 2.1.1 for TS-269H.  You probably won’t see 2.1.2 in your App Center.

Update at 2017/5/17 12:03

It has been reported by tcbyxx in #69 of this post  that Malware Removal 2.1.3 is available to download which is not available in release note of Malware Remover yet. It has been tested by stevencheuk that everything goes back normal.

Therefore, you may try to install Malware Removal 2.1.3 in step 7 or 9.

Install Security Patch

Although this attack use a command injection through Photo Station, it’s a good idea to check the latest security patch ASAP especially Security Vulnerabilities Addressed in QTS 4.2.3 Builds 20170121 and 20170124 and Security Vulnerabilities Addressed in QTS 4.2.4 Build 20170313.

It’s not necessary to upgrade your firmware to 4.3.3 if yours is 4.2.x. Just apply the patch.

Best If You May

It is recommended to follow the instruction on An Urgent Fix on the Reported Infection of a Variant of GNU Bash Environment Variable Command Injection Vulnerability on QNAP by reinitialize NAS. They  might also install other backdoors for future visit.

Please read Synology Security Issue and How-to Harden your NAS if you want to know how to secure your NAS.

How to Prevent from Command Injection

I used to work with CodeIgniter and it’s said that CodeIgniter doesn’t run shell commands in CodeIgniter: OS Command Injection on StackOverflow. Actually, there is nothing to do with frameworks. it’s usually because one of the three condition:

  1. Not enough permission to execute with the user rights which runs the web application
  2. Have been disabled in disable_functions in php.ini.
  3. Have been installed Suhosin which is a PHP security extension.

To check disabled functions in your NAS, run phpinfo() or get them from phpinfo() Reports on NAS. Then search for disable_functions section.

To stop exec() and shell_exec(), just add it to disable_functions in php.ini as below. QTS‘s php.ini on can be found in /mnt/HDA_ROOT/.config/php.ini and /etc/config/php.ini according to QNAP QTS Configuration and Executable Files.

disable_functions = "exec, shell_exec"

To stop eval(), you cannot use disable_functions because it is a language construct. eval() is not a function. You need Suhosin and check Suhosin HOWTOs: eval() and other language constructs for detail.

I recommend to read PHP: How To Disable Dangerous Functions on StackOverflow for more discussion and Suhosin on wiki to understand what it can harden your php.

Reference

  1. CodeIgniter
  2. CodeIgniter: function_usable($function_name)
  3. Detail Explain of QNAP Malware Remover 2.1.0
  4. FreeBAS: FreeBSD Man Pages PS(1)
  5. GitHub: OhGodAPet/cpuminer-multi
  6. HKEPC: [求救] QNAP 中咗malware
  7. Intel® Celeron® Processor N3150
  8. Intel® Celeron® Processor N3160
  9. Marvell: Marvell 88F6282 SoC
  10. mineXMR.com
  11. Mobile01: QNAP用戶注意! QTS 4.3.3 被植入XMR挖礦木馬,建議先不要升級!
  12. Mobil01: QTS 4.3.3 正式版問題討論區 #86
  13. Mobile01: sapow的會員資訊
  14. OWASP: Code Injection
  15. OWASP: Command Injection
  16. OWASP: Testing for Command Injection (OTG-INPVAL-013)
  17. php: disable_functions
  18. php: eval
  19. php: exec
  20. php: shell_exec
  21. phpinfo() Reports on NAS
  22. QNAP QTS Configuration and Executable Files
  23. QNAP: An Urgent Fix on the Reported Infection of a Variant of GNU Bash Environment Variable Command Injection Vulnerability
  24. QNAP Forum: CPUMiner being Injected through Vulnerable Photo Station on QNAP x86 Models
  25. QNAP Forum: disk_manage.cgi hogging CPU usage
  26. QNAP Wiki: Add items to crontab
  27. QNAP: App Center
  28. QNAP: Downloads and other support for TS-119P II
  29. QNAP: Downloads and other support for TS-251A
  30. QNAP: Downloads and other support for TS-253A
  31. QNAP: Malware Removal 2.1.0
  32. QNAP: Malware Removal 2.1.1
  33. QNAP: Malware Removal 2.1.2
  34. QNAP: Malware Removal 2.1.3
  35. QNAP: Malware Remover
  36. QNAP: Photo Station
  37. QNAP: QTS
  38. QNAP: Security Advisory for XMR Mining Program
  39. QNAP: Security Vulnerability Addressed in Photo Station 5.4.1 and 5.2.7
  40. QNAP: Security Vulnerabilities Addressed in QTS 4.2.3 Builds 20170121 and 20170124
  41. QNAP: Security Vulnerabilities Addressed in QTS 4.2.4 Build 20170313
  42. QNAP: TS-269H
  43. StackOverflow: CodeIgniter: OS Command Injection
  44. StackOverflow: PHP: How To Disable Dangerous Functions
  45. Synology Security Issue and How-to Harden your NAS
  46. Suhosin: The PHP security extension
  47. Suhosin HOWTOs: eval() and other language constructs
  48. superuser: ps aux output meaning
  49. Wiki: ARM architecture
  50. Wiki: Cron
  51. Wiki: Secure Shell
  52. Wiki: Suhosin
  53. Wiki: vi
  54. Wiki: x86-64

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s