Today when I tried to open one of my websites https://longswims.net/ in order to update its contents, it stopped working throwing HTTP 500. I immediately signed in the web server and found out that the .php file was injected unknown code.
I immediately turned off the web server and investigate the situation. I founded out multiple .php files across multiple web sites were injected unknown code causing them to stop working. The attacks are limited to .php files in the folders /var/www and /opt where the user-account www-data is writable. According to the timestamps the attack started at 06:33 HKT today by modifying the plugin file akismet/akismet.php and hello-dolly/hello-dolly.php in my website https://blog.miklcct.com/ . Further investigation of my web server log yielded the following:
blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:16 +0800] "GET /phpmyadmin/ HTTP/1.1" 200 3548 "http://blog.miklcct.com/phpmyadmin/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:22 +0800] "POST /phpmyadmin/index.php HTTP/1.1" 302 20 "http://blog.miklcct.com/phpmyadmin//index.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:23 +0800] "GET /phpmyadmin/index.php HTTP/1.1" 200 14329 "https://blog.miklcct.com/phpmyadmin/index.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:33 +0800] "GET /phpmyadmin/index.php HTTP/1.1" 200 14329 "http://blog.miklcct.com/phpmyadmin//index.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:34 +0800] "POST /phpmyadmin/import.php HTTP/1.1" 200 10200 "http://blog.miklcct.com/phpmyadmin//import.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:36 +0800] "GET /phpmyadmin/index.php HTTP/1.1" 200 14537 "http://blog.miklcct.com/phpmyadmin//index.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:42 +0800] "POST /phpmyadmin/import.php HTTP/1.1" 200 10887 "http://blog.miklcct.com/phpmyadmin//import.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:43 +0800] "GET /phpmyadmin/index.php HTTP/1.1" 200 14634 "http://blog.miklcct.com/phpmyadmin//index.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:45 +0800] "POST /phpmyadmin/import.php HTTP/1.1" 200 8480 "http://blog.miklcct.com/phpmyadmin//import.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:50 +0800] "GET /phpmyadmin/index.php HTTP/1.1" 200 14862 "http://blog.miklcct.com/phpmyadmin//index.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:52 +0800] "POST /phpmyadmin/import.php HTTP/1.1" 200 10455 "http://blog.miklcct.com/phpmyadmin//import.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:53 +0800] "GET /phpmyadmin/index.php HTTP/1.1" 200 14936 "http://blog.miklcct.com/phpmyadmin//index.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:55 +0800] "POST /phpmyadmin/import.php HTTP/1.1" 200 8603 "http://blog.miklcct.com/phpmyadmin//import.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:56 +0800] "GET /wp-login.php HTTP/1.1" 200 1707 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:32:58 +0800] "POST /wp-login.php HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:02 +0800] "GET /wp-admin/ HTTP/1.1" 200 21380 "https://blog.miklcct.com/wp-login.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:06 +0800] "GET /wp-admin/ HTTP/1.1" 200 21381 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:07 +0800] "GET /wp-admin/plugins.php?plugin_status=inactive HTTP/1.1" 200 13816 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:09 +0800] "GET /wp-admin/plugin-editor.php?plugin=akismet/akismet.php&Submit=Select HTTP/1.1" 200 12953 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:10 +0800] "POST /wp-admin/plugin-editor.php HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:12 +0800] "GET /wp-admin/plugin-editor.php?a=1&plugin=akismet/akismet.php&file=akismet/akismet.php HTTP/1.1" 200 13110 "https://blog.miklcct.com/wp-admin/plugin-editor.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:13 +0800] "POST /wp-content/plugins/akismet/akismet.php HTTP/1.1" 403 282 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:14 +0800] "GET /wp-admin/plugin-editor.php?plugin=hello-dolly/hello.php&Submit=Select HTTP/1.1" 200 12455 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:15 +0800] "POST /wp-admin/plugin-editor.php HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:16 +0800] "GET /wp-admin/plugin-editor.php?a=1&plugin=hello-dolly/hello.php&file=hello-dolly/hello.php HTTP/1.1" 200 12619 "https://blog.miklcct.com/wp-admin/plugin-editor.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:06:33:17 +0800] "POST /wp-content/plugins/hello-dolly/hello.php HTTP/1.1" 200 12 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
The above indicated the attacker was from IP address 2a07:5741:0:7c0::1 and probably used Firefox to sign in the WordPress back end, opened the plugin editor and put the unknown code. Before that, he might have used phpMyAdmin to manipulate the database.
I suspect that the attacker might know the password as the log suggested that the attack was done from a normal way of using the website, however all my WordPress and MySQL accounts have strong passwords consisting of lower case and upper case letters and also numbers which make brute-forcing unlikely.
In order to remove the polluted codes and recover the website, I removed the contents of /opt and /var/www and restored a backup taken at 00:49 today, which was done automatically weekly from another VPS using rsync.
However, I still needed to found out the attack entry point and block it before I could restart the website again. I looked at the MySQL database of the compromised WordPress website and found out that there was an extra user which shouldn’t exist, that means the attacker had made a user in my WordPress installation to make the attack.
Finally, by looking the web server log more thoroughly, I found the following line:
blog.miklcct.com 2a07:5741:0:7c0::1 - - [12/Apr/2020:01:10:11 +0800] "GET /wp-config.php~ HTTP/1.1" 200 3177 "http://blog.miklcct.com//wp-config.php~" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
I tried to open the link, f**k it, the whole config including the database user name and password was leaked, which meant the attack was trivial from that point onward. I remembered that the Apache config on some OSes, by default, blocked the serving of such backup files but it was not the case, causing the attack.
At that point, I had to change the Apache web server config to block the serving of files ending with ~ (which was automatically created by text editors), restore the database of my WordPress blog to an earlier version on 2020-04-10 before the attack happened, changed the passwords of the app user in the database and the www-data system account. After that I believed that my VPS was clear and restarted the website again.
1 thoughts on “Personal blog was cracked today”