Table of contents of the article:
The hidden cost of compromised sites on shared hosting servers
Anyone who runs a shared hosting server — whether it's a LAMP/LEMP stack, cPanel, Plesk, DirectAdmin, or a custom infrastructure — has almost certainly faced this scenario: One of the hundreds of websites hosted on the server gets compromised, and before anyone knows it, it has silently sent out thousands of spam emails for hours, sometimes days.
The consequences are immediate and devastating. The server's IP address ends up on major blacklists like Spamhaus, Barracuda, and SORBS. All legitimate clients of that server suddenly stop sending emails. The data center opens an abuse ticket. The reputation of the IP—and often the entire range—is compromised. And the worst part? Understand as site is responsible is like looking for a needle in a haystack.
But the consequences don't end there. A server that sends spam consumes CPU, memory, and network bandwidth resources abnormally. The compromised sites' PHP-FPM processes can saturate available workers, slowing down all other sites hosted on the same server. In extreme scenarios, the upstream provider may decide to suspend the entire server or even null-route the IP. And let's not forget the legal implications: anti-spam regulations (CAN-SPAM in the US, GDPR in Europe, Legislative Decree 196/2003 in Italy) also impose liability on those who, through negligence, allow the sending of unsolicited commercial communications from their infrastructure.
This is a problem we deal with on a daily basis at Managed Server, and that's why we developed SMTP Connection Monitor — an open source tool that solves the problem in seconds.
Why traditional approaches don't work
Before diving into our solution, it's important to understand why conventional methods for identifying a spam source on a shared server are so frustrating and, in most cases, completely inadequate.
Mail logs don't tell the whole story. Control /var/log/maillog o /var/log/mail.log It displays email senders and recipients, but on a multi-tenant server with separate PHP-FPM pools for each user, the system user who originated the sending is often absent or unreliable. If the compromised script uses the function mail() PHP's native email routing means the email goes through the local MTA (Postfix, Exim), which logs it as coming from the web server user—not the individual site owner. On a server with 200 sites, this doesn't help at all.
Direct SMTP connections are completely invisible to mail logs. Modern PHP malware is sophisticated. Instead of using mail(), opens direct socket connections to external SMTP servers using libraries like PHPMailer, SwiftMailer, Symfony Mailer, or raw calls to fsockopen() e stream_socket_client()These connections completely bypass the local MTA. They don't appear in any mail logs. Emails are sent directly from the PHP process to Gmail, Outlook, or botnet relay SMTP servers, and your logs show absolutely nothing.
Network tools have no process-level context. It could be launched tcpdump o ngrep and see TCP connections to port 587, but all you get is the source IP (your server) and destination IP. There's no way to know which process, user, or site initiated that connection. On a server with hundreds of active PHP-FPM pools, this information is practically useless.
strace is impractical on scale. Attack strace to every PHP-FPM worker on a production server is unrealistic. It introduces significant performance overhead, is fragile (a worker that dies and is recreated escapes trace), and would require potentially tracing hundreds of processes simultaneously.
PHP-side solutions have drawbacks. Approaches like auto_prepend_file o disable_functions since php.ini They require PHP configuration changes for each pool, can break the functionality of legitimate sites, add latency to every single PHP request, and do not cover cases where the spammer uses a compiled binary or a script in another language.
SMTP Connection Monitor: The Solution
SMTP Connection Monitor is a lightweight program written in C that uses the Linux kernel's auditing subsystem to intercept and log all outgoing connections to SMTP ports (25, 465, 587, and 2525) in real time. For each connection, it displays the exact initiator: the system username, process name, PID, destination IP, and current working directory.
The tool is available on GitHub under the MIT license: https://github.com/MarcoMarcoaldi/smtp-monitor
It operates in three modes: an interactive ncurses-based terminal interface with colored output and keyboard controls, a text mode on stdout for piping to other tools, and a file-logging mode for continuous background monitoring.
Technical Architecture: How It Works Under the Hood
The technical architecture is what makes this tool both reliable and efficient. Here's what happens when you launch it.
Injecting audit rules into the kernel
At startup, the monitor injects audit rules into the kernel via auditctl:
auditctl -a always,exit -F arch=b64 -S connect -k smtp_mon_<PID>
auditctl -a always,exit -F arch=b32 -S connect -k smtp_mon_<PID>These rules instruct the kernel to generate an audit event whenever any process on the system invokes the syscall connect() — the system call used to initiate a network connection. The rules cover both 64-bit and 32-bit ABIs to catch every possible caller. When the monitor terminates, the rules are automatically removed, leaving no trace on the system.
Real-time log parsing with inotify
Instead of opening a raw netlink socket to the kernel audit subsystem (which would conflict with auditd and potentially compromise existing audit infrastructure), the monitor reads /var/log/audit/audit.log in real time using the API inotify of Linux. Think of it as a tail -F highly optimized implemented in pure C.
This design coexists perfectly with auditdThere's no need to stop, restart, or reconfigure the audit daemon. The monitor simply reads the same log file that auditd writes.
It also transparently handles log rotation. Monitoring the parent directory for events IN_CREATE, detects when the log file is rotated (new inode) and automatically reopens the new file, without losing a single event.
Multi-record correlation via serial number
The Linux audit subsystem does not output a single record per event — it outputs multiple related records from a common serial number in the field msg=audit(TIMESTAMP:SERIAL). For a syscall connect(), typically three related records are found:
Record SYSCALL — contains process metadata: PID, UID, effective UID (euid), path to the executable, and command name.
Record SOCKADDR — contains the address of the raw socket (saddr) in the form of a hexadecimal-encoded binary blob, which includes the address family, destination port, and IP address.
CWD Record — contains the working directory of the process at the time of the syscall.
The monitor maintains a circular cache of 4.096 recent SYSCALL records. When a SOCKADDR record arrives, it extracts its serial number, looks up the corresponding SYSCALL in the cache, decodes the socket address into hex format, and if the destination port matches an SMTP port, creates an event with all the related information.
Socket address decoding
Field saddr It's a hex dump of the structure sockaddr of the kernel. For IPv4, the format is: 2 bytes for the address family (0200 = AF_INET), 2 bytes for the port in network byte order, and 4 bytes for the IP address. For IPv6: 2 bytes for AF_INET6 (0A00), 2 bytes for the port, 4 bytes of padding, and 16 bytes for the full IPv6 address.
The monitor decodes both formats, converts them into human-readable IP strings. inet_ntop(), and automatically filters connections to localhost to avoid false positives.
Effective UID Resolution: The Detail That Makes the Difference
This is a critical aspect for shared hosting. When PHP-FPM is configured with single-user pools (as it should always be in a multi-tenant environment), Each pool runs under a different system user. The audit SYSCALL record contains both uid (real UID) that euid (effective UID). The monitor uses theeuid because it correctly reflects the identity of the user under which the PHP-FPM worker is actually operating.
The field parser uses a whole-word matching with boundary detection to avoid a nasty bug: naively search uid= in the audit line it would also match fsuid=, suid= o auid= as substrings, producing completely incorrect results. Our parser checks whether the character preceding the field name is a word separator (space, colon, or start of string).
As a further fallback, if theeuid is 0 (root), the monitor reads directly /proc/PID/status to get the actual UID. This handles edge cases where the audit record might reflect the master process rather than the worker.
Asynchronous Reverse DNS
Pressing [R] In the interactive interface you enable reverse DNS resolution, which is performed in a dedicated background threadThis means the interface never blocks waiting for a DNS query. The results are stored in a thread-safe cache of 512 entries, so each unique IP is resolved only once. The resolver uses getnameinfo() with the flag NI_NAMEREQD to get PTR records, allowing you to view hostnames as mail-wr1-f108.google.com next to the raw IPs.
Thread safety and multi-thread architecture
The program uses three concurrent threads: the main thread handles terminal I/O and ncurses interface rendering; the log reader thread reads and parses the audit file in real time; and the RDNS resolver thread resolves reverse DNS queries in the background. All shared data structures—event buffers, SYSCALL cache, DNS cache—are lock-protected. pthread_mutex_t to ensure consistency in highly competitive scenarios.
Installation and use
The tool compiles as a single binary with only two dependencies: ncurses and pthreads.
# Installazione delle dipendenze di compilazione
# RHEL / AlmaLinux / Rocky / CentOS:
sudo dnf install gcc ncurses-devel
# Debian / Ubuntu:
sudo apt install gcc libncurses-dev
# Compilazione
gcc -O2 -o smtp-monitor smtp-monitor.c -lncurses -lpthread
# Esecuzione
sudo ./smtp-monitorFor interactive mode, launch it without arguments. You'll see a full-screen terminal interface with color-coded events by protocol, a LIVE/PAUSED status indicator, and keyboard controls for scrolling, clearing, RDNS toggles, and more.
For continuous monitoring in production, use file logging mode:
sudo nohup ./smtp-monitor -f /var/log/smtp-connections.log &The output format is one line per connection:
[2026-02-18 13:39:28] USER=sito-compromesso.it PID=1087389 PROC=php-fpm PORT=587 DST=2a00:1450:4001:c21::6c CWD=/home/sito-compromesso.it/htdocs
[2026-02-18 13:39:30] USER=altrosito.com PID=1087401 PROC=php-fpm PORT=465 DST=2a01:4f8:1c1c:5870::1 CWD=/home/altrosito.com/htdocs
[2026-02-18 13:40:15] USER=root PID=29441 PROC=postfix/smtp PORT=25 DST=5.75.214.156 CWD=/var/spool/postfixIn five seconds of reading that log, you know exactly which user — and therefore which website — is sending mail to external SMTP servers.
Operational workflow in practice
Here's how we typically use this tool in a real-world incident response scenario.
Step 1: Start monitoring. As soon as you notice outgoing spam (blacklist alerts, data center tickets, anomalous mail queues, Munin or Zabbix reporting unusual SMTP connections), log in to the server via SSH and launch the monitor in log mode.
Step 2: User identification. Within a few minutes (or seconds, if the spam is actively occurring), repeated entries will appear from one or two user accounts making SMTP connections. The USER column immediately indicates which hosting account is compromised.
Step 3: Account Investigation. The CWD column narrows the search to the document root. It then checks for recently modified files, suspicious PHP scripts, loaded webshells, outdated CMS plugins, or nulled themes containing backdoors.
Step 4: Containment. Suspend the compromised user's PHP-FPM pool, block outgoing SMTP access with iptables/nftables for that specific user, or quarantine the compromised files.
Step 5: Documentation. The log file provides timestamps for each individual connection, useful both for customer communication and for responding to abuse reports from upstream providers.
For proactive monitoring, you can combine the log file with shell analysis:
# Top utenti SMTP nell'ultima ora
sudo ./smtp-monitor -l | tee /tmp/smtp.log &
sleep 3600 && kill %1
awk '{print $2}' /tmp/smtp.log | sort | uniq -c | sort -rn | head -20The tool also integrates easily with Fail2ban for automatically blocking users who exceed predefined SMTP connection thresholds, and with logrotate for automatically managing log file rotation.
Impact on performance
A common concern when discussing kernel audit-based monitoring is the performance impact. The audit rule on connect() It actually adds a small overhead to every network connection on the system—not just SMTP ones. On busy servers handling hundreds of concurrent connections per second, We measured less than 1-2% CPU impact.
The monitor itself is extremely light. It sleeps on inotify and wakes up only when new data is written to the audit log. Memory consumption is about 15-20 MB, dominated by the event buffer and correlation cache. The RDNS resolver thread is completely asynchronous and never blocks event capture.
Key point: Audit rules are automatically removed when you exit the monitor, so there's no residual overhead when the tool isn't running. You can run it for as long as necessary for diagnosis and then close it without worry.
Comparison with other approaches
Over the years, we've experimented with several different approaches. Here's how SMTP Connection Monitor stacks up against the alternatives.
Il parsing mail logs This only works for emails sent through the local MTA. Direct SMTP connections from PHP are completely invisible. It also doesn't reliably identify the responsible system user.
tcpdump and ngrep They can intercept network traffic but can't tell which process or user initiated the connection. On a multitenant server, this is the missing piece that makes the tool useless for our needs.
strace It can identify the process but is extremely invasive. Attacking it on hundreds of PHP-FPM workers on a production server is impractical and introduces significant performance degradation.
PHP auto_prepend_file It can log script names but requires changing the PHP configuration for each pool. It's fragile, can disrupt legitimate sites, and adds latency to every single PHP request served by the server.
eBPF / bpftrace It's technically excellent but requires a recent kernel, BCC/bpftrace tooling installed and configured, and a level of specialized expertise that most hosting administrators lack. It's the right solution for kernel engineers, not for day-to-day hosting management tasks.
SMTP Connection Monitor hits the sweet spot: kernel-level visibility, zero code changes, zero PHP configuration changes, zero disruption to existing services, and simple deployment based on compiling and running a single binary.
Limitations
No tool is perfect, and transparency about limitations is important for informed evaluation.
The monitor requires root privileges. Audit rules management and access to /proc They require elevated permissions to resolve UIDs.
Unable to identify the specific PHP script responsible for the connection. PHP-FPM reads the file .php in memory and closes it before executing the code, so the script path is not available in /proc/PID/fd upon connection. However, the CWD column shows the document root, which immediately identifies the hosting account and website involved. — sufficient information to proceed with the investigation.
On very busy servers, tracing all syscalls connect() Increases audit log volume. The monitor only processes events related to SMTP ports, but auditd still writes everything to disk. Make sure your audit log rotation is configured appropriately to handle the additional volume.
Download it and try it
SMTP Connection Monitor is open source under the MIT license.
GitHub: https://github.com/MarcoMarcoaldi/smtp-monitor
It's a single C file of about 1.100 lines, with no external dependencies other than ncurses and pthreads. It compiles in less than a second. It runs on any Linux distribution with kernel audit support—which today means practically all of them: AlmaLinux, Rocky Linux, CentOS, RHEL, Debian, Ubuntu, and derivatives.
If you manage shared hosting infrastructure and have ever wasted hours trying to track down the source of a spam outbreak, try this tool. It could save you an entire afternoon of debugging—and perhaps even another abuse ticket from the data center.
