Simple Mail Transfer Protocol (SMTP): architecture, packet analysis, server configuration, and security testing

What is SMTP?

SMTP is the application-layer protocol used to submit and transfer email between mail systems. RFC 821 specified it in 1982, and RFC 5321 is the current core specification. Email infrastructure has changed a lot since then, but SMTP still carries messages across organizational and network boundaries.

It was designed to move messages between systems that could not always reach each other directly or be online at the same time, so it uses a store-and-forward model. A sending system hands a message to another mail server, which queues it if the destination is temporarily unavailable and later delivers it locally or forwards it on.

That makes SMTP tolerant of temporary failures, but it leaves the mail server with real work: queuing, retry policy, recipient validation, relay authorization, and delivery status. A server has to separate legitimate mail from unsolicited traffic, decide whether it is responsible for a recipient domain, and stop unauthorized clients from using it to distribute messages.

SMTP does not by itself encrypt every connection, authenticate every sender, guarantee that the visible sender address is genuine, or protect against malicious content. Those jobs fall to TLS, SMTP authentication, SPF, DKIM, DMARC, anti-spam filtering, and server-side controls.

For a security practitioner, that means looking at both the protocol exchange and the software that implements it. A server can follow the specification and still be vulnerable because of an outdated MTA, a bad relay policy, an exposed authentication service, or a flaw in a webmail application on the same infrastructure.

The email processing Chain

Email delivery involves four different “agents” – the Mail User Agent (MUA), Mail Submission Agent (MSA), Mail Transfer Agent (MTA), and Mail Delivery Agent (MDA). They are responsibilities, and a deployment does not need four separate machines or processes to have them.

Mail User Agent (MUA)

The MUA is the application a user composes, sends, and reads mail in. Desktop clients, mobile apps, and webmail all play this role, though webmail adds an application server between the browser and the mail infrastructure.

To send, the MUA passes the message to a submission service, usually over an authenticated SMTP connection or an application-specific interface that eventually reaches an MSA. To read, it usually accesses the mailbox through IMAP or another mailbox-access mechanism.

The difference between submission and retrieval matters in security assessments. SMTP sends and transfers messages, while IMAP and POP3 access stored ones. Compromising either exposes email, but the attack surface and the controls differ.

Mail Submission Agent (MSA)

The MSA accepts outgoing messages from users and applications and applies submission policy: requiring authentication, checking sender authorization, enforcing size limits, and capping how many messages a client can submit.

Port 587 is the standard submission port under RFC 6409. Port 465 is also used for submission over implicit TLS, as described in RFC 8314. With implicit TLS, the handshake starts as soon as the TCP connection is established. With STARTTLS, the client and server begin with a plain SMTP exchange and upgrade to TLS if the extension is available and accepted.

An MSA is more than an SMTP relay on a different port. Submission and relay serve different purposes and need different policies. An authenticated user may be allowed to submit mail for external recipients, while an unauthenticated connection to the public inbound service should not get the same privileges.

Mail Transfer Agent (MTA)

The MTA transfers email between mail systems. It accepts messages, evaluates routing information, queues them when necessary, and forwards them to the next destination.

To deliver to another domain, an MTA generally queries DNS for the domain’s Mail Exchange (MX) records, which identify the servers responsible for accepting its mail, and then connects to one of them over SMTP. A message from alice@example.net to bob@example.org might pass through a submission server run by example.net, an outbound MTA, and an inbound MTA for example.org, which hands it to a local delivery component or another internal service.

Real paths vary. Cloud email platforms, hosted mailboxes, filtering gateways, and enterprise deployments often combine roles or add relays. What stays constant is the split between submitting a message, transferring it between systems, and placing it in a mailbox.

Mail Delivery Agent (MDA)

The MDA handles final delivery into a local mailbox or mailbox storage system. Depending on the implementation, that can mean writing the message to disk, updating mailbox indexes, applying server-side rules, or passing it to a storage service. The MDA can be a separate component, but some MTAs deliver locally themselves, and larger deployments use dedicated mailbox servers or hosted storage.

After delivery, the recipient reads the message over IMAP, in a webmail application, or through another mailbox interface. POP3 is still available in some environments but is less common where mailboxes are synchronized across devices.

How a message moves through the system

A typical delivery sequence looks like this:

1. The sender composes a message in an MUA.
2. The client submits the message to an MSA, usually over an authenticated connection.
3. The submission service applies its policies and passes accepted mail to an outbound MTA.
4. The outbound MTA finds the destination using DNS and attempts an SMTP connection.
5. The receiving MTA evaluates the connection, sender, recipient, and applicable filtering policies.
6. If it accepts the message, it queues it or passes it to a local delivery component.
7. The recipient reads the delivered message through a mailbox service.

A message can pass through several MTAs on the way. Each may accept responsibility for the next stage, but acceptance by one server does not mean the message has reached the final mailbox. A later failure can still cause a delay, a non-delivery report, or other handling defined by the mail system.

SMTP ports, commands, and session states

SMTP runs over TCP. Port 25 carries transfer between mail servers, port 587 carries submission, and port 465 carries submission over implicit TLS. Treating them as interchangeable leads to wrong firewall rules and authentication policies.

Port 25 generally has to stay reachable between independently operated mail systems for direct delivery. An organization can block outbound port 25 from workstations and application servers while allowing it from designated relays, which makes it harder for compromised hosts to send mail straight to external servers.

SMTP is command-driven. The server answers with numeric status codes and often explanatory text, and the client has to read those responses instead of assuming a command worked because the server replied.

Connection establishment and EHLO

After the TCP three-way handshake, the server normally sends a greeting starting with 220. The client identifies itself with EHLO, which requests the server’s extended capabilities. A legacy client may use HELO, but EHLO is required to negotiate extensions.

A simplified exchange:

S: 220 mail.example.org ESMTP
C: EHLO client.example.net
S: 250-mail.example.org
S: 250-STARTTLS
S: 250 SIZE 52428800

These lines are illustrative, not a transcript from a particular server. A real server advertises capabilities according to its configuration, software version, and connection state. It may offer STARTTLS, SIZE, AUTH, or PIPELINING, but no capability list is mandatory.

The EHLO response is useful in packet analysis because it exposes part of the server’s protocol surface. Advertising a feature does not mean the feature is configured correctly, and it does not mean the server is vulnerable.

MAIL FROM and RCPT TO

Once the session is initialized, the client starts a mail transaction. MAIL FROM gives the envelope sender, and RCPT TO names one or more envelope recipients.

C: MAIL FROM:<alice@example.net>
S: 250 2.1.0 Sender accepted
C: RCPT TO:<bob@example.org>
S: 250 2.1.5 Recipient accepted

The envelope sender and recipient belong to the SMTP transaction and are separate from the visible From: and To: headers inside the message. That gap matters for security: an attacker can make a message look like it came from a trusted address while supplying a different envelope sender.

How the server answers depends on its policy. It may reject a recipient because the address is invalid, the server is not responsible for the domain, the sender is unauthorized, or an anti-abuse rule fired. Some servers deliberately defer certain checks until later in the transaction to limit directory enumeration and information disclosure.

DATA and message termination

After the envelope is set, the client sends DATA, and the server normally answers 354, meaning it is ready for the message content.

C: DATA
S: 354 End data with <CRLF>.<CRLF>
C: From: Alice <alice@example.net>
C: To: Bob <bob@example.org>
C: Subject: SMTP test
C:
C: This is a test message.
C: .
S: 250 2.0.0 Message accepted for delivery

The message is headers, a blank line, and a body. The end-of-data marker is a line containing a single period, preceded by the required line termination. Because message content can contain lines that begin with a period, SMTP uses dot-stuffing to tell them apart from the marker.

The final 250 means the server has accepted responsibility for the message at that stage. It does not prove the mailbox received it or that anyone read it. Once the message is in a queue, the server is expected to handle later delivery attempts under its retry and failure policies.

Implementations also have to track command order and transaction state. Commands sent in the wrong state produce errors, and message-size or recipient limits can cause rejection. These behaviors matter when you investigate malformed SMTP traffic or differences between what a client sent and what a server recorded.

QUIT and connection closure

After the transaction, the client can send QUIT, and the server typically replies 221 and closes the session. A connection can also end because of a timeout, a network failure, or server policy. One connection can carry several transactions, subject to server policy and protocol state, so examine a capture across the full session instead of treating each command as an independent connection.

Packet-level analysis of SMTP with Wireshark

A capture shows connection establishment, commands, server responses, advertised capabilities, TLS negotiation, and the timing of acceptance or rejection. Start by capturing on the relevant network interface. If the client and server share a machine, loopback traffic may need a separate capture. For a remote server, the capture point must see the traffic path, and an unrelated interface will not necessarily show the exchange.

Useful Wireshark display filters:

smtp
tcp.port == 25
tcp.port == 587
tcp.port == 465
tls

The smtp filter matches traffic Wireshark recognizes as SMTP. The port filters isolate conventional SMTP ports, though non-standard ports and encrypted sessions may need extra work. Port 465 carries TLS from the start, so SMTP commands are not normally visible as plaintext. On ports 25 and 587, SMTP content may be visible before a STARTTLS upgrade, and anything after a successful TLS negotiation is encrypted.

Examining the TCP handshake

Check the TCP connection before reading the SMTP exchange. The SYN, SYN-ACK, and ACK packets set it up. Retransmissions, resets, and long gaps point to packet loss, filtering, an unstable service, or timeouts.

A completed handshake says nothing about whether SMTP works. A port can accept connections while the application sends no valid greeting, drops the session, or imposes restrictions that block delivery.

Identifying the SMTP greeting and capabilities

Find the server’s initial 220 greeting and the client’s EHLO or HELO, then compare the advertised extensions with the expected configuration.

A banner may reveal a software name or version, but banners can be suppressed, changed, or wrong, so treat the version as a first clue. A reliable assessment also checks package versions, vendor advisories, distribution-specific patches, and configuration.

Following a complete conversation

Wireshark’s Follow TCP Stream feature reconstructs the readable data in a connection, which makes the command sequence of unencrypted SMTP easier to read than packet-by-packet inspection. Identify EHLO, MAIL FROM, RCPT TO, DATA, and the final server response. Rejected transactions are the most informative because they show where the server enforces policy. A 550 response, for example, may mean the requested action is not permitted or the recipient is unavailable, depending on the command and the enhanced status code.

Multiline responses use the character after the three-digit code to signal continuation: a hyphen usually means another line follows, and a space marks the last one. Misreading that leads to wrong conclusions about which capabilities a server supports.

Understanding STARTTLS in a capture

With STARTTLS, encryption is negotiated inside the SMTP session. The server advertises the extension, the client issues STARTTLS, the server says it is ready, and the TLS handshake follows. From then on, commands and message data are encrypted.

A successful handshake gives you an encrypted transport. It does not show that the client validated the certificate or that the authentication policy is appropriate. Testing should also cover certificate validation, supported TLS versions, cryptographic settings, and whether the application enforces encryption where required.

Opportunistic STARTTLS and mandatory TLS differ. If policy lets mail go out unencrypted when TLS is unavailable, a downgrade or interception scenario can affect confidentiality. Whether that risk is acceptable depends on the deployment and the mechanisms used to enforce transport security.

What packet captures cannot show

A capture is evidence of observed network behavior. Encrypted payloads are not readable without decryption material and suitable capture conditions, and a capture leaves out application logs, queue activity, internal relays, and mailbox delivery events. Correlate it with MTA logs, authentication logs, queue identifiers, DNS results, and message-tracking records, especially when you are investigating delayed mail, suspected spoofing, unauthorized relaying, or a possible compromise.

Building and configuring an SMTP server with Exim

Exim is a widely used MTA for Unix-like systems. It handles SMTP reception, routing, queue management, and delivery through a configurable set of rules and transports. That flexibility suits both small deployments and complex environments, and it also means the configuration needs careful review.

The example below is a basic Exim deployment on Debian-derived Linux. Package names, service names, default paths, and options differ between distributions and releases, so production deployments should follow the documentation for the exact package in use.

Installing Exim

On a compatible Debian-based system:

sudo apt update
sudo apt install exim4

The installation may launch a configuration wizard. Where it is available, you can also run:

sudo dpkg-reconfigure exim4-config

The wizard asks how the server will be used: sending mail through another host, receiving mail directly from the internet, or local delivery only. The right choice depends on whether the machine is a submission server, an outbound relay, an inbound MX host, or a local-only mail service. Define that role first. Do not pick a public-facing configuration just to make SMTP available, because the listening interfaces, accepted domains, relay policy, and delivery configuration all follow from the role.

Reviewing the main configuration decisions

Exim configuration comes down to a handful of decisions.

The mail name sets the system’s mail identity, and the list of local domains determines which recipient domains the server treats as locally deliverable. Neither is the same as the set of domains the server may relay for.

A service that only accepts local application mail may not need to listen on a public interface. A server that receives internet email must be reachable on the right interface and protected by network controls.

The server also has to tell mail for its own domains apart from mail bound for external domains. An inbound server should not let arbitrary unauthenticated clients relay to arbitrary external recipients. Submission services should apply authentication and authorization instead.

Exim uses routing rules to decide how to deliver messages. Receiving mail for a domain publicly normally requires suitable MX records, and outbound delivery depends on resolving destination records and reaching the relevant mail hosts.

Debian’s Exim packaging may support a split configuration spread over several files. That is a maintenance choice, not a security control, so make changes through the supported mechanism for the installed package.

Validating the service

Check the service state:

sudo systemctl status exim4

Review the effective configuration with the utilities the installed version supports. Exim has command-line options for displaying configuration and validating selected settings, so check the local manual before applying changes.

For an initial deployment, verify that the server listens only on the intended interfaces and ports, that local delivery works, and that the configured domains match the deployment’s purpose. If it should receive external mail, verify the public DNS records and connectivity from an authorized external test host.

A service that starts is not necessarily securely configured. Check relay restrictions, authentication, TLS, message-size limits, logging, queue management, and access to administrative functions separately. An accidental relay permission causes serious abuse problems even on fully patched software.

Testing relay restrictions

Test relay policy from a host the server does not trust, using a controlled test domain and mailboxes so nothing goes to unrelated recipients. The question is whether an unauthenticated external client can submit mail for an unrelated external recipient. A properly restricted server rejects the attempt, though the exact response depends on policy and on the stage at which the check runs.

Keep legitimate delivery separate from relay abuse. A public MX server is expected to accept mail for the domains it serves, so accepting a valid local recipient is not evidence of an open relay.

Security weaknesses in SMTP and mail infrastructure

SMTP’s security problems come from the limits of the original protocol, weak authentication and trust, configuration errors, and vulnerabilities in the software that processes messages. The categories overlap, but they call for different fixes.

Open mail relays

An open relay accepts and forwards mail from unauthorized clients to arbitrary external destinations. Attackers use open relays to send spam, phishing, and malicious attachments, and the abuse can get the server’s IP addresses blocklisted and disrupt legitimate mail.

Open relay exposure is a configuration and authorization problem, and it does not necessarily indicate a software vulnerability. Check which clients may relay, whether submission interfaces require authentication, and whether trusted networks are defined narrowly enough to avoid excessive privileges.

Sender spoofing and domain authentication

The envelope sender and the visible message headers do not, on their own, prove that the sender is authorized to use a domain. Attackers exploit that gap to make fraudulent messages look like they come from a trusted organization.

SPF lets a domain owner publish a DNS policy naming the hosts authorized to send its mail. DKIM adds a cryptographic signature as evidence that selected headers and content were signed with a domain-associated key. DMARC lets the owner say how receivers should treat messages that fail authentication and alignment checks, and it provides reporting.

Each covers a different part of the problem. SPF does not establish that the visible From: address is genuine, DKIM does not make a message benign, and DMARC does not replace malware filtering or user authentication. A message that authenticates correctly can still be phishing if an authorized account or sending system has been compromised.

TLS downgrade and interception risks

Without transport encryption, anyone with access to the network path can observe or modify SMTP traffic. That matters most when credentials or sensitive content travel unencrypted.

STARTTLS protects sessions when negotiation and certificate validation work correctly. Opportunistic encryption does not stop every downgrade if the sender will continue without TLS. Mandatory TLS for designated destinations, with suitable certificate and transport-security controls, reduces that risk.

TLS also has limits. It protects the connection between participating endpoints, not the message for its whole lifetime. Messages can sit unencrypted at endpoints or intermediate systems, and TLS does not stop a compromised mail server from reading what it processes.

User enumeration and information disclosure

Recipient-validation responses can reveal whether a mailbox exists. VRFY and EXPN have historically been used for this, though what they return depends on server configuration. Enumeration helps attackers build targeted phishing campaigns or pick accounts for password attacks. Servers can limit disclosure by restricting those commands, normalizing error responses where appropriate, applying rate limits, and monitoring repeated recipient probes.

Not every response reliably confirms an account. Some systems defer validation, accept recipients into a filtering pipeline, or return deliberately ambiguous answers, so an assessment should separate confirmed findings from inconclusive behavior.

Authentication attacks and credential exposure

Submission services may support PLAIN, LOGIN, or other SASL mechanisms, and their security depends on the implementation and configuration. Mechanisms that send credentials in an easily recoverable form must be protected by TLS, and server policy should keep unauthenticated users from gaining submission privileges they do not need.

Password guessing, credential stuffing, and account takeover are risks wherever mail accounts face the internet. Useful controls include multifactor authentication where the client and service architecture support it, rate limiting, strong account-recovery procedures, and monitoring for abnormal submission activity.

SMTP AUTH does not authorize relaying every possible message. After authentication, the server still has to apply policy on permitted sender identities, message volume, and destination behavior.

Resource exhaustion and malformed messages

Mail servers parse commands, headers, bodies, addresses, extensions, and protocol state across many concurrent connections. Weak input validation, memory-management errors, and mishandled message boundaries can cause crashes, resource exhaustion, or code execution.

Queues are a resource problem too. Too many connections, large messages, repeated delivery failures, or abusive traffic can consume memory, storage, CPU, and outbound bandwidth. Limits on message size, recipients per transaction, concurrent sessions, and queue retention contain this, although limits set too tightly interfere with legitimate mail.

These weaknesses are implementation-specific. SMTP being old does not make a given server vulnerable. The evidence comes from the software version, vendor advisories, the deployment’s configuration, and reproducible testing.

Historical case study: Microsoft Exchange ProxyLogon

The 2021 ProxyLogon incident shows a mail platform being compromised through vulnerabilities outside the SMTP command sequence. Exchange Server is a larger application stack with web-facing services, so the security of a deployment depends on more than the SMTP listener.

ProxyLogon is an exploit chain involving several vulnerabilities in on-premises Exchange Server. One of the main flaws, CVE-2021-26855, was a server-side request forgery that could be exploited before authentication under affected conditions. Others, including CVE-2021-27065, allowed arbitrary file writes in the attack chains. Combined with additional weaknesses and conditions, they let attackers gain access to vulnerable servers and, in affected cases, execute code or establish persistent access.

Attackers could read mailbox data and place web shells on compromised systems. A web shell is a server-side script that lets an attacker issue commands through a web-accessible path, and it can keep access alive after the initial exploitation.

On 13 April 2021, the US Department of Justice announced a court-authorized operation to copy and remove certain web shells from hundreds of vulnerable computers running on-premises Exchange Server in the United States. The operation removed identified web shells only. It did not patch the underlying Exchange vulnerabilities or show that affected systems were free of other malicious tools, and the department advised affected organizations to follow remediation guidance and investigate their systems. Source: US Department of Justice

An SMTP server can be configured correctly while another component of the same platform exposes a route to sensitive data or server-side code execution. A review of a mail system therefore has to cover the web application layer, authentication services, management interfaces, mailbox storage, the operating system, and network access controls.

Historical case study: Exim CVE-2010-4344 and CVE-2010-4345

Exim’s history shows a different class of risk: flaws in the MTA itself. CVE-2010-4344 was a heap-based buffer overflow in older Exim versions that could let a remote attacker execute code with the privileges of the Exim daemon under the relevant conditions. It involved message processing and error handling, a reminder that code paths outside the normal successful delivery sequence can be security-critical.

CVE-2010-4345 was a separate local privilege-escalation vulnerability affecting Exim up to and including 4.72. The Exim runtime account could use an alternate configuration file containing directives that led to command execution. The National Vulnerability Database records it as a local privilege-escalation flaw, not an independent unauthenticated remote vulnerability. Source: NVD, CVE-2010-4345

The two are often discussed together because an attacker who gains code execution as the Exim account may then try the escalation path. Their impacts still have to be assessed separately, since remote code execution as a service account does not automatically mean root.

Nmap documents a historical NSE script for CVE-2010-4344 that also checks for the CVE-2010-4345 privilege-escalation issue. The documentation describes the remote heap-overflow risk and the separate local condition, and it classifies the script as intrusive because testing can corrupt or terminate the affected SMTP process. Source: Nmap NSE documentation

These are historical examples, not a description of Exim’s current security state. Use supported software and apply distribution or vendor security updates. A version string may not reflect the patch state of a distribution package, so check version-based findings against the package’s security history.

Reconnaissance and security testing of SMTP

SMTP reconnaissance starts by finding which mail services are exposed, what role each one plays, and whether the configuration matches the organization’s security requirements. The aim is to learn what a service permits, what it discloses, and whether the deployed software has known weaknesses.

Run active tests only against systems you own or are explicitly authorized to assess. Some SMTP enumeration and vulnerability scripts are intrusive and can disrupt service or send unintended mail, so do not run them indiscriminately against third-party infrastructure.

Service discovery and version identification

Nmap can identify reachable TCP services and try to determine their protocols and versions. A conservative first scan of an authorized target:

nmap -sV -p 25,465,587 <authorized-host>

The result shows whether SMTP-related services are reachable and whether Nmap recognizes the software. It does not show that the service accepts mail, allows relaying, or is vulnerable, and firewalls, proxies, non-standard configurations, and banner suppression can all change the output.

If SMTP turns up on an unexpected port, find out whether the exposure is intentional. Remove or restrict unnecessary public listeners, and protect required ones according to their role.

Enumerating supported SMTP commands

Nmap’s smtp-commands NSE script tries to learn which SMTP commands the server supports using EHLO and HELP. On an authorized target:

nmap -p 25 --script smtp-commands <authorized-host>

Compare the output with the expected configuration. AUTH or STARTTLS in the list means the server advertises the extension. It does not mean the implementation is secure or that the extension can be used without the right permissions.

Testing recipient disclosure

The smtp-enum-users NSE script tries to enumerate users with VRFY, EXPN, and RCPT TO. Nmap classes it as intrusive, and the results depend on server behavior, so they may not confirm that a mailbox exists.

In an authorized assessment, use approved test accounts and avoid large-scale probing. The goal is to find out whether the server reveals more account information than normal operation requires. If the results are ambiguous or validation happens later, the finding should say so.

Assessing relay configuration

An open-relay test checks whether a server accepts a message for a destination outside the domains it serves without proper authorization. Nmap has an smtp-open-relay script for this.

Scope the test carefully: use controlled sender and recipient addresses, and avoid delivering to unrelated users or external systems. Accepting mail for a local domain is normal. The question is whether an untrusted client can use the server to relay to an unrelated external domain.

Record the connection context, authentication state, envelope sender, envelope recipient, server response, and any delivery evidence. A response alone may not settle the question if the server validates later or applies downstream controls.

Vulnerability verification

When a version or configuration appears to match a known vulnerability, check the finding against vendor documentation and the exact installed package. Ask whether the vendor backported a fix, whether the vulnerable feature is enabled, and whether the issue is reachable remotely in this deployment.

Historical scripts that actively exercise memory-corruption bugs can kill services or destabilize the host. On production systems, verify versions and patches before any intrusive test. If active validation is needed, use a test environment or an approved maintenance window.

A complete SMTP assessment documents exposed ports, protocol capabilities, TLS behavior, authentication requirements, relay policy, recipient-disclosure behavior, software and patch state, and relevant logs. Separate observed weaknesses from theoretical risks, and include enough evidence for another administrator to reproduce the conclusion safely.

Hardening SMTP infrastructure

Hardening happens at the protocol, application, operating-system, and network layers. No single setting removes every risk, and a configuration suited to an outbound submission server may not suit a public inbound MX server.

First, keep the MTA and its supporting libraries on supported releases and apply security updates promptly. Follow vendor advisories instead of relying on banner-based version detection. Where a distribution backports fixes, confirm the package’s patch status before concluding that a known vulnerability is present.

Second, enforce relay restrictions. Public inbound servers should accept mail for the domains they serve and reject attempts to relay to unrelated external recipients. Authenticated submission services should require user authentication and sender authorization, with limits on message volume and recipient counts.

Third, configure TLS for the service’s role. Use implicit TLS on port 465 where appropriate, or correctly configured STARTTLS on services that support it. Validate certificates on clients, disable obsolete cryptographic configurations in line with current guidance, and use stricter transport policies where confidentiality demands it.

Fourth, limit network exposure. Allow inbound port 25 where the server must receive internet email, restrict administrative access and submission services to the clients and networks that need them, and stop ordinary workstations and application servers from bypassing designated relays unless there is a specific operational reason.

Fifth, reduce information disclosure and abuse opportunities. Review support for VRFY and EXPN, apply sensible rate limits, monitor repeated authentication failures, and avoid exposing unneeded software details. None of this replaces patching, but it limits what attackers can learn and makes abuse easier to spot.

Finally, monitor the mail system as an operational service. Watch for unexpected queue growth, repeated delivery failures, unusual outbound volume, abnormal authentication patterns, unexplained configuration changes, and connections to destinations that do not fit normal use. Keep logs long enough to investigate incidents, and correlate them with authentication, DNS, firewall, endpoint, and mailbox activity where available.

If you suspect a mail server is compromised, patching is not enough. Determine whether messages or credentials were accessed, whether web shells or other persistence mechanisms were installed, whether unauthorized accounts or forwarding rules were created, and whether other systems were reached. Recovery means fixing the initial weakness, removing unauthorized access, rotating credentials where warranted, and verifying that the configuration and logs can be trusted.

Practical exercises

Do these in a lab or against explicitly authorized systems.

Exercise 1: Identify the SMTP service. Scan a lab server on TCP ports 25, 465, and 587 with Nmap service detection. Record which ports are reachable, what service information comes back, and any uncertainty in the version identification. Explain why an open port does not show that the server is a working mail relay.

Exercise 2: Analyze an SMTP transaction. Capture a test message sent to a controlled mailbox and inspect the TCP handshake, greeting, EHLO, envelope commands, DATA exchange, and final response. Find where the server accepts or rejects the transaction, and explain what the final success response does and does not prove about delivery.

Exercise 3: Examine TLS negotiation. If the lab supports both, compare a STARTTLS connection with an implicit-TLS connection on port 465. Identify where TLS begins, and explain why an ordinary capture cannot reveal later SMTP commands in a correctly encrypted session.

Exercise 4: Review relay restrictions. From an untrusted lab client, test whether the server accepts a message for a domain it serves and whether it relays to a controlled external test domain. Record the authorization state and SMTP responses, and explain why accepting local-domain mail is not evidence of an open relay.

Exercise 5: Investigate recipient disclosure. Test recipient-validation commands against a controlled server using approved test accounts. Determine whether the server reveals account information, returns ambiguous results, or defers validation, then recommend changes that reduce disclosure without disrupting legitimate delivery.

Exercise 6: Review an Exim security advisory. Read the NVD record and Nmap documentation for CVE-2010-4344 and CVE-2010-4345. Explain the difference between remote code execution under a service account and local privilege escalation, and why both appearing in a historical exploit chain does not guarantee either outcome.

Exercise 7: Design a hardened deployment. Propose a configuration for a public inbound MX server and a separate authenticated submission service. Define the listening ports, relay policy, TLS requirements, authentication controls, logging, and network restrictions, and explain which controls address protocol-level risks and which protect against vulnerabilities in the MTA or surrounding applications.

Conclusion

Security problems around SMTP fall into three groups: weaknesses in the protocol’s trust model, errors in server configuration, and vulnerabilities in the software that implements mail services. Open relays and sender spoofing need different fixes from memory-corruption flaws or bugs in a web-facing mail application, so an assessment starts by identifying the service’s intended role, observing what it actually does, and checking findings against authoritative technical evidence.

A successful connection or a version banner shows neither that a deployment is secure nor that it is vulnerable. Packet analysis, service discovery, and targeted tests expose unnecessary exposure, weak relay restrictions, and information disclosure, but conclusions still depend on configuration review, patch verification, and, where appropriate, controlled validation. For a mail server, that means a maintained MTA, a restrictive relay policy, enforced transport security, network segmentation, and ongoing monitoring.