Module 4: Email Protection

Scenario Before we look at how to troubleshoot issues, let’s look at how the XG is commonly
configured as an SMTP proxy. As this diagram shows, we have an internal and external network
with a XG is in-between; we also have a mail server, and a couple of computers with mail clients
on the internal network. When a mail client sends an email, the mail client connects to the mail
server, which accepts the email and sends it to the XG. The SMTP proxy on the XG will perform
any outbound checks that are configured, for example data protection checks, then forward the
email to the email server for the recipient's domain. This would typically mean sending out to a
server on the Internet through the WAN port. The XG will accept inbound emails for its list of
configured internal domains. The SMTP proxy will validate the recipient and perform content
checks before delivering the email to the mail server on the internal network that is configured
for the recipient domain. The mail client on the internal computer will collect the email from the
mail server.

How Email Works

Let’s take a look at how Sophos Email works. First, our sender writes an email to
TECHSUPTraining@[Link] and clicks send. The senders mail client will send the email to
their company’s mail server. The first thing the mail server has to do is figure out where the mail
server for [Link] is. To do this it uses DNS. The DNS server responds with the mail
exchanger (MX) record for the recipients domain ([Link]) which is configured to be the
Sophos Central email servers.

Now the mail server knows that to deliver the email to TECHSUPTraining@[Link] it needs
to send it to the Sophos Firewall. When the Firewall receives the email it scans the email for
viruses and spam and checks the sender against the allow and block list. If the email is clean
then the firewall delivers the email to the recipients mail servers, and from there, the recipients
mail client will download the email.

How Does An SMTP Session Work

The interaction between the SMTP client and the SMTP server mentioned in the first step of the
SMTP procedure represents the actual SMTP session. The rest of the SMTP procedure i.e.
forwarding the e-mail via SMTP from the server to the recipient is covered in a separate article.
Each session consists of a sequence of SMTP commands from the client and responses in the
form of status codes from the server.

Overview of SMTP commands

According to the applicable SMTP specifications, each implementation of the network protocol
must support at least the following eight commands, consisting of 7-bit ASCII characters

Forwarding SMTP requests to a SMTP-Relay-Server

Smarthost is an MTA host which allows a mail server to route emails to an intermediate mail
server rather than directly to the recipient's mail server. In a normal scenario, the internal mail
server sends emails to Sophos Firewall which initiates connection to the recipient’s mail server.
But when using a Smarthost, the internal mail server initiates connection to Sophos Firewall,
which as an MTA, opens an SMTP connection to the configured Smarthost and routes emails to
the recipient’s mail server via the Smarthost. In this case, Smarthost server works as an
intermediate emails routing server. It can be configured from Email > General Settings >
Smarthost settings.

It is a 5 steps process:

From the below email logs and header, we can see that: [Link] email server
([Link]) received email from [Link] server ([Link]). [Link]
([Link]) received an email from Sophos Firewall ([Link] [[Link]]). The
email was generated from an internal PC (PC001-PC [[Link]]).

Step -1 :- When an internal user sends an email, the internal mail server establishes a connection
with the recipient’s mail server and performs the SMTP EHLO HELO process, the internal server
will then send emails to the Sophos Firewall.
Step -2 :- When the XG receives these emails, it performs a SMTP policy check before sending
them out.

Step -3 :- The XG then closes the connection with the internal mail server after the policy check.

Step - 4 :- The XG checks whether any Smarthost is configured on global policy. In this scenario,
Smarthost is configured, so the XG sends emails to recipient’s mail server via smarthost

Step -5 :- Lastly, when the XG receives confirmation that the email has successfully been sent to
the recipient’s mail server from the Smarthost server, it will terminate the connection. Recipient
users will think that the email was received from the internal server (which is behind the XG),
but when the email headers are opened, the complete details are shown.

Smarthost configuration is saved under table "tblsmarthostsetting” and

“tblsmarthostsettingrel”

Note: The difference between Smart host server and an Open relay server is, Smart host only
allows authenticated SMTP traffic, whereas Open relay allows anyone to send emails from it.

MTA Mail-flow diagram

Let's take a more detailed look at how the XG processes emails. The XG uses Exim(4.91) for
handling email, which is an open-source Message Transfer Agent (MTA). The SMTP daemon
(SMTPd), starts Exim processes to listen on the following ports:

• 25 : the default SMTP port

• 587 : authenticated message submission

Note: MTA listens on Port 25 (for Plain & STARTTLS ) and Port 587 where Port 25 available by
default for scanning but you need to perform CLI operations to allow 587 i.e. set service-param
SMTP add port 587. We do not support MTA on Port 465.
All emails are accepted into the exim-in queue where the Exim works as front-end MTA, it accept
all emails on port no 25. Exim accepter takes /static/proxy/smtp/[Link] as configuration
[Link] is going to verify email policies like EHLO check, DOS check, SSL check, Relay permission
check, Allow & block list, SPF, BATV, Recipient verification, Grey listing, RBL check and IP
reputation and then it is going to forward to the SMTPD Queue Manager (QMAN).

SMTPD Qman :- SMTPD is the service manager. It invokes all process, assigns work to respective
processes, maintains their configuration state and handles failures (restart them if required)of
child [Link] QMAN sends email to the Work Queue, which has a SCANNER queue and a
Quarantine queue.

Work Queue :- Once in the Work Queue emails are processed by the SCANNER queue, where
the content checks take place, including all of the antivirus and antispam checks. If the email
fails the checks it is sent to the Quarantine queue, which quarantines the email. Mail scanner is
scanner process. SMTPD sends mails.

to scanner for batch policy processing. SMTPD maintains the pool of scanners. on each email
SMTPD picks one scanner. Scanner process loads the policies and apply email scanning policies
like

1) Avira/Sophos Malware scan

2) Cyren spam scan

4) DLP scanning

5) Sophos Sandstorm scanning

6) File extension and TFT filter

7) SPX Encryption

Exim-out (Forwarder) :- Exim also works as backend MTA. SMTPD invokes Exim forwarder with
configuration file /static/proxy/smtp/[Link]. Exim forwarder takes Emails from
/var/spool/output/input and forwards them as per the forwarding [Link] the email
passes all of the content checks it is forwarded to the exim-out queue, where it is then be relayed
to the mail server configured for the recipient domain. This module will send mail to mail server
as per configuration for routing(MX, Static, DNS), smart host and In case if it fails to send mail,
mail will be submitted to queue When the XG is sending outgoing emails, SMTPd forks an Exim
process that acts as an SMTP client, connecting to the destination SMTP server for the recipient
domain to deliver the message.
Log files

The whole process of accepting the email, moving it between queues, and performing the
checks can be tracked in the smtpd_main.log. Emails are tracked using the message ID ---H/-D ,
which changes as the message moves between queues. we are having other logs files as well
i.e., smtpd_panic.log and smtpd_error.log which are used for mails that mailscanner is not able
to be processed.

The exim-in queue performs all of the connection and routing checks for the emails, including
relay, recipient verification and real-time blackhole list (RBL) checks. All of the checks on the
content of the email itself are performed in the SCANNER queue.

Important: Running the ‘service smtpd:debug -ds nosync‘ is used to enable debug and running
same command a second time, will disable debug logging
Mail Logs on the WebAdmin
The above mail logs shows different mail actions such as Dropped, Delivered, Filtered and
Quarantine as per policy. Rejected messages at SMTP transaction will not appear on log viewer
in XG v17.5. In MTA mode, we see that from the GUI --> Email --> Mail logs page but in legacy
mode, it's not possible to see logs from the GUI.

Enabling, disabling and verify the debug mode for the smtpd service

When debugging smtpd, it uses standard XG debug command which is #service smtpd:debug –
d –s nosync and same command is used to disable it.

As you can see above, running the #service –S | grep smptd displays the current service status
of smptd.

The smtpd_main.log allows us to verify if the debug mode was successfully [Link]
Message MSG Apr 16 03:22:25 [ T_SMTPD-W]: Toggling log level to: DBG that debugging is
running.

service smtpd:exim_debug -ds nosync is specifically used if you want to get some more details
for debugging ex. If you want to know about which RBL blocked. The Message MSG Nov 25
14:19:11 [ T_SMTPD-W]: [MT] running exim in debug mode that debugging is running.

Warning: Ensure to disable any DEBUG status on ANY XG service. The longer you retain the
DEBUG status, the more log entries generated in the corresponding log file and increased the
amount of DISK SPACE occupied.

MTA Mode Configuration files

The main 2 directories for SMTP configurations/policy are /static/proxy/smtp/ and
/cfs/proxy/smtp which includes all information for MTA, SMTP, SPX and exim.

# cd /cfs/proxy/smtp/conf/

# ls
[Link] exim_macro.conf [Link] [Link] [Link] [Link] [Link]
exim_fqdn.conf exim_profile [Link] [Link] [Link] [Link]

# cd /static/proxy/smtp/

# ls

[Link] [Link] exim_default.conf [Link] [Link] [Link]

exim_db.sql [Link] [Link]

MTA Mode Database tables

The above database commands give us detail about mail logs and mail spool. We have below
other important database tables as well which can be useful for troubleshooting

List of relations

# psql -U nobody -d corporate -c "\dt "

# psql -U nobody -d corporate -c "\dt "

Log example for a SMTP connection
The below logs are debug SMTP connection logs :- Every email connection will be having 2
unique exim message ID in format ---H/-D for header and data so we need to grep connection
based on that unique ID

# tail -f /log/smtpd_main.log D

BG Nov 14 03:58:09 [ T_SMTPD-M]: ignore temp file '1iVBtB-0004pF-La-H'

Log example for a virus file

The Email proxy determined that the content was having infected malware file which has been
blocked

Note: Current scanning time interval is 2 mins. Every re-scanning time it will pick updated policy,
if it is dual then both will be scanned, if it is single only primary AV will be scanned.

DBG Nov 15 04:25:06 [ T_SMTPD-M]: ignore temp file '1iVYmo-0000QK-NH-H' 2019-11-15

04:25:06.742 [1632] 1iVYmo-0000QK-NH <= jbrown@[Link] H=[Link]
([Link]) [[Link]]:44310 I=[[Link]]:25 P=esmtp S=1329 M8S=8 RT=0.007s
id=[Link]@[Link] T="Virus Email" from
for frogers@[Link] MSG Nov 15 04:25:06 [ T_SMTPD-M]: new mail queued, add to
inqueue '1iVYmo-0000QK-NH-D' MSG Nov 15 04:25:06 [ T_SMTPD-W]: Mail assigned to 'MS-
2920' for scanning '1iVYmo-0000QK-NH-D
Log example for a spam file
The Email proxy determined that the content is a spam so it has been quarantine as per policy
DBG Nov 15 03:28:07 [ T_SMTPD-M]: ignore temp file '1iVXtf-0007Wh-2s-H' 2019-11-15
03:28:07.120 [28935] 1iVXtf-0007Wh-2s <= jbrown@[Link] H=[Link]
([Link]) [[Link]]:44250 I=[[Link]]:25 P=esmtp S=1035 M8S=8 RT=0.021s
id=[Link]@[Link] T="Spam Email" from for
frogers@[Link]

Log example for a MIME filtering

The Email proxy determined that the content is a having MIME filtering so it has been blocked
as per policy DBG Nov 15 04:16:13 [1iVYeB-0000Ab-To]: DOFILTER DBG Nov 15 04:16:13 [ MS-
2920]: do_filter: 1_filter CRT Nov 15 04:16:13 [ MS-2920]: missing filename in this MIME part !

Log example for a DLP detection and SPX message generation

The Email proxy determined that the content was confidential data and has to be encrypted
with SPX. The SPX generation worked without failure. DBG Apr 16 02:46:05 [ MS-3573]:
notify_sender hfile:/sdisk/spool/tmp/IPA1CJ-zRUoG6-T1-H hsize:400
dfiel:/sdisk/spool/tmp/IPA1CJ-zRUoG6-T1-D dsiz:1210 MSG Apr 16 02:46:05 [ MS-3573]: moving
notification '/sdisk/spool/tmp/IPA1CJ-zRUoG6-T1-D' file to forwarder queue.

Log example for Encrypted PDF generated and sent to the recipient
The recipient gets access to the Firewall portal to generate his own password, the URL and
recipient are added. The PDF is generated and saved and attached to the Email actually delivered
to the recipient. DBG Apr 16 02:46:05 [ MS-3573]: POrtal:1,display org:1 DBG Apr 16 02:46:05 [
MS-3573]: URI:1555407965704395c3NwYWRlQGludGVybmV0Lnd3dw
Recipient verification
Recipient verification is as process that identify and validate the Recipient address with mail
server. Recipient verification is required and implemented to address the following risk factors:

• Accepting mail for non-existent recipients wastes CPU time as the message is unnecessarily
scanned for spam.

• Trying to deliver mail to non-existent recipients may cause SFOS or the back-end server to
create Non-Delivery Notifications Receipts (NDR). These useless notifications are called
backscatter and can get the SFOS machine IP address, or your back-end mail server blacklisted.

Recipient verification flow

• Mail sent to recipient is checked by SFOS, by sending VRFY to internal mail server.

• If Recipient address is accepted, server sends 200 OK to SFOS.

• SFOS accept the RCPT TO values and accepts the mail.

• SFOS forwards the mail to destination Email server.

Note: Verification times out in 30 seconds.

Troubleshooting MTA
Is firewall settings correct to support Standard MTA/Transparent MTA?

o Device Access (Standard MTA)? Enable SMTP relay from Device access [Administrator >>
Device access>>Local service ACL] when XG is used as a direct/standard MTA for specific zone.

o Firewall rule (Transparent MTA)?

Add one Email Clients [POP & IMAP] - business application firewall rule via Firewall and
configure underneath settings to use XG as a transparent MTA

- Source: Zone – Any, Networks – Any

- Destination: Zone – Any, Networks – Any

- Malware Scanning: SMTP & SMTPS

- Routing: Rewrite source address (masquerading)

- Click to Save button

To troubleshoot the firewall related issues, use drop-packet capture, tcpdump or Packet capture
utility on UI [Diagnostics>>Packet capture]

• Check configuration of SMTP profile

• In case of receiving an inbound mail, check the protected domain is configured

correctly in SMTP profile

• Checked relay settings?

• Host based relay (for outbound email traffic)?

- Add/Update hosts/networks details in Host based relay via [Email>>Relay

settings>>Host based relay] if mail is rejected due to “550 Relay not permitted”.
 - Check Maillogs for specific rejected mail and hover the mouse over status to
see the reason. It will show “Relay not permitted for [Link]” host.

Upstream host (for inbound email traffic)?

- Add/Update hosts/networks details in Upstream host via [Email>>Relay settings>>Upstream

host] if mail is rejected due to “550 Upstream Relay not permitted”

- Check Maillogs for specific rejected mail and hover the mouse over status to see the reason. It
will show “Upstream Relay not permitted for [Link]” host

Important: As per architecture, Firewall policy doesn’t get applied on inbound mails received
from internet and are expected to deliver to mails server hosted on cloud like O365 and G-suite.
So SNAT policy is not applied for those mails. To apply firewall policy for all traffic, update
disable_offline_relate to ‘no’ in the file /static/proxy/smtp/[Link] and restart the SMTPd
service.disable_offline_relate = no

Scenario for Disable_offline_relate option

XG is being used as an external mail relay for a client on the WAN side. This client is set to send
to the XG. The XG accepts the email fine. Then the XG has a policy to route emails destined for
the client's domain via MX. The mail server is Office 365. Traffic flow is perfect. Only problem is
NAT policy in which if we have configured SNAT policy then it will not work so we require to
enable this option. This option Disable_offline means "firewall rule binding will be enabled for
only outbound if below option is 'yes'" We can enable disable offline relate option from GUI also
from General settings > Advanced SMTP settings > Route inbound mail through gateway and it
is persisted through a firmware upgrade.
• SMTP connection being rejected even with correct relay settings?

• Is that grey-listing causing this?

- When grey listing is enabled in SMTP profile under spam protection section, then the mail is
rejected with “451 Temporary local problem, please try again!” until the sender identified as
known sender.

- Check Maillogs for specific rejected mail and hover the mouse over status to see the reason.
It will show “Temporary Rejection: Sender IP has been Greylisted” for sender host

- Disable grey listing from SMTP profile under spam protection or configure an exception to skip
grey listing check for specific source hosts/domains or sender/recipient mail addresses to accept
the mails. [not advisable]

- Analyze grey listing database

# sqlite3 /sdisk/exim/[Link]

sqlite> select * from tblgreylisted;

sqlite> select * from tblknownsender;

• Is that missing RDNS record causing this?

o When “Reject invalid HELO or missing RDNS” is enabled from the General settings under the
Advanced SMTP settings, the mail can be rejected with “550 Missing RDNS entry.” If no RDNS
available for sender domain.

o Check Maillogs for specific rejected mail and hover the mouse over status to see the reason.
It will show “No RDNS entry for [Link].”

o Disable “Reject invalid HELO or missing RDNS” or configure an exception to skip RDNS/HELO
check for specific source hosts/domains or sender/recipient mail addresses to accept the mails.
[not advisable]

o Check DNS configuration via [Network>>DNS] and configure the valid DNS servers which
resolve the RDNS.

• Is that invalid RDNS record causing this?

o When “Do strict RDNS checks” is enabled from the General settings under the Advanced SMTP
settings, the mail can be rejected with “550 Invalid RDNS entry for [Link].” If resolved RDNS
not matched back to sender domain. o Check Maillogs for specific rejected mail and hover the
mouse over status to see the reason. It will show “Invalid RDNS entry for [Link].”. o Disable
“Do strict RDNS checks” or configure an exception to skip RDNS/HELO check for specific source
hosts/domains or sender/recipient mail addresses to accept the mails. [not advisable] o Is that
SPF fail causing this? - When “Reject based on SPF” is enabled in SMTP profile under spam
protection section, then the mail is rejected with “550 [Link] is not allowed to send mail from
[Link]”. - Check Maillogs for specific rejected mail and hover the mouse over status to see
the reason. It will show “SPF check failed: [Link] is not allowed to send mail from
[Link]” for sender host - Disable “Reject based on SPF” or configure an exception to skip
SPF check for specific source hosts/domains or sender/recipient mail addresses to accept the
mails. [not advisable.

Verifying Email connectivity using telnet

The above sample shows how to use TELNET when performing testing via SFOS without
authentication. When performing telnet command, reminders are:

• TELNET is normally disabled since it’s not a secured protocol. It broadcast transmissions in
clear text

• When performing TELNET command to test email transmission, ensure it’s in the LAN zone
and not coming from outside

• Some customers have their SMTP Server Banner Greeting disabled when performing TELNET,
which result in SMTP Server Banner greeting 220 not be displayed

• Should testing be done with SFOS protecting Microsoft Exchange on-premise servers (2007,
2010, 2013 and 2016), TELNET service is disabled by default. Enabled it before testing.
Verifying Email connectivity using telnet with authentication

When sending email with authentication, it’s crucial that the Base 64 encoded username and
equivalent password is used. This ensures a security layer is added when performing TELNET
testing.

Since EHLO command is invoked, this means that EHLO verbs are used in conjunction with SMTP
extensions.

When performing TELNET test, ensure that the following SMTP codes are received:

220 - This means that the target SMTP Server or domain has its SMTP Service Ready

250 - Means that the Requested Mail Action has completed. Examples of SMTP Extensions used
in this example are:

SIZE = which gives the server the estimated size of the message before its transmitted and for
certain sizes, provide the client with a warning that it will not accept those which exceeds the
size limitation.

PIPELINING = This means that during connection, the client is not required to wait for a response
when sending the next or subsequent request.

AUTH PLAIN LOGIN = which simply means that the AUTHENICATED USER IS USING PLAIN LOGIN

HELP = Likewise self-explanatory and does not changed the EHLP SMTP arguments In the
example entry of AUTH LOGIN, this invokes the use of the PLAIN mechanism which resulted in
having the Server responded with a 334-reply code. What happens next is that first the Base 64
Logon account is entered and followed by the corresponding Base64 encoded password.
Although the keyword PLAIN is used when initiating the AUTH command, the username and
password are NOT SEND IN PLAIN TEXT.

235 simply means that the AUTH LOGIN (Authentication is successful) The 2nd graphic in this
slide is just the normal way of performing TELNET email testing.
MTA accepts emails but emails do not appear on Email Spool

• Have you checked the error queue?

o If SMTPd (scanner) daemon fails to scan any mail, then mail is moved to error queue.

o To check the error queue, go to Mailspool and select Error, click on Filter button to
display mails.

o Admin can only delete/download the mails from Error queue.

• Is number of scanners not sufficient for deployment?

o SMTPd (acceptor) daemon accepts the mail when all security measurements get passed and
stores the mail to spool directory. Use below command to check the mails accepted by SMTPd
(acceptor) daemon.

ls /var/spool/input/work/ o Check no. of SMTPd scanner

#grep vcore -A 2 /cfs/proxy/smtp/conf/[Link]

- SMTPd scanner processes the mail. Each scanner processes one mail at a time. So, when mail
flow is high then it will take some time to display /process all mails. The mail is displayed in
Mailspool when SMTPd scanner picks it for scan.

- The no. of scanner is assigned based on the HW/SW model type/configuration.

- If high number of mails is suspected due to spammer, then enable antispam mechanisms like
inbound/outbound spam scanning, RBL, IP Reputation, grey listing, SPF, Recipient verification,
Invalid HELO or missing RDNS to stop them.

• Have you checked SMTPd’s error?

o Check SMTPd debug log for any abnormal activity or any undesirable error. Use below
command to put the service in debug mode.

#service smtpd:debug -ds nosync

Emails get stuck in the mail spool

• Is Antivirus service (AVD) working?

o When SMTPd scanner is unable to scan mail with antivirus service, the mail status is updated
with a reason “Mail delivery failure. Mail could not be scanned for malware. [Link].”

o Check antivirus service status from UI – [System services >> Services] or from CLI using below
command.

#service antivirus:status -ds nosync

• Check SMTPd service log - smtpd_main.log and antivirus service log - /log/[Link]

• If no antivirus (AV) signature is available in appliance, then mail is updated with failed to scan
due to malware.

• Is recipient address routable? o SMTPd forwarder tries to forward the mail based on routing
type (MX/DNS Host/Static Host Routing). If it fails to deliver the message, it updates the status
with the specific reason. Hover the mouse over the status to see the reason.

• Is Exim DB have cached the recipient server as non-routable? o Exim has internal logic to retry
& deliver the mails to mail server. The number of retry get reduced if mail server is unreachable
for a longer time and mails are continuously received for that domain. Exim may try to send the
mails at once in a day if mail server is unreachable for more time. So, admin can delete Exim db
by below command to deliver all those mails immediately when mail server is available to
receive the mails.

#rm -rf /var/spool/output/db/

defer (-53): retry time not reached for any host in the smtpd_main.log

F,2h,2m; G,16h,1h,1.5; F,3d,6h (config line in [Link])

• Exim will retry to deliver the mail in every 2mins for first 2hrs if mail delivery is failed. Then
after first 2hrs to next 16hrs, it will get retried on every 1h with an expansion factor of 1.5.
 Then after from 18hrs to next 3days, it will get retried on every 6h. If mail is still getting
undelivered then bounce mail gets generated with the message “retry time exceeded”

Note: This behaviour is domain specific not mail specific.

• In XG, Exim’s queue runner is running in interval of 15secs. On every run, it will check all
pending mails and check whether deliver first time header is present or not in mail file. This
header is added only once and gets removed in case of mail delivery failure. So, if deliver first
time header is set, then that mail will get tried by queue runner. If it fails to deliver the mail,
then it will get retried by next queue runner. But due to unreachability of retry time, the queue
runner prints the message “defer (-53): retry time not reached”. So once retry time gets reached
then mail will get retried. If mail delivery gets failed, then it will get requeued and get retried by
next queue runner. If mail is sent successfully then retry time for that domain gets reset.

• One can explicitly reset the retry time db by deleting files located at spool/output/db/retry*.
Afterwards you may want to retry the all-queued mails by command “exim -qff” or restart the
smtpd service. Mails may put on in retry loop if they get undelivered.

Tip:[Link]
retry_configuration.html

Important: If a new Email policy was created on top of the one already matched be the old rule
will also cause this. The Emails in spool are not matching the new rule and retry indefinitely. In
such a case we have move the spooled emails to the new rule.

• Is XG waits for recipient to register for SPX password? o When Mail is SPXed encrypted and
Recipient-specified SPX template is configured, then mail status is updated with SPX blocked. o
Once recipient registers the password, the mail would be SPXed with registered password and
delivered to recipient.

• Is XG waits for sandbox results to come back? o Any files attached in mail found as sandstorm
eligible are sent to sandstorm server for advanced analysis. During this period, the mail is
updated with In-progress status. o To see sandstorm activity, go to [Advanced
threat>>Sandstorm Activity]. o If any error is found during the analysis, then admin must release
manually from sandstorm activity page and mail would be delivered to recipient accordingly.

• Does any temporary rejection message receive from Mail server? o If any server rejects the
message with 4xx error in response, then SMTPd forwarder updates the message with proper
error and one can see that message by hovering mouse over the status of message.
Emails not scanned

• Verify using Packet Capture MONITOR & ANALYZE > Diagnostics > Packet Capture

• Scroll-down to SMTP Settings and verify the value in Don’t Scan Emails Greater Than * & select
Action for Oversized Emails

• Verify relevant scanning SMTP/S,POP3,IMAP scanning is applied or not

• Use TCPDUMP at port 25 to capture SMTP traffic

• Verify X-CTCH headers in /log/smtpd_main.log file or /log/[Link]

Check the smtpd_main.log and /log/[Link] using the advanced shell.

Not able to trigger file/MIME filter for MTA

You can find TFT list published by Labs here : [Link]

1. Check MIME types detected by SMTPd scanner i. search_tft_cache_entry() Entry 'TFT/PCAP-
A' found-> hash '506' ii. search-pattern: application/[Link] iii. search-pattern:
[Link]

2. Check mime type using file command in linux i. # file --mime-type [Link] ii. [Link]:
application/[Link]

3. When File filter matched then [filterd] keyword is appended into Subject header. i.
add_header: 'Subject: [filtered] test image file'.

SASI integration

SFOS Antispam Changes

New to version 18.5 MR3+ and v19.0 EAP 2, the Sophos Firewall is moving away from Cyren to
our in house Sophos Labs Spam engine. The Sophos Firewall is the last Sophos product still
utilizing Cyren services. From the customers perspective and configuration side, nothing is
changed as the changes made are under the hood updates. Previously, the CTIPD service was
used for IP reputation checks and CTASD for anti-spam scanning, which are all parts of the third
party Cyren services. After updating, the SXL and SASI services will take over for reputation/AS
checks. In the event a false positive or false negative needs to be submitted use the following
article:
[Link]
The following may be used for SASI troubleshooting. The primary SASI troubleshooting log is
[Link], which will log any pattern updates and statuses. While the SASI pattern updates and
antispam database files are stored in the /sdisk/sasi directory. Wiki Reference:
[Link]
+SASI+Support+Documentation Managing the antispam engine service uses the standard
service commands. The curl command listed below can be performed to run a basic health check
of the SASI daemon. Below is a snippet of a healthy SASI database.

Sample configuration file for SASI: (The SASI configuration file is located at
/cfs/sasi/[Link]. It is generated by an opcode, which means edits to this file are not
preserved after a reboot.)

This is an example of what the logs will look like during a successful and a failure event. The
above displays the LASEd being updated with the latest version. Where the below output would
be displayed in the event of a failure.
SASI Configuration File

The following is the [Link] log referencing where the updates are being fetched along
with authentication information. This is automatically generated; any edits made here don’t
persist in reboots. Additional configuration options can be found here if required:

[Link]
+-+SASI+Support+Documentation

Troubleshooting Anti-spam technologies

BATV Flow diagram BATV is a mechanism wherein an outgoing Email server adds a tag to the
Envelope From address of all outgoing Emails. For example, if an Email address goes out with
From address as the Envelope From is changed to , where 'SBDGAUJ' is the added tag. This tag
is generated using an internal mechanism and is different for each email sent. If any bounce is
received by the Incoming email servers, they are checked to see if the Bounce address has the
proper tag (in above case 'SBDGAUJ'). If not, the email is rejected. BATV ensures that your Email
users do not become a victim of bounce floods.

1. jsmith@[Link] send a mail to lfox@i_hate.[Link] and hand it off through

[Link] mail server. However, unbeknownst to jsmith, lfox has recently changed her email
address.

2. [Link] mail server signs SMTP MAIL FROM by adding a cryptographic tag.

3. The recipient email server, mail.i_hate_spam.com, sees that the person lfox, does not exist.
4. The mail server accepts the message, but then bounces it back with a null sender and puts the
original, signed, MAIL FROM information into the RCPT TO field.

5. When the message reaches [Link] mail server, it sees that the message is a bounce. It
decrypts the RCPT TO information, which is subsequently verified, so it accepts the message,
and it is delivered straight to inbox.

6. Meanwhile, evil spammer sends a message to lfox at mail.i_hate_spam.net while forging

jsmith@[Link] address.

7. Mail.i_hate_spam.net accepts the message, discovers that it can’t deliver it (because lfox
doesn’t exist there either) and then bounces it back to jsmith since jsmith appear to be the one
who sent the message.

8. When the bounced message hits [Link] email server, the server sees that the message
is an NDR. However, because the RCPT TO field is not signed, and [Link] email server
knows that all genuine outbound mail from customers is signed, the message is rejected.

Verifying BATV in the Mail logs

In this example, we have 2 different bounce back mail logs in which “virus found: BATV Demo1”
got accepted as it is contained a valid signature and BATV DEMO 2 got rejected due to an invalid
signature

Troubleshooting steps :

- 1. Enable debug for smtpd and provide the smtpd_main.logs

#service smtpd:debug -ds nosync

Check smtpd_main.log

2. Email sample of original email (inbound) and where applicable original outbound email
(outbox).

Important: Points to remember

• There will be an exception policy for BATV where admin can except BATV for IPs, senders or
recipients.

• Rejected bounce mail should be logged to mail log

• Default action of BATV will be a REJECT

• Validity of prvs signed mail is 7 days, after this time tags will expire

• These tags are not stored in the database because they are generated with a prvs secret key
and sender address at the time of mail processing.

• Some mail transfer agents may reject a message whose envelope sender address was modified
using BATV. In this case, you need to create an exception rule for the senders, recipients, or
domains affected.

SPF Flow diagram

A SPF (Sender policy framework) record is a public DNS TXT entry that specifies the IP addresses
that send email on behalf of your domain. This is used by receiving mail servers to validate IF
envelope sender IP address is authorized to send mail from the domain for the purposes of email
spoof prevention Sender policy framework (SPF) is an email authentication mechanism that
provides both antispam and antiphishing protection. This is based on RFC 7208 which replaced
the previous release under RFC 4408. As a validation protocol, it utilizes DNS records to identify
legitimate or invalid inbound email. Once enabled, Sophos Firewall will detect and reject SMTP
connections from unauthorized email sender domains. Here’s an example of SPF at a high level:
The administrator of [Link] creates a DNS TXT record specifying emails sent from
[Link] should only originate from IP [Link]. A user of [Link] domain sends an
email. The email will be sent by the [Link] email server whose IP is [Link] SPF works by
looking at the domain of the Return-Path value included in the email’s headers. The receiving
server extracts the domain’s SPF record and then checks if the source email server IP is approved
to send emails for that domain. Once the recipient’s email server receives the email, it takes
email domain [Link] and performs a TXT lookup, looking for the SPF record The
recipient’s email server [Link] then compares the source IP of the email with the one
listed in the SPF record.

The email is then accepted as the source IP of the email matches what is specified by the SPF
record and if the sender IP does not match the SPF record, the email will be dropped.

Reference:

RFC 7208: [Link]

SPF has 2 scenarios when processing inbound email. The first one is when rejecting unauthorized
or illegitimate connections. The second is once the email source IP is verified via DNS + SPF
check, connection is accepted, and inbound email processed. Though SPF-related log events are
stored in ‘/log/smtpd_main.log’, smtpd service must be placed in ‘debug’ mode before SPF log
events are recorded. To enable debug logging to run the command:

‘#service smtpd:debug -ds nosync’.

Note: Running the ‘service smtpd:debug -ds nosync‘ command a second time, will disable debug
logging.

Verifying SPF records with nslookup

To perform lookup and check a valid domain SPF record, run the command ‘nslookup –q=txt ’.
In this example, we find the SPF records for [Link]. The action MTA will perform on specific
SPF results.
Grey listing flow

Overview

• grey listing is an industry technology which temporarily blocks emails from being received by
XG. This is by rejecting the messages for a specific amount of time. The key item here is to group
them by set (Sender email address, sender IP address, Recipient Address and Message Subject)
and block incoming emails for, say, 4 to 15 minutes.

• Data set is compared with the SMTP proxy internal database; if data is not yet recorded, a
corresponding record entry and time stamp is created in the database. This rejects the email for
a period (example: 4 minutes). Once the time expires and email resent again, the message is
accepted.

• The positive note is that it doesn’t use pattern, data analytics or heuristics in analyzing the
email but on certain behavior. An example is if an email with the same message subject is
received by the recipient every 4 minutes. The behavior can be construed as malicious and might
be a possible spam. grey listing ensures the message is blocked for the time being.

• grey listing should only be applied to Inbound emails.

• Client sends Inbound email, and it will be intercepted by SMTPD service. Sender/Host should
be Greylisted/Temporarily rejected for 4 minutes after first attempt. SMTPD service will wait for
same mail with same mail content and MAIL FROM and RCPT TO content after Grey listing
timeout.

• For grey listing, the timer is called at 23:59 daily. So, the entry will be removed after the timer
execution even if 24hr/7days/31days are completed. For example, if the 1st mail is received at
1:00 pm on 13th August and sender is inactive for next 7 seven days then the entry will be
removed at 23:59 on 20th August.

Identifies the following data set:

• Sender Email Address

• Sender IP Address

• Recipient Address

• Message Subject MTA service will accept and check against grey list as per below:

• If Spam exception is matched against inbound email, then email will be checked for
Trusted host and RDNS look up respectively. If host/domain is in trusted list and RDNS
lookup is successful then, MTA will accept the mail and as per profile match it forwards
the mail-to-mailserver as per routing.

• If there is no match in exception list then it verifies for known sender list, if sender
entry is present in database and if this entry is for more than 31 days then MTA service
will remove that entry and perform the above steps again. If 31st day limit is not expired
for that known sender, then it verifies for 7-day limit expiry where MTA service checks
inactivity for that sender. If that sender is inactive for 7 days or more, entry for that host
will be removed. If the sender is active, it updates last access time, and an email will be
accepted. Maximum limit in known hosts table is 1 million.

• If the sender is not Greylisted before then it will be added to Greylisted email. After
being Greylisted, email will be rejected. If host does not retry mail, remove entry after
24 hours.

• Sender/Host should be added to Known Hosts if same mail is retried within 5 minutes
to 24 hours after the first attempt.

Note: One notable instance of using grey listing is if a known, legitimate site had been hacked or
infected and in turn, sending out emails which are either spam or virus-infected. grey listing
effectively blocks the originating MTA and ensure the customer’s domain are protected during
the duration of this issue.
Log example for a reject and accept of grey listing

The SMTPd log example shows us 2 different logs for initial rejections of mail due to grey listing
and then after 4 minutes mail has been accepted

Database

When the recipient server address was Greylisted

#psql -U nobody -d iviewdb -p 5433 -c "select * from tblgreylisted”

#psql -U nobody -d iviewdb -p 5433 -c "select * from tblmaillog where decription ='Temporary
Rejection: Sender IP address has been Greylisted.’”

When XG added sender server IP address to trusted sender list

#psql -U nobody -d iviewdb -p 5433 -c "select * from tblknownsender"

DKIM (Domain Keys Identified Mail) Flow diagram

DomainKeys Identified Mail (DKIM) is a method for E-mail authentication, allowing a person
who receives email to verify that the message actually comes from the domain that it claims to
have come from. The need for this type of authentication arises because spam often has forged
headers and improves the recognition of bogus senders i.e. phishing [Link] example, a spam
message may claim in its "From:" header to be from sender@[Link], when in fact it is not
from that address, and the spammer's goal is only to convince the recipient to click on a link in
the body of the email which leads to some other Web site. Because the email is not actually
from the [Link] domain, the recipient cannot have any effect by complaining to the
system administrator for [Link]. It also becomes difficult for recipients to establish
whether to give good or bad reputations to various domains, and system administrators may
have to deal with complaints about spam that appears to have originated from their systems
but didn't. DKIM adds a header named "DKIM-Signature" that contains a digital signature of the
contents (headers and body) of the mail message. The default parameters for the authentication
mechanism are to use SHA-256 as the cryptographic hash and RSA as the public key encryption
scheme, and encode the encrypted hash using Base64. The receiving SMTP server then uses the
name of the domain from which the mail originated, the string _domainkey, and a selector from
the header to perform a DNS lookup. The returned data includes the domain's public key. The
receiver can then decrypt the hash value in the header field and at the same time recalculate
the hash value for the mail message (headers and body) that was received. If the two values
matches, this cryptographically proves that the mail originated at the purported domain and has
not been tampered with in transit. DomainKeys Identified Mail (DKIM)

DKIM Configuration

Step-1 - Generate signing key (public/private key pair)

The recipient can query DNS to get the public key for the domain and use this to verify to the
hashing and signature of the email to confirm that it was signed be the indicated domain and
that the header has not been tampered with in transit. To configure DKIM you first need to
create the public and private keys. On the UTM you can do this by running the commands:

• To create a 2048 bit private key: openssl genrsa –out [Link] 2048

• To extract the public key: openssl rsa –in [Link] –out [Link] –pubout –form PEM

• To strip unwanted characters from the public key: grep –v –e “^-^ [Link] | tr –d
“\n”

o The public key you will need in the next step is highlighted here. Be careful not to
include the hostname at the end.

Important: Sophos Firewall quarantines DKIM-signed emails that use RSA SHA-1 or have key
length less than 1024 or in excess of 2048 characters.

# openssl genrsa -out [Link] 2048

Generating RSA private key, 2048-bit long modulus

e is 65537 (0x10001)

# openssl rsa -in [Link] -out [Link] -pubout -outform PEM

writing RSA key

# grep -v -e "^-" [Link] | tr -d "\n

Step-2 Publish public key for own domain(s) in DNS

The next step is to create a record in DNS. Each domain key requires a selector that is included
in the email so that the receiving server can look it up. In this example I am using the hostname
of the XG.

• The hostname for the DKIM record should be ._domainkey.[Link].

• The record is a TXT (text) type record.

• Due to the length of the record, in BIND9 you need to split it across several lines using quote
marks.

• The record should start with the DKIM version and the key algorithm.

o There are other flags that you can include in the DNS record, including “t=y;” which
indicates it is for testing and that recipients should ignore your DKIM signature.
Tip: We can also generate DNS record directly from [Link] but
it is not recommended as it is not secure.

Step-3 Configure private key and selector in WebAdmin

To configure DKIM on the UTM, you need to paste the private key into the DKIM section, enter
the selector name for the key in DNS so that recipients can look it up, and choose which domains
you want the UTM to sign using DKIM.

Important: User's uploaded private key for DKIM signing will be stored at following location on
appliance at ls -lah /var/exim/dkim/ S

tep-4 Test your configuration

You can test DKIM using Gmail. In an email, use the menu to view the original message and you
will see a summary at the top that shows whether DKIM validation passed or failed.

Database tables

The command psql -U nobody -d corporate -c "select * from tbldkimsigning” will display the
DKIM database table details and tbldkimverification table will give us information about
different paramters for DKIM verifications.

DKIM signature header sample

Details of DKIM signature header

• v, Version—version of DKIM standard being used

• a, Algorithm—cryptographic algorithm used to create the hash

• c, Canonicalization—whether changes to the email like whitespace or line wrapping is allowed.

• s, Selector—selector to query the correct public key from the d value

• d, Domain—the domain that signed the message.

• i, Identity—the identity of the signer, in email address format

• b, header= is the hash data of the headers listed in the h= tag; this hash is also called the DKIM
signature and encoded in Base64.

• Bh, signature= is the computed hash of the message body. The value is a string of characters
representing the hash determined by the hash algorithm.

Verifying DKIM records with nslookup

In order to lookup and check a valid domain DKIM record, run the command ‘nslookup –q=txt
._domainkey.’. In this example, we find the DKIM records for [Link]. The DKIM header can
be retrieve from the received mail as given in below example :-

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=[Link]; s=20161025;

h=mimeversion:from:date:message-id:subject:to;
bh=XAn7uCqRPomeltBFugglUWPrjx5efSz0nML/heF3OxQ=;
b=s4R9j4LDFlVTpDN9w4fg6P51QSn3Qztb/+CNY+K3UQFT4Zmn9ocX9kcVI1jpUcCWR9N7s7RazH
iPpoM1EGF
Q7ZjtiUnoNRt6QeCELP17qcXXC60m46G5RpBFaq9qrm368bYEzy4yDfATyRwXzQVufAe/J8HzO2a
SUX1I6jY+p/
GHoSc4NQfXS8Dcuih1Dpbg37gwWhap+wRgUjrFgPW4C0D147AcycBtjpAGHSSGj+6CI3WcGiotC
u7d4hW3yn
LlURw6u67+NYa9tEiNFvpVVNJj4aucb/CpKcahymna10YHHhIX5TCIblKEj3qZzxAjNcd1FTnZZrWXe
LzlwHHKEw=

# nslookup -q=txt 20161025._domainkey.[Link]

Domain Name Server# [Link] Domain Name # 20161025._domainkey.[Link] Resolved

Address 1# "k=rsa;
p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAviPGBk4ZB64UfSqWyAicdR7lodhyta
e+EYRQVtKD
hM+1mXjEqRtP/pDT3sBhazkmA48n2k5NJUyMEoO8nc2r6sUA+/Dom5jRBZp6qDKJOwjJ5R/OpH
amlRG+YRJQ qR"
"tqEgSiJWG7h7efGYWmh4URhFM9k9+rmG/CwCgwx7Et+c8OMlngaLl04/bPmfpjdEyLWyNimk7
61CX6KymzYi

To validate SPF entries from an Apple, Linux or Unix shell, we can use dig:

#dig txt ._domainkey.

Verifying DKIM record for the Domain

• To sign outgoing messages with DKIM you have to configure your DNS settings on provider
side.

• Please note without a txt record you cannot use DKIM.

Tip: We can verify same TXT record from any linux based machine by executing command “dig
nameofthekey._domainkey.[Link] TXT +short”

Inbound Verification UI

DKIM verification help to validate inbound emails through the source domain name and
message integrity with Cryptographic authentication that prevents email spoofing. DKIM
Inbound eliminate spam and guarantee message contents from domains using DKIM that you
trust.

Inbound Verification flow

Should DKIM verification fail (e.g., Body hash mismatch with the signature, indicating email
body modification in transit or the signature could not be verified, indicating a forged signature
or a header modification) - The failed DKIM action is applied. An invalid DKIM signature is
detected (e.g. the sending domain’s public key was not in the TXT record or it has an invalid
syntax) - The invalid DKIM action is applied. No DKIM signature is found (e.g. the Email does not
have a DKIM signature for this domain) - The no DKIM signature found action is applied. Other
mail rules are applied

Note: If a single mail is sent to two users an exception policy to skip DKIM verification is created
for one recipient. Should DKIM verification fail for the unexempt user then mail will be rejected
for both the recipients.
Sophos Sandstorm

Detecting Zero-Day Threats using Sandstorm

• Zero-day threat protection

• Supported in MTA(Mail Transfer Agent) mode

• Applicable to Inbound SMTP policy

• Email > Policy > Add policy > Malware Protection

Services and log files

• To verify if Sandstorm service is running, use the above command: service –S | grep
‘mta\|sandbox’ to display the following services:

• smtpd is the SMTP MTA service responsible in allowing XG to absorb, process and
handover the entire email + attachment to either downstream or upstream MTA

• Sandboxd is the main Sandstorm Sandboxing daemon/service

• Sandbox_reportd is the supporting Sandstorm daemon/service responsible in logging

and displaying all the required statistical information of Sandstorm for XG

• We used smtpd_main.log as the main transactional log file when this feature is
enabled in XG. To support Sandstorm, we add two new additional logs namely:

1. Sandbox_reportd.log

2. [Link] As standard, these logs are all located in the /log directory
Log example for Sandbox eligible file detected - smtpd_mail.log

Sandbox eligible file detected smtpd_mail.log example

While Sophos AV returned a „clean“ result, the executable attachment was detected as
Sandstorm „eligible“. The sandboxd service took over, generated an ID for the file and
transferred it to the pending folder on the storage.

Log example for the smtpd service polls the Sandstorm result

The smtpd service polls the Sandstorm result until it returns as either clean or malicious. In this
example, its clean. Afterwards, smtpd service continues with either message delivery or takes
the configured action for malicious AV results.

Note: Virus-infected emails and emails found malicious by Sandstorm can’t be release by SMTP
quarantine.
Log example for the file eligible result in the Anti-Virus log ([Link])

Log Line: 2016-12-14 11:59:52 :[INFO] 3 sophos__scanfile: write eligible log and check excluded
file types for component: EMAIL

Description: Confirms that the file has been identified by the Mail proxy

AwarrenSMTP (Legacy mode)

AwarrenSMTP Packet Flow

Awarrensmtp is module which mainly takes care of email message filtering for Virus, spam
content over SMTP protocol. It includes external communication with mail client, server and
internal communication with Anti Virus Engine (AVIRA) for virus infection, Anti Spam engine
(CTASD & CTIPD) for Spam & Virus Outbreak classification of email message, Logging Daemon
(Garner) for reporting purpose. Awarrensmtp service runs on tcp Port 25
• New TCP connection commence the SMTP transaction since its opening it for an inbound
SMTP mail relay

• When SMTP mail traffic is classified as mark for filtering, it hands it over to AwarrenSMTP
daemon to start the pre-requisite checks

Antivirus scan : Mail will be submitted for Antivirus scanning, clean mail will be forwarded for
Anti-spam verification, infected mails will not be forwarded for anti-spam verification. Mail will
be delivered or dropped as per configuration.

Antispam scan: Virus clean mails will be verified against Anti-spam configuration and forwarded
to proxy for taking decision.

Proxy will act and continue as per result provided Mail Server sends Greeting Message SMTP
Traffic marked for Filtering and Submit to Awarren daemon The objective of the Simple Mail
Transfer Protocol (SMTP) is to transfer mail reliably and efficiently. EHLO message exchange - as
designed in RFC 2821 dictates that

The main identification feature is for ESMTP clients to open a transmission with the command
EHLO (Extended HELLO), rather than HELO (Hello, the original RFC 821 standard). An ESMTP
server would return the code 250 OK in a multi-line reply with its domain and a list of keywords
to indicate supported extensions. A RFC 821 compliant server would return error code 500,
allowing ESMTP clients to try either HELO or QUIT. IP Reputation CTIPD Engine spam
classifications, URL categorization, and malware detection.

Unsupported ESMTP

Sophos Anti Spam detects Spam mails by checking IP addresses, Domain, Email addresses or RBL
(Real-time Blackhole List). Sophos allows administrators to Whitelist or Blacklist Email
Addresses. Emails from Whitelisted Email Addresses are bypassed during Anti Spam scanning
while Emails from Blacklisted Addresses are blocked. This article describes how you can create

White List and Black List and use them for allowing or blocking Emails.

Blacklist: -Create an Address Group called “Blacklist” consisting of Email Addresses whose Emails
are to be blocked.

Starting SMTP connection

Whether we used awarrenSMTP, its recommended to: Set the logging to debug. This is done by
running the command:

AwarrenSMTP = #service awarrensmtp:debug –ds nosync

The logs are populated almost as per below which includes keyword, Log entry and description
:- To start with:

new_smtp_session : This starts the AwarrenSMTP process by showing the new_smtp_session.

This means that a new email is earn mark to the domain being protected by XG
new_smtp_session: (fd:19) [Link]:4027 == SMTP ==> (fd;20) [Link]:25

Server greeting message: Generates the server greeting to indicate that SMTP service that
actually greets the Inbound SMTP request. This is ok but to some organization, they do suppress
SMTP banner since it gives information of what service, product and port are opened
SER < 220 [Link] Microsoft ESMTP MAIL Service ready

EHLO: EHLO or Extended Hello is a part of the ESMTP commands or service [Link]
EHLO command ensure flexibility and is supported by majority of SMTP servers around the
world.

CLI < EHLO

MAIL FROM: This simply specifies the email address or account of the message sender

SER > MAIL FROM: RCPT-TO-SMTP: Is the recipient or the receiver of the email message

SER > RCPT TO:janedoe@[Link]

Command AUTH completed: This signifies that the SMTP Proxy processing the inbound email
have successfully verified and authenticated the recipient’s email address or its UPN Universal
Principal Name. This is normally the email address of the recipient if the environment is using
Microsoft Windows Active Directory

SER < 235 2.7.0 Authentication successful

authorization successful Command 'AUTH‘ completed

DOCTIPDSCAN: The keyword in this SFOS entry is the presence of the CTIPD which is the Cyren
scanning component. Using the handle_ctipd_requests:scan it will check the Source IP Address
of the inbound email message and provide a X-CTCH message header with a corresponding
RefID. This classifies if the email source is CLEAN or MALICIOUS

DOCTIPDSCAN

handle_ctipd_requests:scan:[Link]

handle_ctipd_requests:IN event on ipd_sfd

But is x-ctch-request-id:999

x-ctch-refid:rid=0001.0A150302.529C36D5.0053

ctipd_callback: result for [Link] is CLEAN

For the Mail Size, this depends on the maximum email size configured using the SFOS UI. Take
note that in this example the maximum size of 4096 or 4MB has been exceeded hence the
terminal or final action for the email is dropped. Drop action means that the SMTP connection
is terminated and not part of the email whether it’s a header, body or attachment is saved.
Another item to be clear is that Mail Size settings includes the attachment. Why? Because as per
RFC 2822 and its subsequent RFC that superseded it which is RFC 5322 dictates that the
attachment is folded or included in the email BODY.

ss->mail_size = 5344

mail oversized:exceeds 4096 bytes

CLI > 221 Oversized mail dropped

Trigger the DOVIRUSSCAN action

And this is performed on the Email which is temporarily placed at the /var/tmp/ox20000018
directory. This is done since as an AV engine and its corresponding pattern files cannot scanned
any message until its Header (Mail Header) and Body (Details) are placed as a physical file.
DOVIRUSSCAN processing '/var/tmp/ox200000018'

The next stage here is the keyword, result:clean. In this example, the sample testing resulted in
having CLEAN action successfully removed or verified that the email is free from any malware.
Checking the Log Entry, for this row, it displays: DOFILTER Filter Action triggered to perform
Malware Scanning do_filter:globalsignature which uses the AV signatures downloaded for either
Avira, Sophos or BOTH SCANCONTENT It calls on the Scanning Component enabled. For AV
Scanning it goes to the AV component and you can see that both TFT and DLP (Data Loss
Protection) is set to zero (0) since it’s not triggered for this stage. handle_clean():Scanned
result:clean As stated, email is CLEANED, and no malware is either found or it is successfully
removed. DOFILTER do_filter:globalsignature SCANCONTENT AV:1 TFT:0 DLP:0 handle_clean():
Scanned result:clean BLOCKED is the other way around for this scenario. Instead of having a
CLEAN email or action, the email has detected a malware and should CLEAN Action not
successful, BLOCKED action is triggered to ensure that recipient and organization is protected
from POSSIBLE INFECTION.

DO FILTER

do_filter:globalsignature

SCANCONTENT AV:1 TFT:0 DLP:0

client_writer:written:33 byte to av

scan_file_for_virus():len='19' data=9xea600468, filename='var/tmp/0x20000005b' index ='23'

BLOCKED

DLP or Data Loss Prevention ensures that data leaks of sensitive or confidential organizational
data is applied. This is now a standard for a lot of companies whether they are Government,
Private Corporation or even start-ups. When DLP is ENABLED, SFOS takes its cues from the CCL
or the Content Controlled List whether it’s: the default template downloaded from Sophos Labs
used a Customized CCL which is tailor-fit to the customer’s organization Typical data points
included in a CCL are Social Security Number, Credit Card number that uses the standard either
15 digits(AMEX) or 16 digits (JCB/VISA/MASTERCARD)

SCANCONTENT AV: 0 TFT:0 DLP:1; CCL's matched; DODLP Action; compared_matched_ccl(); if

condition TRUE; ACCEPT

The keywords applied by AwarrenSMTP when DLP Actions are applied are: ACCEPT - The email
is tagged as ACCEPT by DLP since all conditions are verified as TRUE REJECT & DROP shared the
same final action which is QUARANTINE the Outbound email to the Qbin directory. The
difference is that when REJECT action is applied, a notification is generated and send to the
SENDER while DROP simply drop the connection and no notification generated.
Antivirus

• Avira - Maximum Performance (efficient SMTP security processing = less resources used)

• Dual Scanning - Maximum Security (ensure thorough scanning but consume more resources,
hence slower and might affect

• Performance specially during high-traffic mail situations (bulk emails)

Antispam

The topmost action that triggers antispam scanning is the Anti-Spam Rule matching which
verifies the ACTUAL SENDER & RECIPIENT EMAIL ADDRESS. Once the top-level action is
triggered, this resulted in a systematic scan flow that is performed from LEFT to RIGHT.

• Antispam scanning is performed by CTASD at port 8080 if its INBOUND and port 8089 if its
OUTBOUND. The idea behind separating the ports is to ensure that mail traffic are processed
accordingly in the most economical and efficient way.
• Once email is identified as either INBOUND or OUTBOUND, the SOURCE IP ADDRESS is
matched against the IP GROUP.

• Furthermore, the IP Address (if its INBOUND) is checked against the RBLs & Trusted IP list. The
purpose is to ensure that IP address are both:

o Legitimate and not SPOOFED

o And the Sender IP address reputation is CLEAN and NOT LISTED IN ANY RBLs (Examples
are [Link], [Link], [Link], [Link] and [Link])

• Email Size is also checked to ensure that it follows the Maximum Mail Size configured by SFOS.
This feature based on the experience where DDOS or DOS-based email attacks often have huge
file attachments and eventually occupies the entire FREE DISK SPACE of a MAIL SERVER. This will
shut down the Mail Server and prevent the organization from sending or receiving emails.

• MIME Headers are also checked if they are completed. Header information such as SUBJECT,
FROM, TO & OTHERS are important part of the email envelope. Having one or many of these
often resulted in having SPOOFED or SPAM emails coming through the organization.

• For OUTBOUND emails, there is special emphasis in implementing DLP or Data Leak Protection
which ensures CONFIDENTIAL and SENSITIVE information ARE NOT DISCLOSED and DISTRIBUTED
to the PUBLIC. Information such as Social Security Number, Passport Number, 15, 16 & 19 digit
Credit Card Numbers are just a few of these IMPORTANT DATA that are being addressed by
having using CCL template or a Customized CCL configured with SFOS.

• Finally SPX Action is applied with a RANDOM GENERATED PASSWORD to ensure that if an email
message is QUARANTINE but needs to be retrieved, additional security layers are applied. For
the Spam Classification, as per Cyren we normally used either of the following above;

• SPAM → If the email is classified as Spam due to the number of hits in terms of keywords
present in the email.

• PROBABLE SPAM → This is tagged since the email might be a SPAM but needs verification

• VIRUS OUTBREAK →The email contains a virus or other of form of MALWARE that is spreading
through email transmission

• PROBABLE VIRUS OUTBREAK → Email containing virus or other forms of malware are spreading
but since its hasn’t reach critical proportion, its tagged PROBABLE VIRUS OUTBREAK

POSSIBLE SMTP ACTIONS are TERMINAL ACTIONS performed by SFOS to emails coming via
SMTP. Terminal actions means that should it find the email message and/or attachment
MALICIOUS, it applied BEST MATCH FILTER ACTIONS which result in having the email processed
and SKIP other scanning ACTIONS.

The possible SMTP actions are:

1. REJECT: This ACTION not only let SFOS to refuse the INBOUND or OUTBOUND email from
being relayed TO and FROM but also generated a NOTIFICATION indicating REJECT ACTION
APPLIED. For Internal users, this is fine but for INBOUND emails, its BEST PRACTICE TO use DROP
instead due to security reasons.
2. DROP: Drop is a preferred action specially for INBOUND emails since it provides additional
security by not generating a notification. However some organization specially those prone to
FALSE POSITIVES would request for REJECT instead of DROP but that is on a case-to-case basis.

3. ACCEPTS: The email is tagged as CLEAN, and the resulting ACTION is ACCEPT it and hand it
over for mail queuing to either DOWNSTREAM or UPSTREAM MTA for delivery

4. PREFIX CHANGE: Is by modifying the SUBJECT PREFIX base on the scan result

5. CHANGE RECIPIENT: This is normally performed on emails that are either INFECTED but
contains important information that is required by the organization or for those SENSITIVE
INFORMATION being transmitted OUTBOUND and have to be re-routed.

Service and log files

Awarrensmtp service is responsible in processing emails that passes through the SMTP protocol
and is confirmed running by issuing the command:

#service awarrensmtp:status -d –s nosync

If the Firewall or Systems Administrator received a 200 OK or 200 RUNNING, this means that the
awarrrenSMTP service or daemon is running correctly

• In legacy mode, if there are more than 20 recipients in an email and emails are getting
delivered to the first 20 recipients however to the rest of the recipients the emails are not getting
delivered or if you want to retry email sending in legacy mode then we need to enable Fast ISP
mode.(console> set service-param SMTP fast-isp-mode on )

• Fast ISP mode approach is firmware upgrade safe.

• In legacy mode, If grey listing is enabled on the server side, which results mails are getting
rejected. This is because legacy mode doesn't support e-mail retry. If e-mail fails to be sent,
legacy mode proxy generates a notification and inform the sender. So, as per grey listing, failed
e-mail should retried but this is rejected with this log entry: "451 Temporary local problem,
please try again!"
• If mail contains multiple recipients, policy rule which match first will apply to all recipient. Ex.
rule1 for yahoo with spam enable rule2 for gmail with spam disable if mail contains yahoo and
gmail domain recipients, rule 1 will match first so, spam scan apply to both recipients.

• SPX will work partially. Only sender generated and onetime generate password methods will
work with fast isp mode.

The command switch format is the same as any other SFOS daemons and processes. So for the
following like:

To enable debug, we issue the command: # service awarrensmtp:debug -d -s nosync This

command is broken into the following:

• Service to commence the command line issue that a process or daemon is being tasked to
perform a specific function

• awarrensmtp is the name of the Legacy SMTP process or daemon • debug is the task being
asked to the daemon to perform. For this example, it sets the debug level from normal to
verbose logging. This provides detailed logging for the purpose of isolation, audit and debugging
purpose.

Important: Ensure to only set the daemon to debug if [Link] the SFOS is running in DEBUG
mode and no troubleshooting is required, ensure to use the same command to disable it. -d -s
nosync is used if the customer is using a single Firewall device. If they are using SFOS on HA (High
Availability) either Active/Passive or Active/Active, removed the -nosync since it stands for No
Synchronization. Synchronization is required if customer deployed [Link], stop and restart are
used when troubleshooting or ensuring that the service is running.

You can place awarrenSMTP to advance debug as well by below command :-

#ps | grep awarrensmtp

Result will get PID of awarrensmtp.

Then issue "kill -40 " command, so awarrensmtp service will run in to debug mode.

You can verify same thing from awarrensmtp logs

# grep Toggling /log/[Link]

MESSAGE Nov 25 23:33:08 [4152280512]: Toggling log level to: MAX

Awarrensmtp communicates with Inbound / Outbound CTASD engine over TCP Port 8088 &
8089 using HTTP Protocol respectively for Spam & Virus Outbreak classification of email
message. CTASD daemon first looks in its cache file further communicates with CYREN data
center if needed

# netstat -anep | grep 25

tcp 0 0 :::25 :::* LISTEN 0 3663209 15152/awarrenSMTP

# netstat -anep | grep 8088

tcp 0 0 [Link]:8088 [Link]:* LISTEN 0 4014182 19706/[Link]

# netstat -anep | grep 8089 tcp 0 0 [Link]:8089 [Link]:* LISTEN 0 3998061 19704/[Link]
Virus attachment is not detected

• Verify if AV scanning is applied on firewall rule where email traffic passes (SMTP & SMTPS)

• Verify if the default Anti-Spam rule for “Virus Outbreak” is configured properly with
SMTP/SMTPS and POP3/IMAP actions.

• Verify the email header "X-Sophos-AV-Policy" to ensure that the mail has been scanned by
the antivirus engine

• Verify if the antivirus rules are configured properly for “Infected” and “Suspicious” emails.

• Verify the "X-Sophos-smtpxv-version" header to ensure that the latest signature pattern
versions are used

• Verify “Don’t scan email greater than” configuration, In PROTECT > Email > General settings,
check whether a limit on the size of email to be scanned is configured, and what action is applied
if that size is exceeded.

We would recommend setting the size to ‘0’, which will scan files up to 50MB, and the action
for oversized emails to ‘Reject’. This will ensure that all emails are scanned for viruses or
rejected. Addition of Sophos SMTP Proxy Header for version information INFO Dec 02 12:59:26
[3076545424]: add header: ‘X-Sophos-smtpxy-version: [Link]’ Email Signature addition if any

Block File Types filter

INFO Dec 02 12:59:26 [0x20000018]: DOFILTER DEBUG Dec 02 12:59:26 [3076545424]: do filter:
7_filter Adding one more header for Sophos SMTP AV Policy / Rule INFO Dec 02 12:59:26
[3076545424]: add_header: ‘X-Sophos-AV-Policy: default' Tip: If attachment is not being
blocked, check the MIME type of the file which is not being scanned (check
POP-IMAP Proxy

• The warren service handles the email antivirus and anti-spam filtering received using
POP/IMAP protocol

• This includes communication with the client application, server, AVIRA antivirus engine,
Commtouch Advanced Security Daemon [CTASD], Commtouch IP Reputation Daemon [CTIPD]
and firewall (ipset). The warren service utilizes TCP Ports 109 for POP3 and 142 for IMAP4 traffic.

• Moreover, warren sends event logs of virus / spam scanning results to the garner logging
daemon for reporting & troubleshooting purposes.

• Sophos Firewall automatically applies the default POP-IMAP scan policy (default-pop-av) to
POP3/S and IMAP/S traffic which strips virus-infected attachments from inbound emails and
replaces the message body with a notification message.

Verifying the warren service and communication

Warren service command: Status, debug, stop, start, restart

# service warren: status –d –s nosync

200 RUNNING

Warren Listens on TCP port 109,143 for POP & IMAP respectively :

# netstat -anep | grep warren | grep LISTEN

tcp 0 0 [Link]:109 [Link]:* LISTEN 0 56382 4717/warren

tcp 0 0 [Link]:142 [Link]:* LISTEN 0 56383 4717/warren

Log file location: /log/[Link]

Quarantine

File location

XG provides only the last 7 days of quarantine mails. Despite the messages stored already in the
system, appliance reporting should be enabled to get the quarantine mail listing. In Sophos
Firewall Quarantine, emails will stay at quarantine area until 90% of quarantine area is filled up.
After 90%, Sophos Firewall checks the quarantine space on each new quarantine mail and it that
exceeds quarantine limit, it remove older quarantine mails. Quarantine Digest is an email that
contains a list of quarantined messages filtered by Sophos Firewall and stored inside the user
Quarantine Area. Once configured, Quarantine Digest is generated and send based on the
configured frequency to the user. The Digest provides a link to User My Account where user can
access their quarantined messages and take necessary actions.

Run via shell:

# timer all:summary | grep digest

Spam_digest_mail REPEATJOB [0 6 10***] Thu Jun 02 10:06:00 2019

# timer spam_digest_mail:status
“name”:“spam_digest_mail”,“status”:“Enable”,“type”:”repeatjob”, “exec_time”: “Thu Jun 02
10:06:00 2019”, “exec_str”: “opcode_spam_digest_thread text (null)”

Troubleshooting the Quarantine digest

No Quarantine digest is generated/sent

• Check Quarantine digest [Email>>Quarantine digest] is configured and is enabled for users.

• Make sure user’s email address is not added in Skip quarantine reports in [Email>>Quarantine
digest].

• Check Mails are quarantined for that user for configured email frequency. Only quarantined
spam mails are included in Quarantine Digest report.

• Check Notification settings [Administrator>>Notification settings] are configured properly.

• Check timer for spam digest mail to send quarantine digest report.

• timer spam_digest_mail:status

• Test release of Quarantine digest report manually using below command and check [Link]
& smtpd_main.log. #opcode spam_digest_thread -ds nosync

• check mailSize for specific user in [Link]. It should not be zero.

• Check Mailspool & Maillogs for Quarantine mail release.