## :memo: Bounce and FBL processing for EMA

Author: @z3d3m0n
Status: Draft
Date: 07.01.2022

#
# Abstract
This document describes implementing the Bounce and Feedback loop processing of the PowerMTA and EMA

- The Scheme of how all works out

EMA->MTA: Sends an email 
Note right of MTA: MTA writes a log
MTA-->Logs: bounce.csv & fbl.csv
Note left of MTA: MTA responds via Json 
MTA->EMA: Posts bounce and feedback info  via API

*note:*
*MTA - Mail transfer agent
EMA - email marketing*

:rocket:

the process is folowing:
the EMA sends an email via MTA and MTA writes 3 type of logs ,
* account.csv information on the delivry
* bounce.csv information on the bounces ( the error messages)
* feedback.csv (information on the abuses)

the main objective is to process the bounce.csv and feedack.csv retreaving the message id and bounce reason and passing it via API back to the EMA

**MTA Bounce field mapped record Description**
* **type** bounceRecord[0] type - always b
* **timeQueued** bounceRecord[1] Time message was queued to disk
* **bounceCat** bounceRecord[2] likely category of the bounce (see Section 1.5), following the recipient which it refers
* **vmta** bounceRecord[3] VirtualMTA selected for this message, if any
* **orig** bounceRecord[4] originator (from MAIL FROM:)
* **rcpt** bounceRecord[5] recipient (RCPT TO:) being reported
* **srcMta** bounceRecord[6] source from which the message was received. the MTA name (from the HELO/EHLO command) for messages received through SMTP
* **dlvSourceIp** bounceRecord[7] local IP address PowerMTA used for delivery
* **jobId** bounceRecord[8] job ID for the message, if any
* **dsnStatus** bounceRecord[9] DSN status for the recipient to which it refers
* **dsnMta** bounceRecord[10] DSN remote MTA for the recipient to which it refers
* **dsnDiag** bounceRecord[11] DSN diagnostic string for the recpient to which it refers

An example xontents of the `bounce.csv`:

type,timeQueued,bounceCat,vmta,orig,rcpt,srcMta,dlvSourceIp,jobId,dsnStatus,dsnMta,dsnDiag,header_Message-Id


b,2022-01-06 15:28:58+0100,bad-mailbox,mta1,[e-mail],[e-mail],[127.0.0.1] (5.196.190.67),178.33.203.119,,5.0.0 (undefined status),us-smtp-inbound-2.mimecast.com (205.139.110.221),smtp;550 Invalid Recipient - https://community.mimecast.com/docs/DOC-1369#550 [jseBMzviPLisH4ECmPEBjA.us374],<[e-mail]>

**Bounce category description**

$bounceCategories: An array of bounce-category strings. This is only used when processing MTA accounting files. The default is $bounceCategories = array("bad-mailbox","bad-domain","routing-errors","inactive-mailbox");. If the array is empty, all records are processed as bounces.

$softbounceCategories: An array of soft-bounce-category strings. This is used to classify bounces in EMA and other campaign tools and is used to determine if a bounce is a soft- or hard-bounce.. The default is $softbounceCategories = array("bad-configuration", "bad-connection", "content-related", "invalid-sender", "other", "policy-related", "quota-issues", "spam-related", "virus-related");. If the array is empty, all bounces are classified as hard-bounces.

**prepared arrays for the setup**

// Handle the following bounce-categories only
// Leave empty to handle all bounce-categories
$bounceCategories = array("bad-mailbox","bad-domain","routing-errors","inactive-mailbox","spam-related", "policy-related");

// Specify soft-bounce-categories. This is used to classify bounces in MailWizz as a soft-bounce. If not in the list or if the
// list is empty, all bounces are considered hard-bounces
$softbounceCategories = array("bad-configuration", "bad-connection", "content-related", "invalid-sender", "other", "policy-related", "quota-issues", "spam-related", "virus-related");

# Bounce Processing

**Stand alone processing**

 cat bounce.csv | /usr/bin/php ./bounce/bounce.php

**Automatic processing**

 <acct-file |/usr/bin/php /opt/pmta/bounce/bounce.php>
    records b
    record-fields b timeQueued,bounceCat,vmta,orig,rcpt,srcMta,dlvSourceIp,jobId,dsnStatus,dsnMta,dsnDiag,header_Message-Id
</acct-file>

**MTA Fbl field mapped record Description**

In addition to the standard FBL fields, we write out the following fields:

* header_X-HmXmrOriginalRecipient: Outlook places the original recipient into a customer header, we map this back to rcpt
* header_subject: We log the subject as it allows to identify the nature of the emil
* header_BatchId: Typically the x-job-id
* header_Message-Id: The unique message-id
* header_List-Unsubscribe: The unsubscribe link - this might be useful to just post a HTTP-GET from Port25
* header_List-Id: The List-Id - most mailers such as use the standard header
* header_X-Mw-Subscriber-Uid: The Subscriber-UID.
* header_X-Mailer-RecptId: The Subscriber-UID provided by Interspire
Some FBL providers (such as OpenSRS/ReturnPath) will anonymise the recipient-email address and in those cases we need to use the Subscriber-ID/List-ID to unsubscribe the user.

# FBL Processing

 <acct-file /var/log/pmta/fbl.csv>
    records feedback-loop
    map-header-to-field f header_X-HmXmrOriginalRecipient rcpt  # hotmail recipient
    record-fields f *, header_subject, header_BatchId, header_Message-Id, header_List-Unsubscribe, header_List-Id, header_X-Mw-Subscriber-Uid, header_X-Mailer-LID, header_X-Mailer-RecptId
</acct-file>

<acct-file |/usr/bin/php /opt/pmta/bounce/bounce.php --logfile=/var/log/pmta/fbl-processor.log>
    records feedback-loop
    map-header-to-field f header_X-HmXmrOriginalRecipient rcpt  # hotmail recipient
    record-fields f *, header_subject, header_BatchId, header_Message-Id, header_List-Unsubscribe, header_List-Id, header_X-Mw-Subscriber-Uid, header_X-Mailer-LID, header_X-Mailer-RecptId
</acct-file>

# API Parameters
#

| Variable | Description |
| -------- | -------- |
| $message_id | Message's id |
| $type | One of 4 types: sent , bounced , reported , failed |
| $bounce_type | Required if type is bounced soft or hard|
| $report_type | Required if type is reported - spam , failed, abuse |
| $description | Notification message |

Request:

{
   curl -X POST -H "accept:application/json" -G \
http://0.0.0.0/api/v1/notification \
-d api_token=dM8Ps4R0f7hdkX6qUsyOJ9956tBmBks0WSpCEjWLw1WbTwNfpJdDSKmIAEkl  \
-d message_id=[e-mail]  \
-d type=bounced \
-d bounce_type=hard \
-d description=Email+address+does+not+exist
}

to sum up:

we open the file bounce.csv

take these fileds fileds from`bounce.csv`:

bounceCat
dsnDiag
header_Message-Id


bad-mailbox
smtp;550 Invalid Recipient - https://community.mimecast.com/docs/DOC-1369#550
[e-mail]

determine if bounceCat is a soft or hard bounce via arra()

a push it back via api

**inspiration**
https://github.com/magicdude4eva/port25-bouncehandler
https://github.com/postmastery/pmtabounce/blob/master/cron.php