# SendGrid <> Proofpoint blocked IP resolution
### Background / Next Steps
[ProofPoint](https://www.proofpoint.com/us) is an e-mail security / screening service that is blocking our e-mail traffic from SendGrid at the IP level [see here](https://ipcheck.proofpoint.com/?ip=159.183.139.153). In order to resolve, we will need to work with each blocked STT to submit a support ticket / whitelist request for our IP to ProofPoint.

This hack.md will outline the STTs we need to work with and planned guidance to provide them in order to resolve.

#### Current Steps:
* [Opened one last support ticket w/ SendGrid](https://support.sendgrid.com/hc/en-us/requests)
* If unsuccessful w/ support case, engage STTs below and provide guidance to whitelist our IP.
* 10.16.23 Update:
* Identified additional IP associated to our SendGrid account: 159.183.208.143
* Confirmed, free of ProofPoint block
* Next Steps: Test deliverability from new IP to ProofPoint STTs, discuss options forward for: maintain two separate IPs, merge to new IP (verify microsoft is playing nice as well) etc.

### STTs blocked by ProofPoint

**[x]** indicates that recipients in these jurisdiction received emails after the IP was switched on 10/27/2023.

- [ ] Montana
- [x] Louisiana
- [x] California
- [x] Hawaii
- [x] Rhode Island
- [x] Missouri
- [x] Mass
- [x] Manilla
- [x] dhs.ga.gov

### Original ProofPoint IP look up / error message:
https://ipcheck.proofpoint.com/?ip=159.183.139.153

554 Blocked - see https://ipcheck.proofpoint.com/?ip=159.183.139.153

### DMARC errors following switch to new non-blocked IP:
DMARC (Domain-based Message Authentication Reporting and Conformance) exists as a third instruction set for how a receiving email server should handle an email that fails SPF and DKIM validation.

So far in testing we've encountered blocked emails due to a few variants of DMARC errors from different STTs.

- Cook Inlet Tribal Council (citci.org) with ```451 4.7.5 Temporary error evaluating DMARC policy```
- Montana (mt.gov) with ```451 4.7.5 Temporary error evaluating DMARC policy```
- Kentucky (ky.gov) with ```440 4.4.0 DMARC Error From Sender talk to your email admin```
- Menominee Indian Tribe of Wisconsin with ```451 4.7.5 Temporary error evaluating DMARC policy```
- Tanana Chiefs with ``` 440 4.4.0 Local Policy Violation, try again later...```

To solve for this issue, SendGrid support has recommended that we implement a DMARC record to supplement setup we've already done for rDNS, DKIM, and SPF. This means initiating one more DNS change to add a simple DMARC record along the lines of this example:

More SendGrid documentation on [implementing DMARC](https://docs.sendgrid.com/ui/sending-email/how-to-implement-dmarc) and [what DMARC is](https://docs.sendgrid.com/ui/sending-email/dmarc/).

---




[ProofPoint customer success center (for paid customer support)](https://proofpoint.my.site.com/support/PPSup_CommunitiesCustomLogin?startURL=%2Fsupport%2Fcts_pdr_lookup%3FretURL%3D%252Fsupport%252F500%252Fo)
### Support Ticket and E-mail guidance
#### Support Ticket
Information requested by ProofPoint to validate whitelist request:
1. Recipient of blocked e-mail: provide domains blocked in response
2. Type of e-mails: provide commentary on expected communciations and workflow from our IP address

#### E-mail Guidance

Greetings [insert STT],

We are writing to inform you that TDP is unable to deliver e-mail communications to your domain [insert STT domain], due to an e-mail security service your office utilizes. The service, ProofPoint, is blocking e-mail traffic from our IP address and you will need to submit a support ticket in order to receive communications from TDP, we have outlined applicable steps below to accomplish this.

1. Navigate to ProofPoint's [customer success center](https://proofpoint.my.site.com/support/PPSup_CommunitiesCustomLogin?startURL=%2Fsupport%2Fcts_pdr_lookup%3FretURL%3D%252Fsupport%252F500%252Fo) and submit an IP Address whitelist support ticket.

If you're unable to log into ProofPoint's customer success center, please follow the following steps in addition to engaging your IT team to identify who has access.

1. Navigate to ProofPoint's dynamic IP address lookup address and enter TDP's IP: 159.183.139.153. ProofPoint will acknowledge they're blocking traffic from our IP and you can submit a ticket to resolve.


#### Chat between Alex and ACF/HHS email team on 11/1/2023

> ACF: some other systems have been having issues and we're trying to track them down. so the weird part is that our DMARC usually shows up as "passed" and valid" but that "temporary  error. The HHS email team is opening up a ticket with Microsoft to determine what the error is. can you provide me with the sendgrid public IPs wher ethe mail could be coming from

> Alex: 159.183.208.143
>
> Alex: also sent the domains with DMARC errors
> 
> ACF: HHS is investigating. I will let you know what they find
> 
> Alex: are the other systems having this issue using SendGrid?
> 
> ACF: No. we think its mostly on the receiver side. MS is tagging things in a strange way sometimes. so its setting other email security tools off. proofpoint is a security tool... maybe their using as a spam filter? problem is when its tagged in a strange way it lowers the trust score and mail tools that are set really high will flag it as spam. But HHS email team can confirm after more investigation. 

### Communication in 11/16/2023 with HHS Email Ops Team

> Thanks for reaching out to the email ops team. We have an open SEV regarding this issue (SEV 3 - Service Alert - Ticket 903879). 

> At present, we believe this may be related to an ongoing DNS distributed denial of service attack (DDOS) against HHS, which is preventing DNS resolvers from completing the lookups required for SPF and DMARC authentication, resulting in intermittent failures and rejections. We are seeing an outsized number of failures from recipient domains using Exchange Online in particular, and are working with our DNS engineers to as well as Microsoft’s to resolve the issue.

> Do you have any recent bounce/rejection messages or error codes from your SendGrid delivery failures that you could provide? Failing that, the From address used by your application would be helpful in our analysis.

Alex response:

> Here are the error messages we are receiving, by jurisdiction. In parentheses is the email domain that blocked delivery of the message. I’ve also included some screenshots from sendgrid. 
We are sending emails from our system TDP which is located at tanfdata.acf.hhs.gov. the sender listed in the emails is  [e-mail]. 
•	Cook Inlet Tribal Council (citci.org) with 451 4.7.5 Temporary error evaluating DMARC policy
•	Montana (mt.gov) with 451 4.7.5 Temporary error evaluating DMARC policy
•	Kentucky (ky.gov) with 440 4.4.0 DMARC Error From Sender talk to your email admin
•	Menominee Indian Tribe of Wisconsin with 451 4.7.5 Temporary error evaluating DMARC policy