A need to protect SMTP traffic is not uncommon. In general, you can’t always protect it. Inside your organization, it’s pretty easy – you can easily implement digital signatures or encryption for emails, but if you want to go outside, things are becoming more complicated. You can still use digital signing (S/MIME) on emails, but if your certificate is issued by your internal PKI, it probably won’t be trusted on recipient side. That will not prevent functionality of digital signature, but will affect trust.
If you want to send encrypted emails outside your organization, things are even more complicated. If you want to use just built-in Outlook features for message encryption, you will need to have public key of any recipient that you want to send encrypted message to. Inside your organization, this is not an issue as you can publish certificates in AD DS. However, outside, on the Internet this is not an easy job. Sure, there are some third party tools for this, but let’s see what we can do without them.
Domain Security is a feature of Exchange Server (both 2010 and 2013) that can secure SMTP traffic between two Exchange organizations. It is implemented on server level, and it works without configuring any options on user (sender or recipient) side. Domain Security uses mutual TLS authentication to provide session-based authentication and encryption. Mutual TLS authentication is different from TLS as it’s usually implemented. Usually, when you implement TLS, client will verify the server certificate, and authenticate the server, before establishing a connection.
With mutual TLS authentication, each server verifies the connection with the other server by validating a certificate that’s provided by that other server, so clients are not included at all. We establish secure SMTP channel between two Exchange Servers, usually over the Internet.
Clients, Outlook and Outlook Web App, will be aware that Domain Security is established. Green icon with check mark will be shown on each messages exchanged between servers on which Domain Security is implemented.
As you can see, Domain Security can be applied between two (or more) known Exchange organizations. Still, it can’t protect whole SMTP traffic that comes and goes from your Exchange organization, but it can efficiently protect SMTP traffic between partner organizations.
Let’s see how to configure it. I’ll show the procedure for Exchange Server 2013, but most of it can be applied to Exchange Server 2010 also. Let’s assume that we want to establish Domain Security between two Exchange organizations named adatum.com and treyresearch.net (Yes, I’m using domain names from Microsoft Learning courses, but since I write these courses, I just get used to that 
)
1. Establish certificate trust between organizations
As said before, Domain Security relies on certificates. Because of this, you should first establish certificate trust between two organizations where you want to implement Domain Security. You can do it on several ways. If both organizations are using publicly trusted certificate on Exchange servers, you are good to go. If that’s not the case you will have to cross-import Root CA certificates on both sides. Alternatively, you can also issue certificates for SMTP for both Exchange organization from a single trusted RootCA. Anyway, the point is that each Exchange server must trust the certificate installed (and assigned to SMTP service) on another Exchange server. Achieve this in any way you like. Besides establishing trust, make sure that certificate common name is same as the name that Exchange server provides in HELO/EHLO conversation.
However, it’s important to notice one thing here. In Exchange Server 2010, you would be doing this on Edge Transport server or if you didn’t deploy one, on Hub Transport server. Since these two roles are no more in Exchange 2013, these certificates should be installed on CAS servers which, in Exchange Server 2013, host FrontEnd Transport Service. Also, it is important that certificate you want to use for Domain Security is assigned to SMTP (it can be assigned to other services as well)
2. Configure Domain Security
As both sides/companies will be sending and receiving emails, following procedure should be done on both sides, but domain names should be used vice-versa.
First, open Exchange Management Shell and execute this cmdlet :
Get-TransportConfig | FL
You will get whole list of transport settings but we want two of them : TLSReceiveDomainSecureList and TLSSendDomainSecureList. If you were not configuring Domain Security so far, you will have these two values empty. To use Domain Security we must populate these parameters with appropriate values.
TLSReceiveDomainSecureList – specifies the domains from which you want to receive domain secured email by using mutual Transport Layer Security (TLS) authentication
TLSSendDomainSecureList – specifies the domains from which you want to send domain secured email by using mutual TLS authentication
If we are on adatum.com side, we will execute following:
Set-TransportConfig -TLSSendDomainSecureList adatum.com and
Set-TransportConfig –TLSReceiveDomainSecureList treyresereach.net
After this, when you run the Get cmdlet again you should have these values:
Logically, on treyresearch side we will issue same commands but domains will be inverted. If you have more than one company with requirement for Domain Security, you can provide their domain names too. This cmdlet accepts multivalued parameters.
3. Configure connectors
Now, we will create dedicated connectors for Domain Security. Let’s first create the send connector. You can use Exchange Admin center for this. Navigate to mail flow, click Send connectors and add new one. In a wizard, type the name of the connector and select Partner type. Don’t use the smart host, but leave MX record as a method to send mail. For connector address space, type the domain name from other Exchange organization. If you are on Adatum.com side, you will type treyresearch.net. On the source server page of wizard, select Mailbox server that you want to use as a source. It doesn’t matter which one you will choose since we will configure connector to proxy through CAS. When you create the connector, double click it and then enable options “Proxy through client access server”. You can also configure maximum message size for this connector if you want, and enable protocol logging.
Now back to Exchange Management Shell,and execute :
Get-SendConnector –identity ConnectorName | FL
Look for the value of parameter DomainSecureEnabled. It should be True. If it’s not you can easily set it with Set-SendConnector –identity ConnectorName –DomainSecureEnabled:$true
Let’s now configure Receive connector. Back to EAC, click mail flow and then click receive connectors. In Select server drop-down list choose your Client Access server.
Select Partner for connector type, configure receiving IP address if you want (or just leave all available) but on remote network settings page, you should configure only the IP address assigned to another organization Exchange server. This should be (public) IP from which partner’s Exchange server sends email. After you create the connector double click it, and click on security tab. Make sure that authentication options are set like on following screenshot.
4. Test the Domain Security
Easiest way to test this is to just send email from one organization to another from Outlook. If you get the message with green check mark, you are all set. If not, then you’ll need some troubleshooting. You can enable protocol logging by executing :
Set-ReceiveConnector Internet -ProtocolLoggingLevel Verbose, and
Set-SendConnector Internet -ProtocolLoggingLevel Verbose
to verify TLS channel. If message doesn’t arrive to recipient but doesn’t come back as NDR, you should check queue.
Or you can just wait for my next blog post, where I will discuss some troubleshooting .
