IPsec (as discussed in Part 3) provides a very high level of security based on the exchange of encryption keys and the validation of the IP header information. NAT, on the other hand, which we discussed in part 5, is not primarily a security mechanism but it does provide a degree of address isolation by hiding the internal IP addresses of broadcast equipment from external networks.

For many applications requiring access to public networks such as the internet, NAT is commonly used to translate private internal addresses into publicly routable addresses. This conflicts with how IPsec operates because portions of the packet used for authentication and integrity checking may be modified during NAT translation.

NAT works by replacing the private source IP address and source port number of an outgoing packet with a public IP address and a dynamically allocated port number. The NAT router maintains a translation table so that return packets can be mapped back to the original internal device.

This approach works well for normal TCP and UDP traffic because the transport headers remain visible to the router. However, problems arise when encryption protects parts of the packet that NAT would normally modify, as is the case with IPsec.

Solving Address Validation

There are several methods for operating VPN connections across NAT environments. Some VPN technologies, such as SSL/TLS-based VPNs, avoid many of the compatibility problems associated with IPsec by operating over standard TCP or UDP sessions that can pass more easily through NAT devices and firewalls. However, IPsec remains widely used because it provides strong network-layer security and transparent encryption for IP traffic.

There are many more NAT-Traversal (NAT-T) solutions and each has their own benefits. The IPsec solution maintains the method of encapsulating security payload, but it does require compliance with multiple protocols that must be enabled to move through the firewalls and comply with network translators. The protocols that address the compatibility issues between IPsec and NAT are highlighted in RFC 3715 (IPsec Network Address Translation Compatibility Requirements), and the solutions are addressed in:

• RFC3947 – Negotiation of NAT-Traversal in IKE (Internet Key Exchange)
• RFC3948 – UDP Encapsulation of IPsec ESP (Encapsulating Security Payload) Packets
• RFC5996 – IKE (Internet Key Exchange) Protocol

Although there is no dedicated RFC (Request-for-Comments) document for IPsec NATT, RFC3947 describes the process of detection and key negotiation.

The essence of achieving VPN over NAT-T consists of two parts. During phase 1, the two IKE peers determine whether NAT devices exist along the path between them and whether NAT-Traversal must be enabled. Then phase 2 initiates UDP encapsulation of the IPsec packets so that NAT devices can correctly translate the traffic while preserving the encrypted payload.

Establishing IKE Connection

Phase 1 is started by the initiator issuing a specific vendor ID payload to the receiver, to advertise support for various features within RFC3947. These messages consist of defined strings that are “hashed” to provide 16-byte representations of the specific vendor ID string. Upon receipt, the receiver will send their own vendor ID payloads back to the initiator to verify compliance. In doing so the two end routers, also known as hosts or IKE peers, establish an intention to exchange data between each other.

The next process is for the initiator to determine if any NAT devices exist between the two IKE peer protocols. Each IKE peer may or may not exist in the NAT router so we cannot assume it has any knowledge of a NAT. To verify this, the initiator generates an NATD payload using hashes derived from the source IP address and UDP port information associated with the IKE session.

Upon receipt of the NAT-D message, the IKE peer locally recomputes the hash using the received source IP address and UDP port information. If they are the same then no NAT took place (otherwise, they would be different). And importantly, the receiver now knows both the original internal address/port combination and the translated public address/port combination.

Keeping Address Records

Having the original IP address is important as the receiver can use this to validate the authentication header and encrypted segment of the VPN packet. This is needed as the authentication and encryption calculations would otherwise be based on the device’s original source IP address and port number. The NAT device itself has no awareness of the encrypted IPsec payload and only modifies the outer transport headers required for address translation.

Furthermore, having the source IP and UDP port addresses after the NAT allows the sender to populate the destination IP address and UDP port number when sending data back to the initiator. When the NAT router associated with the IKE peer’s initiator receives the packet, it will have a copy of them in its lookup table and change the destination IP address and UDP port numbers back to those of the original device so that the device can perform and validate its authentication checks and decryption. Once the NAT-D negotiation has been completed and the respective IKE peers can exchange IPsec (VPN) packets, phase 2 takes over through UDP encapsulation of the IPsec ESP packets. Once NAT-Traversal has been negotiated, these packets are commonly encapsulated using UDP port 4500 so that NAT devices can correctly track and translate the traffic flow.

Assuming at least one NAT process exists within the route, the outer IP address and UDP port header are likely to be changed. However, each IKE peer now has the critical information regarding the original source IP address and UDP/TCP port number, allowing it to correctly decrypt and authenticate the protected portion of the message while maintaining high levels of security through the NAT translation.

Legacy IPsec security protocols like the Authentication Header (AH), which was developed to ensure network packets are from a trusted source and remain unaltered in transit, are particularly incompatible with NAT because AH protects portions of the IP header itself. Any address modification performed by NAT invalidates the authentication calculation. ESP operating in UDP encapsulation mode is therefore normally used for NAT-Traversal deployments.

Operational Considerations

Although NAT-T solves the compatibility problem between IPsec and NAT, it also introduces additional operational considerations for broadcast systems. Unlike traditional enterprise data applications, live video and audio streams are often highly sensitive to latency variation, packet reordering, and transient packet loss.

One consequence of NAT-T is the addition of another encapsulation layer through the UDP wrapper. Although the overhead is relatively small, the additional packet headers increase bandwidth usage and may contribute to fragmentation if packet sizes are not carefully controlled. This is especially important for high-bitrate contribution feeds and remote production workflows operating close to the maximum transmission unit (MTU) size of the network.

Firewall configuration can also become more complex. NAT-T commonly uses UDP port 4500, while the initial IKE negotiation frequently uses UDP port 500. Broadcast engineers must therefore ensure that intermediate firewalls and security appliances correctly permit and maintain these sessions. Aggressive timeout policies or state-table limitations within firewalls may cause unexpected VPN drops during long-duration live broadcasts.

Troubleshooting NAT-T connections can also be difficult because the encrypted ESP payload prevents deep inspection of the media traffic itself. Conventional packet analysis tools may only reveal the outer UDP encapsulation, making it harder to identify whether latency, jitter, or packet loss is occurring within the encrypted media flow or elsewhere in the network path.

For remote production and cloud-based broadcasting workflows, NAT-T provides a practical method of securely transporting media across untrusted networks, but careful attention must still be paid to latency budgets, MTU sizing, firewall behavior, and long duration session stability.

Maintaining Connections

A NAT keepalive mechanism is also used so that intermediate NAT devices retain their address translation state. This typically consists of periodically sending small UDP keepalive packets between the two IKE peers. Without these keepalives, the NAT device may remove the translation entry after a timeout period, preventing subsequent encrypted packets from reaching the correct destination. If the public address or UDP port mapping changes, the IKE session may need to be renegotiated.

Although NAT-T provides a solution to the interaction of IPsec (VPN) and NAT translation, it does so at the cost of latency and care must be taken when using it, especially if IP addresses and port numbers change. For normal transactional internet traffic this isn’t usually an issue. For latency-sensitive video and audio streaming applications, on the other hand, it can be a challenge.