IPsec AH vs ESP: Which One Should You Choose for CISSP?

IPsec AH (Authentication Header) provides data integrity and origin authentication but lacks encryption. ESP (Encapsulating Security Payload) provides integrity, authentication, and confidentiality through encryption. For CISSP, remember that AH is rarely used today because ESP can perform AH's functions while also securing the payload's privacy.

What is the fundamental difference between AH and ESP?

When you're diving into Domain 4 of the CISSP, you'll encounter IPsec as the gold standard for network-layer security. The core difference comes down to one word: confidentiality. Authentication Header (AH) is designed for integrity and authenticity. It ensures that the packet hasn't been tampered with and that it actually came from the claimed sender. However, AH does not encrypt the data. If a hacker intercepts an AH packet, they can read every bit of the payload in plain text.

Encapsulating Security Payload (ESP), on the other hand, is the heavy lifter. It provides everything AH does—integrity and authentication—but adds the critical layer of encryption. This means ESP ensures confidentiality, making it impossible for unauthorized parties to read the data. In a real-world scenario, if you are protecting sensitive corporate data over the public internet, ESP is your only viable choice. We always tell our students: if the exam question mentions 'privacy' or 'confidentiality,' stop looking at AH and start looking at ESP.

Why is AH often considered limited in modern networks?

You might wonder why we even study AH if ESP is so much better. The answer lies in how AH handles the IP header. AH signs the entire packet, including the outer IP header. While this sounds secure, it creates a massive problem with Network Address Translation (NAT). Because NAT modifies the IP address in the header to route traffic between private and public networks, it changes the data that AH has signed. This causes the integrity check to fail, and the receiving device drops the packet.

This incompatibility makes AH practically useless for most modern VPNs and cloud environments where NAT is ubiquitous. In contrast, ESP does not protect the outer IP header in the same way, allowing it to pass through NAT gateways without breaking the connection. When you're practicing with our 1,000 expert-curated CISSP questions at Cert Sensei, look for scenarios involving NAT—they are almost always a hint that AH is the wrong answer.

How does ESP provide superior security capabilities?

ESP is the versatile tool of the IPsec suite. It doesn't just encrypt the payload; it can be configured to provide authentication and integrity as well. This is often referred to as 'ESP with authentication.' By using a Hashed Message Authentication Code (HMAC), ESP ensures that the encrypted data hasn't been altered during transit. This dual-capability effectively renders AH redundant for most security architectures.

From a CISSP perspective, you need to understand that ESP protects the payload but, by default, leaves the original IP header exposed (unless used in Tunnel Mode). This allows routers to still move the packet efficiently while the actual data remains a secret. If you're struggling to visualize this, think of AH as a signed envelope where the letter inside is visible, and ESP as a locked, armored safe that is also signed and sealed.

What is the difference between Transport and Tunnel Mode?

Regardless of whether you use AH or ESP, you have to choose a mode of operation. Transport Mode is used for host-to-host communication. In this mode, only the payload of the IP packet is encrypted or authenticated; the original IP header remains intact. This is efficient but reveals the source and destination identities to anyone sniffing the wire. It's typically used for internal traffic between two servers in the same data center.

Tunnel Mode is what you'll use for Site-to-Site VPNs. In this mode, the entire original IP packet (header and payload) is encrypted and then wrapped inside a brand new IP header. This hides the internal network topology from the public internet, providing an extra layer of security. If you see a question about 'gateway-to-gateway' security, Tunnel Mode is your answer. We recommend using our domain-level analytics to track your performance in this specific area, as network security is a frequent stumbling block for many candidates.

How do you decide between AH and ESP for the CISSP exam?

When you're staring at a multiple-choice question, use a simple elimination process. First, ask: 'Is confidentiality required?' If yes, eliminate AH immediately. Second, ask: 'Is the traffic passing through a NAT device?' If yes, eliminate AH. In 95% of modern exam scenarios, ESP is the correct choice because it provides a complete security triad (Confidentiality, Integrity, and Availability).

However, don't ignore AH entirely. You might see a legacy scenario where only integrity is required and overhead must be kept to an absolute minimum. In those rare cases, AH is the answer. The key to passing the CISSP isn't just memorizing definitions; it's understanding the 'why' behind the technology. That's why we provide detailed expert reasoning for every single answer on our platform, helping you move from rote memorization to true architectural understanding.

How does the IPsec handshake facilitate these choices?

Before AH or ESP can even start moving data, the two devices must agree on how to do it. This happens via the Internet Key Exchange (IKE) protocol. IKE operates in two phases. Phase 1 establishes a secure, authenticated channel (the ISAKMP SA), and Phase 2 negotiates the actual IPsec Security Associations (SAs). This is where the devices decide: 'Are we using ESP or AH? Transport or Tunnel mode? AES or 3DES?'

Understanding this handshake is crucial because it shows that IPsec is a framework, not a single protocol. The SAs act as the 'contract' between the two peers. If the SAs don't match—for example, if one side expects ESP and the other sends AH—the connection will fail. Mastering these nuances is what separates a passing score from a failing one. Spend time drilling these concepts with practice exams to ensure you can recognize these patterns under the pressure of the actual exam clock.

❓ Frequently Asked Questions