CSE 4215
Network Security
IPsec
IPsec (Internet Protocol Security)
▪ IP Security (IPsec) is a collection of protocols designed by Internet Engineering Task Force
(IETF) to provide security for a packet at the network level.
▪ Provides security for a packet at the network or IP level
▪ Provides
✔ Authentication
✔ Integrity
✔ Confidentiality
✔ Key management
▪ Applicable to use over LANs, across public & private WANs, & for the Internet
1/29/2026 2
The Need of IPsec
▪ To guard against serious security incidents involving-
IP spoofing
o intruders creating packets with false address then taking advantages of OS exploits
Eavesdropping and Sniffing
o attackers listen for user ids and passwords and then just walk into target systems
▪ To enhance the security of the programs (HTTPS, PGP, S/MIME etc.) that use the security
services at the application and transport layers.
▪ To provide the security of the programs that do not use the security services at the
application and transport layers.
▪ To provide security for node-to-node communication programs such as routing protocol.
1/29/2026 3
IP Security Scenario
1/29/2026 4
▪ IPsec works in two modes-
Two Modes of IPsec
Transport Mode
o IPsec protects what is delivered from the transport layer to the network layer.
o IPsec in transport mode doesn’t protect the IP header.
Transport Layer Transport Layer Payload
IPsec Layer IPsec hdr IPsec trlr
Network Layer IP hdr IP Payload
Tunnel Mode
o IPsec protects the entire IP packet. It takes an IP packet, including the header, applies IPsec security methods to
the entire packet, and then adds a new IP header.
o Tunnel mode is used when the sender or receiver is not a host.
Network Layer IP hdr IP Payload
IPsec Layer IPsec hdr IPsec trlr
Network Layer New IP hdr New IP Payload
1/29/2026 5
Two Security Protocols of IPsec
▪ IPsec defines two protocols-
Authentication Header (AH)
Encapsulating Security Payload (ESP)
1/29/2026 6
Authentication Header (AH)
▪ It is designed to authenticate the source host and to ensure the integrity of the payload carried in the IP
packet.
▪ It uses a hash function and a symmetric key to create a message digest. The digest is inserted in the AH.
▪ The AH is then placed on the location based on the mode (Transport or Tunnel).
IP hdr AH Transport Layer Payload Padding
8 bits 8 bits 16 bits
Next Header Payload Length Reserved
Security Parameter Index (SPI)
Sequence Number
Authentication Data (Digest)
(Variable Length)
Fig. : Fields and positions of AH in transport mode 1/29/2026 7
Authentication Header (AH) (Cont’d)
▪ The addition of an AH follows these steps-
1. An AH is added to the payload with the authentication data field set to 0.
2. Padding may be added to make the total length a multiple of some number of bytes.
3. Hashing is based on the total packet excluding those fields of the IP header that are
changed during transmission.
4. The authentication data (Message Digest) is inserted in the AH.
5. The IP header is added after changing the value of the protocol field of the IP header
to 51.
1/29/2026 8
Fields of AH
▪ Next Header (8 bits): Identifies the type of data contained in the payload data field (such
as TCP, UDP etc.)
▪ Payload Length (8 bits): It does not define the length of the payload. It defines the length
of the AH.
▪ Security Parameters Index (32 bits): Identifies a security association.
▪ Sequence Number (32 bits): A monotonically increasing counter value which provides an
ordering information for a sequence of datagrams.
▪ Authentication Data (Variable Length): The result of applying a hash function to the
entire packet excluding those fields of the IP header that are changed during transmission.
1/29/2026 9
AH
AH provides source authenticity and integrity, but not confidentiality.
1/29/2026 10
Encapsulating Security Payload (ESP)
▪ It is designed to provide source authentication, integrity, and confidentiality.
▪ ESP adds a header and trailer.
▪ ESP’s authentication data is added at the end of the packet.
Authenticated
Encrypted
IP Header ESP Header The Rest of the Payload ESP Trailer Authentication Data (variable
length)
32 bits 32 bits
Security Parameter Index (SPI) 8 bits 8 bits
Padding
Sequence Number Pad Length Next Header
Fig. : Locations of ESP header and trailer in transport mode 1/29/2026 11
ESP (Cont’d)
Fig. : ESP packet format
1/29/2026 12
ESP (Cont’d)
▪ The addition of an ESP follows these steps-
1. An ESP trailer is added to the payload.
2. The payload and the trailer are encrypted.
3. The ESP header is added.
4. The ESP header, ESP trailer, and payload are used to create the authentication data.
5. The authentication data is added to the end of the ESP trailer.
6. The IP header is added after changing the value of the protocol field of the IP header
to 50.
1/29/2026 13
Fields of ESP
▪ Security Parameters Index (32 bits): Identifies a security association.
▪ Sequence Number (32 bits): A monotonically increasing counter value; this provides an anti-replay
function, as discussed for AH.
▪ Payload Data (variable): This is a transport-level segment (transport mode) or IP packet (tunnel
mode) that is protected by encryption.
▪ Padding (0–255 bytes): The purpose of this field is-
✔ If an encryption algorithm requires the plaintext to be a multiple of some number of bytes (e.g., the multiple of
a single block for a block cipher), the Padding field is used to expand the plaintext (consisting of the Payload
Data, Padding, Pad Length, and Next Header fields) to the required length.
✔ Additional padding may be added to provide partial traffic-flow confidentiality by concealing the actual length
of the payload.
1/29/2026 14
Fields of ESP (Cont’d)
▪ Pad Length (8 bits): Indicates the number of pad bytes immediately preceding this field.
▪ Next Header (8 bits): Identifies the type of data contained in the payload data field.
▪ Authentication Data/Integrity Check Value (variable): A variable-length field (must be an integral
number of 32-bit words) that contains the Integrity Check Value computed over the ESP packet minus
the Authentication Data field.
1/29/2026 15
ESP in Two modes
1/29/2026 16
IPsec Services
Services AH ESP
Access Control Yes Yes
Message Integrity Yes Yes
Entity Authentication Yes Yes
Confidentiality No Yes
Replay Attack Protection Yes Yes
1/29/2026 17
IPsec Services (Cont’d)
▪ Access Control: When a packet arrives at a destination, and there is no security association already
established for this packet, the packet is discarded.
▪ Message Integrity: A digest of data is created and sent by the sender to be checked by the receiver.
▪ Entity Authentication: The security association and the keyed-hash digest of the data sent by the
sender authenticate the sender of the data.
▪ Confidentiality: The encryption of the message in ESP provides confidentiality.
▪ Replay Attack Protection/Anti-Replay Service: A replay attack is one in which an attacker obtains
a copy of an authenticated packet and later transmits it to the intended destination. The Sequence
Number field is designed to thwart such attacks.
1/29/2026 18
Anti-Replay Service
▪
1/29/2026 19
Anti-Replay Service (Cont’d)
At Receiver-End:
▪ The receiver should implement a window of size W, with a default of W = 64.
▪ The right edge of the window represents the highest sequence number, N, so far received for a valid
packet. For any packet with a sequence number in the range from N - W + 1 to N that has been
correctly received (i.e., properly authenticated), the corresponding slot in the window is marked.
1/29/2026 20
Anti-Replay Service (Cont’d)
At Receiver-End:
▪ Inbound processing proceeds as follows when a packet is received:
✔ If the received packet falls within the window and is new, the MAC is checked. If the packet is authenticated,
the corresponding slot in the window is marked.
✔ If the received packet is to the right of the window and is new, the MAC is checked. If the packet is
authenticated, the window is advanced so that this sequence number is the right edge of the window, and the
corresponding slot in the window is marked.
✔ If the received packet is to the left of the window or if authentication fails, the packet is discarded; this is an
auditable event.
1/29/2026 21
The Question
Why do we need AH?
1/29/2026 22
Security Association (SA)
▪ A security association is a contract between two parties.
▪ It creates a secure channel between them.
▪ It is a very important aspect of IPsec.
Fig. Simple SA (Unidirectional Communication)
1/29/2026 23
Internet Key Exchange (IKE)
▪ IKE (Internet Key Exchange) is a protocol used in IPsec to:
▪ ✔ Authenticate communicating parties
▪ ✔ Securely exchange cryptographic keys
▪ ✔ Establish security associations (SAs)
▪ It automates the setup of secure IPsec communication.
▪ IKE is a complex protocol based on three other protocols:
✔ Oakley
✔ SKEME
✔ ISAKMP
1/29/2026 24
Why IKE is Needed
IPsec requires:
▪ Secret keys for encryption (ESP)
▪ Secret keys for authentication (AH)
▪ Security parameters for communication
▪ IKE securely negotiates all of these over an insecure network.
Main Functions of IKE
▪ Mutual Authentication
▪ Confirms identities of sender and receiver
▪ Key ExchangeGenerates shared secret keys (using Diffie-Hellman)
▪ Security Association (SA) Establishment
▪ Defines: Encryption algorithms, Authentication algorithms, Key lifetimes 1/29/2026 25
IKE PHASES (TRADITIONAL IKEV1)
IKE works in two main phases:
Phase 1: Establish IKE SA (Secure Channel)
Used to create a secure authenticated tunnel for further negotiation.
Steps:
▪ Exchange proposals (encryption, hash, DH group)
▪ Perform Diffie-Hellman key exchange
▪ Authenticate each other using: Pre-shared key (PSK), orDigital certificates, or Public-key encryption
▪ Generate a secure IKE SA
1/29/2026 26
IKE PHASES (TRADITIONAL IKEV1)
Phase 2: Establish IPsec Sas
Used to negotiate the actual IPsec security parameters so that Secure data transfer can begin
Steps:
▪ Agree on:
• AH or ESP
• Encryption and authentication methods
• Keys for data transfer
▪ Create IPsec SAs for data communication
1/29/2026 27
Why IKE is Needed
IKE Phases (Traditional IKEv1): IKE works in two main phases:
Phase 1: Establish IKE SA (Secure Channel)
Used to create a secure authenticated tunnel for further negotiation.
Steps:
▪ Exchange proposals (encryption, hash, DH group)
▪ Perform Diffie-Hellman key exchange
▪ Authenticate each other using: Pre-shared key (PSK), orDigital certificates, or Public-key encryption
▪ Generate a secure IKE SA
1/29/2026 28
Why IKE is Needed
IKE Phases (Traditional IKEv1): IKE works in two main phases:
Phase 1: Establish IKE SA (Secure Channel)
Used to create a secure authenticated tunnel for further negotiation.
Steps:
▪ Exchange proposals (encryption, hash, DH group)
▪ Perform Diffie-Hellman key exchange
▪ Authenticate each other using: Pre-shared key (PSK), orDigital certificates, or Public-key encryption
▪ Generate a secure IKE SA
1/29/2026 29
Reference
▪ Slide Courtesy:
Mohiuddin Ahmed
Lecturer,
Department of CSE, RUET
1/29/2026 30
Thank
You
1/29/2026 31