IP security is a collection of protocols designed by IETF(Internet Engineering Task Force) to provide security for a packet at network layer. IPsec help to create authenticated and confidential packet for IP layer:

  1. Transport layer : In this mode, IPsec protects what is delivered from transport layer to network layer(i.e data). In other words, it protects the payload.
    Note that transport layer dose not protect the header but only payload. The transport mode is normally used when we need host to host communication protection of data.
  2. Tunnel mode : IPsec protects the entire packet. It takes an IP packet including the header applies IPsec methods to the entire packet then adds a new IP header and the new. Ip header has the different information than the original. IP header the internal mode is normally used between 2 routers, between a host and a router, between a host and a new header is created or context of header change.

    Screenshot from 2013-12-18 22:34:10

Ipsec has two types of protocols :
1. Authentication header protocol
2. Encapsulation security protocol

Authentication header protocol
The authentication protocol is designed to authenticate the source host and to ensure the integrity of payload carried in IP packets. The protocol uses a hash function and a symmetry key to create a message digest then the digest is inserted in the authentication header. The AH is then placed in the appropriate location based on the modes (the transport/tunnel mode). When an IP data-gram carries an AH, the original value in the protocol field of the IP header is replaced by the value 51, a field inside the authentication header i.e the next header field holds the original value of the protocol field.
In addition of an authentication header follow the steps:

  • An authentication header is added to the payload with the authentication data field set to NAT(Network address translation)
    Default gateway (Router to which we are converted)
    Global  IP and Local IP
  • Padding bits are added to make the total length even for a particular hashing algorithm
  • Hashing is based on total packet. However only those field of the IP header that do not change during transmission are included in the calculation of the message digest.
  • The authentication data are inserted in authenticated header.
  • The IP header is added after the value of protocol field is changed to 51.

Encapsulation Security Protocol
The AH protocol does not provide security. It provides only source authentication and data integrity. The IPsec defined an alternative protocol that provides source authentication integrity and privacy. EPS adds the header and trailor. EPS authentication data are added at the end of the packet which makes its calculation easier. When an IP datagram carries an ESP header and trailer the value of the protocol field in the IP header is 50. A field inside the ESP trailer holds the original value of the protocol field and the ESP procedure follows these steps:

  • An ESP trailer is added to the payload
  • The payload and trailer are encrypted
  • The ESP header is added
  • The ESP header payload and ESP trailer are used to create the authentication data
  • The authentication data are added to the end of the ESP trailer
  • IP header is added after the protocol value is changed to 50