GATE CS 2013 Q13 — Concept of Layering: OSI and TCP/IP Protocol Stacks

To solve this, we break down the process into two main security goals: Authentication (Digital Signature) and Confidentiality (Encryption).

1. Encryption Phase (at Sender $X$)

• Step 1 (Digital Signature): To sign the message $M$ so that the receiver can verify it originated from $X$, $X$ signs/encrypts the message (or its hash) using $X$’s private key. This produces the signature $\sigma$.
  $$\sigma = E_{PR_X}(M)$$
• Step 2 (Confidentiality): To ensure that only the intended receiver $Y$ can read the combined payload $\langle M, \sigma \rangle$, $X$ encrypts it using $Y$’s public key.
  $$\text{Ciphertext} = E_{PU_Y}(\langle M, \sigma \rangle)$$

Thus, the sequence of keys used for encryption is: $X$’s private key followed by $Y$’s public key.

2. Decryption Phase (at Receiver $Y$)

The decryption process must reverse the encryption steps in a Last-In, First-Out (LIFO) order:

• Step 1 (Outer Decryption): $Y$ first decrypts the outer layer of the received ciphertext using $Y$’s private key to retrieve $\langle M, \sigma \rangle$.
  $$\langle M, \sigma \rangle = D_{PR_Y}(\text{Ciphertext})$$
• Step 2 (Signature Verification): $Y$ then verifies the digital signature $\sigma$ using $X$’s public key.
  $$M = D_{PU_X}(\sigma)$$

Thus, the sequence of keys used for decryption is: $Y$’s private key followed by $X$’s public key.

This matches the sequence described in Option D.