The TLS Handshake

The TLS Handshake: Core Concepts

When your browser connects to https://example.com, a complex negotiation happens before a single byte of web page data is transmitted. The TLS handshake establishes the cryptographic parameters for the session: which protocol version to use, which cipher suite, and what keys to use for encryption. Understanding this process is essential for diagnosing connection issues, evaluating security configurations, and understanding what TLS actually protects.

What the Handshake Accomplishes

The TLS handshake has three goals:

1. Authentication — Verify the server's identity using its certificate (and optionally the client's identity)
2. Key exchange — Establish a shared secret that will be used to derive symmetric session keys
3. Cipher suite negotiation — Agree on the algorithms for encryption, integrity, and key derivation

TLS 1.3 Handshake Walkthrough

TLS 1.3 (the current version) simplified the handshake to one round trip (compared to two in TLS 1.2):

Client                                Server
  |                                       |
  |--- ClientHello ----------------------->|
  |   (supported versions, cipher suites,  |
  |    key_share for ECDHE, random)        |
  |                                       |
  |<-- ServerHello ------------------------|
  |   (selected version, cipher suite,    |
  |    key_share for ECDHE, random)        |
  |                                       |
  |<-- {EncryptedExtensions} -------------|
  |<-- {Certificate} ---------------------|
  |<-- {CertificateVerify} ---------------|
  |<-- {Finished} ------------------------|
  |                                       |
  |--- {Finished} ----------------------->|
  |                                       |
  |<====== Application Data =============>|

The {} notation indicates encrypted messages. After the ServerHello, all subsequent messages are encrypted with keys derived from the shared secret.

Step 1: ClientHello — The client sends:

• Supported TLS versions (e.g., TLS 1.3, TLS 1.2)
• Supported cipher suites (e.g., TLS_AES_128_GCM_SHA256)
• A key share for ECDHE (Elliptic Curve Diffie-Hellman Ephemeral) — the client's public key for key exchange
• A random nonce
• Supported groups, signature algorithms, and other extensions

Step 2: ServerHello — The server responds with:

• The selected TLS version and cipher suite
• Its own ECDHE key share
• A random nonce

At this point, both parties can compute the shared secret using ECDHE. Each side combines its private key with the other's public key to derive the same shared secret.

Step 3: Server sends encrypted messages — The server sends its certificate, a signature proving possession of the certificate's private key, and a Finished message. These are encrypted with keys derived from the shared secret.

Step 4: Client sends Finished — The client verifies the server's certificate and signature, then sends its own Finished message. From this point, application data flows encrypted with the derived session keys.

Key Derivation

The shared secret from ECDHE is not used directly as the encryption key. Instead, it is passed through a Key Derivation Function (KDF) to produce multiple keys:

Shared Secret (from ECDHE)
    ↓ HKDF (HMAC-based Key Derivation Function)
    ├── Client write key (for client→server encryption)
    ├── Server write key (for server→client encryption)
    ├── Client write IV (initialization vector)
    └── Server write IV

Using separate keys for each direction prevents an attacker from reflecting messages back to the sender.

Forward Secrecy

TLS 1.3 uses ephemeral Diffie-Hellman (DHE or ECDHE) for key exchange. "Ephemeral" means new key pairs are generated for each session. This provides forward secrecy: even if an attacker records encrypted traffic today and later compromises the server's private key, they cannot decrypt the recorded traffic because the session keys were derived from ephemeral key pairs that no longer exist.

This is a significant improvement over TLS 1.2, where RSA key transport (without forward secrecy) was common. With RSA key transport, the client encrypts the pre-master secret with the server's public key. If the server's private key is later compromised, all recorded sessions can be decrypted.

Where It Fits

The TLS handshake combines all the cryptographic primitives you have studied: asymmetric encryption (certificates, signatures), symmetric encryption (session keys), hashing (key derivation, integrity), and PKI (certificate validation). Study PKI and Certificates first to understand how the server's identity is verified, then use this knowledge to understand the full handshake.

The TLS Handshake: Practice

The TLS handshake is best understood by observing it. The exercises below use OpenSSL and Wireshark to capture and analyze real handshakes.

Exercise 1: Observe a TLS 1.3 Handshake with OpenSSL

# Connect to a server and see the handshake details
openssl s_client -connect example.com:443 -tls1_3 -msg

The -msg flag prints each handshake message. You will see:

>>> ClientHello
<<< ServerHello
<<< EncryptedExtensions
<<< Certificate
<<< CertificateVerify
<<< Finished
>>> Finished

Note that after ServerHello, all messages are encrypted (shown as <<< without type details).

Exercise 2: Compare TLS 1.2 and TLS 1.3

# Force TLS 1.2
openssl s_client -connect example.com:443 -tls1_2 -msg

# Force TLS 1.3
openssl s_client -connect example.com:443 -tls1_3 -msg

Compare the number of round trips:

• TLS 1.2: Two round trips (ClientHello → ServerHello+Certificate → ClientKeyExchange+Finished → ServerFinished)
• TLS 1.3: One round trip (ClientHello → ServerHello+encrypted messages → Client Finished)

TLS 1.3 is faster because it eliminates one full round trip.

Exercise 3: Analyze a Handshake with Wireshark

Capture a TLS handshake in Wireshark:

# Start a capture (requires Wireshark/tshark)
tshark -i eth0 -w tls_capture.pcap

# In another terminal, make a HTTPS request
curl https://example.com

# Stop the capture and analyze
tshark -r tls_capture.pcap -Y "tls.handshake" -V

Look for:

• ClientHello: What cipher suites does the client offer?
• ServerHello: Which cipher suite did the server select?
• Key exchange: What curve is used for ECDHE?
• Certificate: What is the certificate chain?

Exercise 4: Test Cipher Suite Support

# Test if a server supports a specific cipher suite
openssl s_client -connect example.com:443 -cipher 'ECDHE-RSA-AES128-GCM-SHA256'

# Test with a weak cipher (should fail on modern servers)
openssl s_client -connect example.com:443 -cipher 'RC4-SHA'

Modern servers reject weak cipher suites. If the connection fails with "no ciphers available", the server correctly refuses weak encryption.

Check Your Understanding

• Why does TLS 1.3 use ephemeral Diffie-Hellman instead of RSA key transport?
• What happens if an attacker modifies the ClientHello message?
• How does the client verify that the server possesses the private key corresponding to the certificate?

The TLS Handshake: Trade-offs and Interviews

TLS is the most widely deployed security protocol in the world, but it is not a panacea. Understanding its limitations is essential for making informed security decisions.

When TLS Is the Wrong Tool

• Data at rest: TLS protects data in transit. For data at rest (files on disk, database records), use disk encryption (LUKS, BitLocker) or application-level encryption.
• End-to-end encryption: TLS terminates at the server. If you need end-to-end encryption where the server cannot read the messages (e.g., messaging apps), use the Signal Protocol or similar application-layer encryption.
• Authentication of the client: TLS authenticates the server by default. Client authentication (mutual TLS) is possible but rarely used on the public internet. For client authentication, use application-layer mechanisms (OAuth, API keys).
• Protection against malware: TLS encrypts traffic but does not prevent malware from being delivered over an encrypted connection. You need antivirus, intrusion detection, and other controls.

TLS 1.2 vs. TLS 1.3

TLS 1.3 removed many legacy features that were sources of vulnerabilities. If a server still supports TLS 1.2, ensure it uses only AEAD cipher suites (GCM, ChaCha20-Poly1305) and ephemeral Diffie-Hellman.

Common TLS Misconfigurations

• Supporting TLS 1.0/1.1: These versions have known vulnerabilities (BEAST, POODLE). Disable them.
• Weak cipher suites: RC4, DES, 3DES, and export-grade ciphers should be disabled.
• Missing certificate chain: The server must send the intermediate CA certificate. If it does not, clients cannot build the chain to a trusted root.
• No HSTS: Without HTTP Strict Transport Security, an attacker can downgrade the connection to HTTP.
• Expired certificates: Automate renewal to prevent outages.

How This Comes Up in Interviews

• "Walk me through the TLS 1.3 handshake." ClientHello (with key share) → ServerHello (with key share) → encrypted server messages (certificate, verify, finished) → client Finished. Emphasize forward secrecy, one round trip, and that all messages after ServerHello are encrypted.
• "What is forward secrecy and why does it matter?" Forward secrecy means that compromising the server's private key does not allow decryption of previously recorded sessions. TLS 1.3 mandates ephemeral Diffie-Hellman, which generates new key pairs for each session.
• "How do you secure a web server's TLS configuration?" Use TLS 1.3 (with TLS 1.2 as fallback), only AEAD cipher suites, ECDHE for key exchange, a valid certificate with the full chain, HSTS headers, and automated certificate renewal. Test with SSL Labs (ssllabs.com/ssltest).