What happens if you have a CA hierachy and one of the intermediary certificates expires? With good planing, nothing should happen: one generates a new intermediary certificate and all future certificates are signed using that new intermediary certificate. Without good planing (and proper execution), it's a big mess. This is my story...

For clarification, I'll refer to the (self-signed) root CA certificate as "root" (by X.509 extension, this certificate is marked as being capable of signing it's own sub-certificates and thus becomes a "certification authority"/"CA"). The "root" certificate is used to sign the "intermediary" CA certificate (this certificate is also "marked" as being capable of signing it's own sub-certificates). The "intermediary" CA signs any number of "entity" certificates (which lack the X.509 extension marker and are thus not capable of signing any sub-certificates).

Proper certificate planing mandates that the validity of all involved certificates should be "inclusive": the validity of the "intermediary" certificate is entirely within the validity of the "root" certificate and all "entity" certificates are created to be valid within the validity of the "intermediary" certificate only. However, adhering to this plan is not trivial: it requires a properly timed transition phase in which a successor "intermediary" certificate is created before the expiration of the current "intermediary" certificate. Also, all "entity" certificates need to be regenerated using the successor "intermediary" certificate. The deployment of those new certificates needs to occur before the expiration of the current "intermediary" certificate. Thus, the current and successor "intermediary certificates need to overlap in their validity.

What happens proper certificate planning didn't occur? A certificate hierarchy that is not "inclusively" valid leads to a situation in which the "intermediary" certificate expires while "entity" certificates are still valid. The issue is that certificate validation fails because the "intermediary" certificate has expired. Renewing the "intermediary" certificate is the correct solution, but: while a regular certificate renewal (not neccessarily but quite) commonly implies the continued use of the same "intermediary" key it slightly changes the identity of the "intermediary" certificate. While the certificate's subject (distinguished name) can - and should - remain unchanged, it's the certificate's serial number that changes. This wouldn't be an issue, if the "entity" certificate didn't include the serial number of its "parent" as part of the parent's identity in the "Authority Key Identifier" extension:

[snip]
            X509v3 Authority Key Identifier: 
                keyid:D4:E6:B9:D9:FC:B4:9B:B2:FD:08:FB:A6:31:CC:88:FD:1E:30:1C:4F
                DirName:/C=DE/ST=NRW/L=Koeln/O=Akkaya Consulting/CN=intermediary/emailAddress=intermediary@akkaya.de
                serial:02
[snip]

A "standard" renewal of the "intermediary" certificate implies a new serial number for the "intermediary" certificate and thus the validation of the "entity" certificate fails because either the (old) "intermediary" certificate has expired or the (new) "intermediary" certificate's serial number does not match the serial number specified in the "entity" certificate's "Authority Key Identifier". Doh.

The question is how to renew the "intermediary" certificate without issuing a new serial number. The solution is actually quite trivial (but not-so-trivial to find):

openssl x509 -in intermediary_old-cert.pem -out intermediary_new-cert.pem \
             -CA root-cert.pem -CAkey root-key.pem \
             -CAcreateserial -CAserial /tmp/dummy-$$.srl -set_serial OLD_SERIAL \
             -days VALIDITY_IN_DAYS_FROM_NOW

Problem solved and solution documented: yeay.