more https://prow.istio.io/view/gs/istio-prow/logs/integ-k8s-126_istio_postsubmit/1763304825484742656

similar with #49650?

i think it's different, that one didn't have istiod crashlooping. but maybe same

So the istiod is restarting many times. Its not failign with this on the initial install.

So probably its crashing. And then fails to come up -- two separate bugs.

I think I get it. We have 2 files, but we do not read them atomically. I can reproduce by flipping between two cacerts.

https://ahmet.im/blog/kubernetes-inotify/ touches on this but doesn't suggest a fix.

I don't understand why we seem to be permanently in that state though. I would expect TestIntermediateCertificateRefresh triggers it and only happens if istiod happens to crash around that time. Maybe there is something more nefarious going on.

Oh I think its simpler, the test is just broken:

$ openssl s_server -accept 9999 -cert ca-cert-alt-2.pem -key ca-key-alt-2.pem
Using default temp DH parameters
error setting private key
40F7FA89237D0000:error:05800074:x509 certificate routines:ossl_x509_check_private_key:key values mismatch:crypto/x509/x509_cmp.c:416:

vs

$ openssl s_server -accept 9999 -cert ca-cert-alt.pem -key ca-key-alt.pem
Using default temp DH parameters
ACCEPT
^C

And its not documented how it is generated... yay.

I don't know how the test works at all. Oh, actually looks like it doesn't actually assert anything 😬