Trust Is the Hard Part: Running a Private CA for a VCF Lab

I’ve run certificate authorities before — but always by hand, for a handful of services that stayed put. Standing up the CA was a careful one-time event: you did it once and then left it alone. This nested VCF lab is the opposite. It gets torn down and rebuilt constantly, so the CA can’t be a ceremony performed once — it has to be something automation stands up, issues from, and re-establishes trust for, every single time the lab comes back.

And there’s a lot of trust to re-establish. A couple of dozen TLS endpoints — vCenter, NSX, SDDC Manager, Operations, Keycloak, a stack of ESXi hosts, and a growing tier of Kubernetes platform services behind the cluster ingress (Harbor, ArgoCD, and whatever else lands there) — and every one of them wants a certificate. I could hand each a self-signed cert and spend the rest of my life clicking through browser warnings, while VCF’s own validation quietly refused to talk to half of them. Or I could stand up a real CA, sign everything from one trusted root, and make the warnings disappear.

I chose the CA. It taught me that issuing the certificate is the easy part.

Lab Environment

The lab runs as a vApp on vCloud Director — a private address space behind a single Linux host that acts as gateway for my operator laptop. That gateway runs tinyproxy and a SOCKS proxy for connectivity, alongside the supporting services the rest of the lab leans on: DNS, DHCP, Keycloak in a container, and the certificate authority this post is about. For a lab, that’s exactly the right weight — something I can stand up, issue from, automate against, and rebuild without ceremony. Putting the CA here is deliberate: it’s the one service every other component has to reach before it can be trusted.

What a Private CA Actually Is

Strip away the ceremony and a certificate authority is just a trusted signer.

A TLS certificate is a public key with a name attached — “this key belongs to vcenter-mgmt.lab.example” — and a signature from someone vouching for that claim. The whole system rests on a single question the client asks every time: do I trust whoever signed this? If the signer is in the client’s list of trusted authorities, the certificate is accepted and you get the padlock. If not, you get the warning.

The clever part is that trust is transitive, and arranged as a chain. At the top sits a root — a self-signed certificate that is the anchor of the whole thing. The root doesn’t sign day-to-day certificates directly; instead it signs an intermediate, and the intermediate signs the leaf certificates that actual services present. A client that trusts the root will, by following the chain, trust everything the intermediate ever signs — without ever having seen those leaf certs before.

That indirection is the entire payoff. You distribute one certificate — the root — to everything that needs to validate TLS, and from that moment every service signed by your CA is trusted automatically. Add a new host next month, issue it a leaf cert, and nobody has to be told. They already trust the root, so they already trust the host.

That’s the theory. Establishing that one root as trusted everywhere turns out to be the whole game.

Why step-ca Is the Right Tool for a Lab

If you want to run your own CA without standing up an enterprise PKI product, step-ca from Smallstep is the obvious answer. It’s a small, open-source online CA: you initialise it once, and from then on it signs certificates on demand. It speaks ACME — the same protocol Let’s Encrypt uses — so anything that knows how to get a cert from Let’s Encrypt can get one from your private CA instead, just by pointing at a different URL.

For a lab, that profile is perfect. I don’t want an air-gapped offline root ceremony with a hardware security module and a witness; I want a CA that’s just another service I can script and rebuild on demand. step-ca is exactly that — one more process on the lab gateway.

One authority, sitting at the centre of the lab, signing for everything. Initialise it, and you have a root. Trust that root once on each machine, and the entire lab can speak valid TLS to itself.

Issuing a Certificate Is the Easy Part

Minting a leaf certificate from step-ca is a single command. Name, a few subject-alternative names, a validity period, done. If that were the whole job, this would be a very short post.

Then comes the part nobody warns you about: every system that has to validate these certificates needs the root in its trust store — and every system keeps its trust store in a different place, in a different format, and updates it in a different way.

The operating system has its bundle. The Java world has its own truststore entirely, which is why a component running on a bundled JRE — like vCenter, or VCF Operations — can sit on a machine whose OS trusts my root and still reject the certificate, because the JVM never looks at the OS bundle. ESXi has its own trust file in its own location. Browsers have their own keychains, sometimes deferring to the OS, sometimes not. Miss any one of these and you get a failure that looks like a network problem — a refused connection, a validation error deep inside VCF — with nothing pointing at the actual cause, which is simply that one party in the conversation has never been introduced to the root.

There’s a second trap waiting underneath the first. Some consumers read their certificate once, at startup, and cache it. Hand them a freshly signed cert and they carry on serving the old one until the service is restarted — so a cert change that “didn’t take” is often a cert change that took perfectly and just hasn’t been picked up yet.

And then there’s the consumer that dictates terms back to you. The modern default for a new CA is elliptic-curve keys — smaller, faster, perfectly standard — and that’s what mine started on. But vCenter won’t accept an elliptic-curve-signed leaf at all; it wants RSA, and there is no negotiating with it. So it negotiated for everyone: to re-issue vCenter’s identity from my root I had to re-key the entire CA to RSA — root, intermediate, and every leaf below them — to satisfy the single most opinionated consumer in the building. That’s PKI in one lesson: “follow the standard” isn’t enough when the other end has its own idea of which standard, and the most rigid participant ends up setting terms for everyone.

None of this is intellectually hard. It’s just distributed, and fiddly, and unforgiving of the one machine you forgot. Which makes it exactly the kind of thing you want to encode once and never perform by hand again.

Lab Build Automation

The CA is initialised once, early, by the same Ansible role that builds the gateway — before any ESXi host or VCF appliance exists, because everything downstream depends on the root already being there. The init generates the root and intermediate, writes the config and a systemd unit, and brings step-ca up as a service. Two deliberate departures from the defaults: the certificate lifetime is stretched from step-ca’s cautious 24-hour default out to roughly 120 days, so a leaf issued on build day stays valid for the lab’s entire evaluation-window life rather than expiring mid-build; and the chain is re-keyed to RSA-3072 straight after init — step ca init insists on elliptic curve and offers no way to say otherwise, so the root and intermediate are regenerated as RSA before the service ever starts. Not the algorithm I’d have chosen, but the one vCenter leaves no say in, so the whole chain wears it and every leaf below, ESXi included, is RSA.

Certificates get to their consumers in whatever way suits the consumer. ESXi hosts can’t run an ACME client during a scripted install, so their certs are pre-signed on the gateway and served over plain HTTP on the install network; each host’s first-boot script fetches its cert and key, appends the root to its local trust file, and restarts the affected services so the new cert is actually loaded rather than cached away. Keycloak gets a cert issued on the gateway and mounted straight into its container. The Kubernetes side is the tidy one: step-ca’s ACME endpoint is wired in as a cert-manager ClusterIssuer, so any ingress in the cluster can request a trusted cert just by annotating itself — exactly the Let’s Encrypt experience, pointed at a private root.

Trust distribution is the half that takes the real care. The root is copied into the gateway’s own OS bundle and published over HTTP for anything that needs to fetch it. ESXi hosts import it at first boot. SDDC Manager and the VCF components get it uploaded into their trusted-certificate store through the platform API and the running trust cache refreshed. cert-manager carries it as a CA bundle on the issuer. The operator’s browser trusts it once, by hand, on the laptop — the one introduction I was happy to leave manual.

Re-issuing the appliances’ own identities is the piece that took longest to automate. Distributing the root is one thing; replacing each appliance’s self-signed certificate with one from that root is another — and VCF 9.1 moved its certificate-replacement API into Operations’ fleet-management layer, so the automation had to follow it there. Now it does: for each management appliance it generates a signing request, has step-ca sign it, and hands the chain back — so vCenter, NSX, SDDC Manager, Operations and the rest all present certificates that chain to the same root everything else already trusts. The whole estate, from ESXi up through the platform to the Kubernetes ingress, ends up signed by one authority I stood up myself.

And then there’s the genuinely awkward stuff — which makes the case for automating all this better than any tidy diagram could. After each replacement VCF runs its own retrust: it goes round re-introducing the appliances to one another under their new certs, re-pinning the fingerprints each uses to recognise its peers. For a certificate from an external CA like mine, that re-introduction quietly doesn’t finish — vCenter serves my cert, but NSX carries on recognising it by the fingerprint of the cert it used to serve, and nothing surfaces the mismatch. Worse, the one host you’d ask to check — Operations, which runs the service that inventories every certificate — couldn’t see its own freshly-issued cert: it kept reporting the old one about itself, so the obvious “did it take?” query lied to me. Both had the same answer, and it’s the lesson of the whole exercise in miniature: stop trusting what a system says about itself, read what it actually serves on the wire, and re-pin the stragglers through the very API VCF uses internally. None of it is hard once you’ve found it. All of it is exactly the kind of distributed, easy-to-miss detail you never want to be rediscovering by hand on the next rebuild — so it’s codified, one switch away from running instead of an afternoon of rediscovery.

What the Certificate Taught Me

I’d run certificate authorities before, but never one that had to tear down and stand back up on command, across a fleet this varied. That’s what changed the lesson for me. The cryptography was never the difficult bit. The difficult bit is that trust is a distributed agreement — every party to it stores its half somewhere different, and all of it has to be re-established the same way every time the lab rebuilds.

The lesson is the one that keeps recurring across this whole lab: the value isn’t in issuing a certificate once. Anyone can do that. The value is in establishing one root, pushing it into trust store after trust store across the lab — the OS bundle, the ESXi trust file, the platform’s certificate store — automatically, every time, so the environment comes up speaking valid TLS end to end with no warnings and no manual introductions. The certificate is a five-second command. The trust is the system.

Here’s the thing about certificates in a lab: they add no functionality. Nothing you build works better for having a trusted cert — so when you’re racing to stand up vCenter or get a cluster talking, they’re the first corner you cut. That’s the trap. The friction of skipping them — the warnings, the refused validations, the half-working consoles — quietly taxes everything you do afterwards. Automating the trust away doesn’t add a feature either; it just removes a tax you’d otherwise pay on every interaction with the lab. And that, more than anything, is what makes the lab a good place to work.

These days every endpoint in the lab presents a certificate that chains back to a root I created, and everything that talks to it simply believes it. No warnings, no exceptions clicked away — even vCenter, which made me re-key the whole chain to RSA before it would shake hands, and even the peers that quietly went on recognising each other by the old fingerprints until the automation went round and re-pinned them. None of it is anything I have to remember to do any more: the root is stood up, the certificates issued, and the trust pushed out on every rebuild — and the appliance re-issue, with its fingerprint re-pinning, is codified alongside as a single switch rather than a procedure to remember. What’s left is just the quiet green padlock that means two machines were properly introduced before they started talking — and the introduction, for once, was mine to make.