Dumping/exporting vSphere certificates/keys

Recently while troubleshooting some SSL certificate issues on VCSA, I ran across the vecs-cli tool. It has the option to manipulate the certificate store used by vCenter.

To export all certificates and keys from all stores into individual files under a new certs/ directory:

mkdir certs
/usr/lib/vmware-vmafd/bin/vecs-cli store list | while read STORE ; do
  mkdir -p certs/${STORE}
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $STORE | awk '/^Alias/{print $3}' | while read ALIAS ; do
    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store $STORE --alias $ALIAS --output certs/${STORE}/${ALIAS}.crt
    /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store $STORE --alias $ALIAS --output certs/${STORE}/${ALIAS}.key
  done
done

You may see some usage errors for some certificates – typically in the TRUSTED_ROOTS and TRUSTED_ROOT_CRLS stores – as these do not have a private key to export.

Then, to identify which certificate is which, you can do things like:

ls -1 certs/*/*crt | while read F ; do
  echo "======== $F:"
  openssl x509 -noout -subject -fingerprint -issuer -in $F
done

You may see errors about files in certs/TRUSTED_ROOT_CRLS/ – these are actually Certificate Revocation Lists, not certificates, so this is expected.

References:

https://kb.vmware.com/kb/2121701

Fixing broken vSphere PSC certificate

I recently managed to break my machineSSL cert on my 2nd vSphere 6 PSC appliance while updating the certificates.

Using the “Reset all Certificates” link wasn’t even working 😦

It would fail at the login with: “Incorrect Password! Try Again! (2 attempt/s left)”

Eventually I discovered the lookupservice had the wrong certificate (pointing lstool.py to working PSC01):

“/usr/lib/vmidentity/tools/scripts/lstool.py list –url https://PSC01.LAB/lookupservice/sdk –no-check-cert –ep-type com.vmware.cis.cs.identity.sso 2>/dev/null”

(by comparing/searching for some text from certificate, I found it had the CA’s cert not the PSC cert – likely due to a file mixup when running replacement script…)

I was able to replace the lookupservice entry by getting the fingerprint of the “bad” certificate:

“openssl x509 -in BADCERT.crt -noout -sha1 -fingerprint”

and then updating lookupservice entry (pointing ls_update_certs.py to working PSC01):

“/usr/lib/vmidentity/tools/scripts/ls_update_certs.py –url https://PSC01.LAB/lookupservice/sdk –fingerprint 00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33 –certfile GOODCERT.crt–user administrator@vsphere.local –password ‘Password!'”

I now had a valid entry in lookup service, so the certificate-manager script could actually connect, but I still needed to tell PSC02 about its “new” certificate.

Using vecs-cli to remove current cert on PSC02:

“/usr/lib/vmware-vmafd/bin/vecs-cli entry delete –store MACHINE_SSL_CERT –alias __MACHINE_CERT”

Then adding in the good one:

“/usr/lib/vmware-vmafd/bin/vecs-cli entry create –store MACHINE_SSL_CERT –alias _MACHINE_CERT –cert /certs/machineSSL.crt –key /certs/machineSSL.key”

Then a reboot and vmware-rhttpdproxy no longer panic’d on starting! 🙂

References:

https://kb.vmware.com/kb/2121701

 

Remove VIB using PowerCLI

I was trying to manually remove NSX VIBs using PowerCLI and found some of the examples and docs confusing, so here is a hopefully clearer example…

Connect-VIServer -Server vcenter.domain.com -User administrator@vsphere.local
@(
"esx01.domain.com",
"esx02.domain.com"
) | foreach {
$esxcli = get-esxcli -V2 -VMHost $_
$esxcli.software.vib.remove.Invoke(@{"vibname" = "esx-vxlan"})
$esxcli.software.vib.remove.Invoke(@{"vibname" = "esx-vsip"})
}

You can get more parameters with:

$esxcli.software.vib.remove.Help()

You can get a list of VIB modules with:

$esxcli.software.vib.list.Invoke() | select Name,ID,Version

Hope it helps someone!
Brian