Part 2 - Kubernetes Vault Integration - ESO
Introduction Link to heading
We now want to migrate from CSI Driver + Vault + syncSecret
→ to External Secrets Operator (ESO) + Reloader, while maintaining:
- Vault as the source of secrets
- Kubernetes Secrets for
env:injection - Auto pod restart when secrets change
Since we are using CSI with Vault, the following is already configured:
Requirement : Description
Vault server is reachable from K8s : we already use vaultAddress: http://<Vault-VM-IP>:8200 in CSI
Vault uses Kubernetes auth : SecretProviderClass uses roleName: sockshop-app
Kubernetes ServiceAccount exists : You’re using default (in CSI) and referencing it in ESO
Vault has a policy to access secrets : CSI already accesses secret/data/sockshop/database
Kubernetes has network access to Vault : Confirmed from CSI’s successful mounts
We don’t need any new Vault-side config.
✅ Required Components Link to heading
- Vault configuration
- ESO CRDs:
SecretStore,ExternalSecret - Kubernetes Secret consumption in the Deployment
- Reloader annotations to auto-restart pods on secret change
----------External Secrets Operator (ESO) + Reloader----------
┌──────────────────────────────────┐
│ Kubernetes Deployment Starts │
└────────────┬─────────────────────┘
│
▼
┌──────────────────────────────────┐
│ ExternalSecret CR is created │
└────────────┬─────────────────────┘
│
▼
┌──────────────────────────────────┐
│ ESO Controller watches CRs │
└────────────┬─────────────────────┘
│
▼
┌──────────────────────────────────┐
│ Auth to Vault via SA or AppRole │
└────────────┬─────────────────────┘
│
▼
┌──────────────────────────────────┐
│ Fetch secret from Vault path │
│ (e.g., kv-v2/secret/my-app) │
└────────────┬─────────────────────┘
│
▼
┌──────────────────────────────────┐
│ Create or update K8s Secret │
└────────────┬─────────────────────┘
│
▼
┌──────────────────────────────────┐
│ Pod mounts secret as env/volume │
└────────────┬─────────────────────┘
│
▼
┌──────────────────────────────────┐
│ Reloader watches secret changes │
└────────────┬─────────────────────┘
│
▼
┌──────────────────────────────────┐
│ On change, restarts the Pod │
└────────────┬─────────────────────┘
│
▼
┌──────────────────────────────────┐
│ App uses updated secret │
└────────────┬─────────────────────┘
│
▼
┌────────┐
│ Done │
└────────┘
🧱 Step-by-Step: Migrate Setup Link to heading
🔧 1. Install External Secrets Operator (ESO) Link to heading
If not already installed:
helm repo add external-secrets https://charts.external-secrets.io
helm upgrade --install external-secrets external-secrets/external-secrets \
--namespace external-secrets --create-namespace
🔐 2. Create a SecretStore for Vault
Link to heading
This defines how ESO authenticates to Vault.
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: vault-backend
namespace: default
spec:
provider:
vault:
server: "http://<Vault-VM-IP>:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "sockshop-app"
serviceAccountRef:
name: default
📦 3. Create the ExternalSecret Resource
Link to heading
This will sync Vault secrets into a Kubernetes Secret:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: sockshop-db-secrets
namespace: default
spec:
refreshInterval: 1m
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: sockshop-db-secrets # Name of K8s Secret created
creationPolicy: Owner
data:
- secretKey: DB_USERNAME
remoteRef:
key: secret/data/sockshop/database
property: username
- secretKey: DB_PASSWORD
remoteRef:
key: secret/data/sockshop/database
property: password
🚀 4. Create the Deployment YAML Link to heading
Replace the CSI volume + mount setup with ESO-based secret usage:
apiVersion: apps/v1
kind: Deployment
metadata:
name: your-sockshop-service
annotations:
reloader.stakater.com/auto: "true" # 👈 Triggers restart on secret change
spec:
replicas: 1
selector:
matchLabels:
app: sockshop-nginx
template:
metadata:
labels:
app: sockshop-nginx
spec:
serviceAccountName: default
containers:
- name: your-app-container
image: nginx:latest
env:
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: sockshop-db-secrets
key: DB_USERNAME
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: sockshop-db-secrets
key: DB_PASSWORD
📝 we can also annotate the deployment with:
annotations:
reloader.stakater.com/secret: "sockshop-db-secrets"
If we want to be explicit.
🔁 5. Install Reloader (if not already) Link to heading
helm repo add stakater https://stakater.github.io/stakater-charts
helm upgrade --install reloader stakater/reloader \
--namespace kube-system --create-namespace
🧪 Optional: Clean Up CSI Link to heading
- Remove the
SecretProviderClass - Remove volumeMounts and CSI volumes in pod spec
- Optionally delete the synced Secrets CSI generated
Commands to execute for testing
# View secret data (base64 encoded)
kubectl get secret sockshop-db-secrets -n default -o yaml
# Decode specific secret values
kubectl get secret sockshop-db-secrets -n default -o jsonpath='{.data.DB_USERNAME}' | base64 -d
echo "" # Add newline
kubectl get secret sockshop-db-secrets -n default -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
echo "" # Add newline
# # Or decode all at once
# kubectl get secret sockshop-db-secrets -n default -o json | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'
ubuntu@k3s1 ~> kubectl get secret sockshop-db-secrets -n default -o jsonpath='{.data.DB_USERNAME}' | base64 -d
sockuser-1⏎
ubuntu@k3s1 ~> kubectl get secret sockshop-db-secrets -n default -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
sockpassword123-2⏎
ubuntu@k3s1 ~>
#### After Change
ubuntu@k3s1 ~> kubectl get secret sockshop-db-secrets -n default -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
sockpassword123-4⏎
ubuntu@k3s1 ~>
ubuntu@k3s1 ~> kubectl get secret sockshop-db-secrets -n default -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
sockpassword123-4⏎
ubuntu@k3s1 ~> kubectl get secret sockshop-db-secrets -n default -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
sockpassword123-4⏎
ubuntu@k3s1 ~> kubectl get secret sockshop-db-secrets -n default -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
sockpassword123-5⏎
ubuntu@k3s1 ~>
ubuntu@k3s1 ~ [124]> kubectl exec -it testdeployment-dbcd7c7c9-z9mwt -- env | grep -i db_
DB_USERNAME=sockuser-1
DB_PASSWORD=sockpassword123-5
STAKATER_SOCKSHOP_DB_SECRETS_SECRET=6ef6e5f1728ba68db52780773680d60df96ded2b
ubuntu@k3s1 ~>
# After updating the password in vault, notice how the pod name has changed. This is because reloader has \
# restarted the pod after the secret change. I only noticed this because when I was testing the env by \
# exec'ing into the pod, it threw an error due to incorrent pod name.
ubuntu@k3s1 ~> kubectl get po
NAME READY STATUS RESTARTS AGE
testdeployment-664c7cdb4c-vrg6q 1/1 Running 0 52s
your-sockshop-service-5bc8fc6c6c-7fw9z 1/1 Running 0 89m
ubuntu@k3s1 ~>
ubuntu@k3s1 ~ [1|1]> kubectl exec -it testdeployment-664c7cdb4c-vrg6q -- env | grep -i db_password
DB_PASSWORD=newtestpassword-0
ubuntu@k3s1 ~> kubectl get secret sockshop-db-secrets -n default -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
newtestpassword-0⏎
✅ Final Result Link to heading
- ✅ Vault remains the source of truth
- ✅ ESO syncs secrets to Kubernetes Secret
sockshop-db-secrets - ✅ Deployment consumes that Secret via
env: - ✅ Reloader watches the Secret and auto-restarts pod on change