Running Your Own Personal HashiCorp Vault on Local Kubernetes: A Security Engineer's Guide

Introduction

Running HashiCorp Vault locally on Kubernetes is an excellent way to learn Vault’s capabilities, test credential management strategies, and develop secure applications without the complexity of a production deployment. This guide walks you through setting up a fully functional Vault instance on your local machine using Kubernetes, complete with auto-unseal, secret engines, and policy management.

Whether you’re evaluating Vault for your organization, developing security tools, or just want a secure local secret store, this setup provides a production-like environment for experimentation and development.

Prerequisites and Environment Setup

Required Tools

Before we begin, ensure you have the following tools installed:

# Check if tools are installed
docker --version          # Docker Desktop or Docker Engine
kubectl version --client  # Kubernetes CLI
helm version             # Helm 3.x
vault --version          # Vault CLI

# Install missing tools
# macOS
brew install kubectl helm vault
brew install --cask docker

# Linux
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz | tar xz
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_linux_amd64.zip

# Windows (using Chocolatey)
choco install kubernetes-cli helm vault docker-desktop

Kubernetes Cluster Options

Choose one of these local Kubernetes options:

Option 1: Docker Desktop (Recommended for beginners)

# Enable Kubernetes in Docker Desktop
# Settings → Kubernetes → Enable Kubernetes
# Wait for cluster to be ready
kubectl cluster-info

Option 2: kind (Kubernetes in Docker)

# Install kind
brew install kind  # macOS
# or
GO111MODULE="on" go install sigs.k8s.io/[email protected]

# Create cluster
kind create cluster --name vault-cluster --config kind-config.yaml

Option 3: minikube

# Install minikube
brew install minikube  # macOS

# Start cluster with sufficient resources
minikube start --cpus=4 --memory=8192 --driver=docker

Cluster Configuration

For kind, use this configuration for better Vault performance:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 8200
    protocol: TCP
  extraMounts:
  - hostPath: ./vault-data
    containerPath: /vault-data

Basic Vault Installation

Method 1: Using Helm (Recommended)

# Add HashiCorp Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Create namespace
kubectl create namespace vault

# Install Vault in dev mode (quick start)
helm install vault hashicorp/vault \
  --namespace vault \
  --set "server.dev.enabled=true" \
  --set "injector.enabled=false" \
  --set "ui.enabled=true" \
  --set "ui.serviceType=NodePort" \
  --set "ui.serviceNodePort=30000"

# Wait for pod to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=vault -n vault --timeout=120s

# Check status
kubectl get all -n vault

Method 2: Manual Kubernetes Manifests

For more control, use custom manifests:

# vault-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: vault
---
# vault-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: vault
spec:
  type: NodePort
  ports:
    - name: vault
      port: 8200
      targetPort: 8200
      nodePort: 30000
    - name: vault-internal
      port: 8201
      targetPort: 8201
  selector:
    app: vault
---
# vault-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.15.0
        ports:
        - containerPort: 8200
          name: vault
        - containerPort: 8201
          name: vault-internal
        env:
        - name: VAULT_DEV_ROOT_TOKEN_ID
          value: "root-token"
        - name: VAULT_DEV_LISTEN_ADDRESS
          value: "0.0.0.0:8200"
        - name: VAULT_ADDR
          value: "http://127.0.0.1:8200"
        volumeMounts:
        - name: vault-data
          mountPath: /vault/data
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
      volumes:
      - name: vault-data
        emptyDir: {}

Apply the manifests:

kubectl apply -f vault-namespace.yaml
kubectl apply -f vault-service.yaml
kubectl apply -f vault-deployment.yaml

Accessing Vault

# Port forward for CLI access
kubectl port-forward -n vault svc/vault 8200:8200 &

# Export Vault address
export VAULT_ADDR='http://127.0.0.1:8200'

# For dev mode, use the root token
export VAULT_TOKEN='root-token'

# Verify connection
vault status

# Access UI at http://localhost:8200/ui

Production-Like Setup with Raft Storage

For a more realistic setup, let’s configure Vault with Raft storage and auto-unseal:

Step 1: Create ConfigMap for Vault Configuration

# vault-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-config
  namespace: vault
data:
  vault.hcl: |
    ui = true
    
    listener "tcp" {
      address = "[::]:8200"
      cluster_address = "[::]:8201"
      tls_disable = 1
    }
    
    storage "raft" {
      path = "/vault/data"
      node_id = "vault-0"
    }
    
    seal "transit" {
      address = "http://vault-transit:8200"
      disable_renewal = "false"
      key_name = "autounseal"
      mount_path = "transit/"
      tls_skip_verify = "true"
    }
    
    service_registration "kubernetes" {
      namespace = "vault"
      pod_name = "vault-0"
    }
    
    api_addr = "http://vault-0.vault-internal:8200"
    cluster_addr = "https://vault-0.vault-internal:8201"

Step 2: Set Up Transit Vault for Auto-Unseal

# transit-vault.yaml
apiVersion: v1
kind: Service
metadata:
  name: vault-transit
  namespace: vault
spec:
  selector:
    app: vault-transit
  ports:
    - port: 8200
      targetPort: 8200
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-transit
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault-transit
  template:
    metadata:
      labels:
        app: vault-transit
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.15.0
        env:
        - name: VAULT_DEV_ROOT_TOKEN_ID
          value: "transit-root-token"
        - name: VAULT_DEV_LISTEN_ADDRESS
          value: "0.0.0.0:8200"
        ports:
        - containerPort: 8200
        command: ["vault", "server", "-dev", "-dev-root-token-id=transit-root-token"]

Step 3: StatefulSet for Vault with Raft

# vault-statefulset.yaml
apiVersion: v1
kind: Service
metadata:
  name: vault-internal
  namespace: vault
spec:
  clusterIP: None
  selector:
    app: vault
  ports:
    - name: vault
      port: 8200
    - name: vault-internal
      port: 8201
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
  namespace: vault
spec:
  serviceName: vault-internal
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.15.0
        ports:
        - containerPort: 8200
          name: vault
        - containerPort: 8201
          name: vault-internal
        env:
        - name: VAULT_ADDR
          value: "http://127.0.0.1:8200"
        - name: VAULT_API_ADDR
          value: "http://vault-0.vault-internal:8200"
        - name: VAULT_CLUSTER_ADDR
          value: "https://vault-0.vault-internal:8201"
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: vault-data
          mountPath: /vault/data
        - name: vault-config
          mountPath: /vault/config
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        command: ["vault", "server", "-config=/vault/config/vault.hcl"]
      volumes:
      - name: vault-config
        configMap:
          name: vault-config
  volumeClaimTemplates:
  - metadata:
      name: vault-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

Step 4: Initialize and Unseal

# Apply all configurations
kubectl apply -f transit-vault.yaml
kubectl apply -f vault-config.yaml
kubectl apply -f vault-statefulset.yaml

# Wait for pods
kubectl wait --for=condition=ready pod -l app=vault-transit -n vault --timeout=120s
kubectl wait --for=condition=ready pod vault-0 -n vault --timeout=120s

# Configure transit vault for auto-unseal
kubectl exec -n vault deployment/vault-transit -- vault login transit-root-token
kubectl exec -n vault deployment/vault-transit -- vault secrets enable transit
kubectl exec -n vault deployment/vault-transit -- vault write -f transit/keys/autounseal

# Initialize main vault
kubectl exec -n vault vault-0 -- vault operator init -recovery-shares=5 -recovery-threshold=3

# Save the root token and recovery keys securely!

Configuring Vault for Development

Enable Essential Secret Engines

# Key-Value secrets engine
vault secrets enable -version=2 kv
vault secrets enable -path=secret kv-v2

# AWS secrets engine
vault secrets enable aws
vault write aws/config/root \
  access_key=$AWS_ACCESS_KEY_ID \
  secret_key=$AWS_SECRET_ACCESS_KEY \
  region=us-east-1

# Database secrets engine
vault secrets enable database

# PKI secrets engine
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki

# Transit for encryption as a service
vault secrets enable transit

Configure Authentication Methods

# Kubernetes auth
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

# AppRole auth
vault auth enable approle
vault policy write my-app - <<EOF
path "secret/data/my-app/*" {
  capabilities = ["read"]
}
EOF

vault write auth/approle/role/my-app \
  token_policies="my-app" \
  token_ttl=1h \
  token_max_ttl=4h

# Userpass for local testing
vault auth enable userpass
vault write auth/userpass/users/testuser \
  password=testpassword \
  policies=default

Create Development Policies

# Developer policy
vault policy write developer - <<EOF
# Read and write to secret/data/dev
path "secret/data/dev/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# List secret/metadata
path "secret/metadata/*" {
  capabilities = ["list"]
}

# Create and manage AWS credentials
path "aws/creds/developer" {
  capabilities = ["read"]
}

# Encrypt/decrypt with transit
path "transit/encrypt/app-key" {
  capabilities = ["update"]
}

path "transit/decrypt/app-key" {
  capabilities = ["update"]
}
EOF

# Security engineer policy
vault policy write security-engineer - <<EOF
# Full access to secret engine
path "secret/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage AWS roles
path "aws/roles/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage database connections
path "database/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Audit log access
path "sys/audit" {
  capabilities = ["read", "list"]
}

# Policy management
path "sys/policies/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOF

Integration Examples

Python Application with Vault

# vault_client.py
import hvac
import os
from kubernetes import client, config

class VaultClient:
    def __init__(self, vault_addr='http://localhost:8200'):
        self.vault_addr = vault_addr
        self.client = None
        self._authenticate()
    
    def _authenticate(self):
        """Authenticate based on environment"""
        # Try Kubernetes auth first
        if os.path.exists('/var/run/secrets/kubernetes.io/serviceaccount/token'):
            self._k8s_auth()
        # Fall back to token auth
        elif os.environ.get('VAULT_TOKEN'):
            self.client = hvac.Client(
                url=self.vault_addr,
                token=os.environ['VAULT_TOKEN']
            )
        else:
            raise Exception("No authentication method available")
    
    def _k8s_auth(self):
        """Authenticate using Kubernetes service account"""
        with open('/var/run/secrets/kubernetes.io/serviceaccount/token', 'r') as f:
            jwt = f.read()
        
        self.client = hvac.Client(url=self.vault_addr)
        self.client.auth.kubernetes.login(
            role='my-app',
            jwt=jwt
        )
    
    def get_secret(self, path):
        """Retrieve secret from KV v2"""
        response = self.client.secrets.kv.v2.read_secret_version(
            path=path,
            mount_point='secret'
        )
        return response['data']['data']
    
    def get_aws_creds(self, role='developer'):
        """Get temporary AWS credentials"""
        response = self.client.read(f'aws/creds/{role}')
        return {
            'access_key': response['data']['access_key'],
            'secret_key': response['data']['secret_key'],
            'session_token': response['data']['security_token']
        }
    
    def encrypt(self, plaintext, key_name='app-key'):
        """Encrypt data using Transit engine"""
        response = self.client.write(
            f'transit/encrypt/{key_name}',
            plaintext=plaintext
        )
        return response['data']['ciphertext']
    
    def decrypt(self, ciphertext, key_name='app-key'):
        """Decrypt data using Transit engine"""
        response = self.client.write(
            f'transit/decrypt/{key_name}',
            ciphertext=ciphertext
        )
        return response['data']['plaintext']

# Example usage
if __name__ == '__main__':
    vault = VaultClient()
    
    # Store a secret
    vault.client.secrets.kv.v2.create_or_update_secret(
        path='my-app/config',
        secret={'api_key': 'secret-value'},
        mount_point='secret'
    )
    
    # Retrieve the secret
    config = vault.get_secret('my-app/config')
    print(f"API Key: {config['api_key']}")
    
    # Get AWS credentials
    aws_creds = vault.get_aws_creds()
    print(f"AWS Access Key: {aws_creds['access_key']}")
    
    # Encrypt sensitive data
    encrypted = vault.encrypt('sensitive-data')
    print(f"Encrypted: {encrypted}")
    
    # Decrypt it back
    decrypted = vault.decrypt(encrypted)
    print(f"Decrypted: {decrypted}")

Kubernetes Pod with Vault Integration

# app-deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "my-app"
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/my-app/config"
        vault.hashicorp.com/agent-inject-template-config: |
          {{ with secret "secret/data/my-app/config" -}}
          export API_KEY="{{ .Data.data.api_key }}"
          export DB_PASSWORD="{{ .Data.data.db_password }}"
          {{- end }}
    spec:
      serviceAccountName: my-app
      containers:
      - name: app
        image: my-app:latest
        command: ['sh', '-c']
        args: ['source /vault/secrets/config && exec my-app']
        env:
        - name: VAULT_ADDR
          value: "http://vault.vault.svc:8200"

Shell Script Integration

#!/bin/bash
# vault-wrapper.sh - Wrapper script for using Vault credentials

# Setup
VAULT_ADDR="http://localhost:8200"
VAULT_TOKEN=${VAULT_TOKEN:-$(cat ~/.vault-token)}

# Function to get AWS credentials
get_aws_creds() {
    local role="${1:-developer}"
    vault read -format=json aws/creds/$role | jq -r '.data | 
        "export AWS_ACCESS_KEY_ID=\(.access_key)\n
         export AWS_SECRET_ACCESS_KEY=\(.secret_key)\n
         export AWS_SESSION_TOKEN=\(.security_token)"'
}

# Function to get secret
get_secret() {
    local path="$1"
    vault kv get -format=json secret/$path | jq -r '.data.data'
}

# Function to encrypt file
encrypt_file() {
    local file="$1"
    local key="${2:-app-key}"
    
    base64 < "$file" | vault write -format=json transit/encrypt/$key plaintext=- | \
        jq -r '.data.ciphertext' > "$file.enc"
}

# Function to decrypt file
decrypt_file() {
    local file="$1"
    local key="${2:-app-key}"
    
    vault write -format=json transit/decrypt/$key ciphertext=@"$file" | \
        jq -r '.data.plaintext' | base64 -d
}

# Example usage
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    # Get AWS credentials and export them
    eval $(get_aws_creds developer)
    
    # Get database password
    DB_PASSWORD=$(get_secret my-app/config | jq -r '.db_password')
    
    # Encrypt a file
    encrypt_file sensitive-data.txt
    
    # Run command with credentials
    aws s3 ls
fi

Backup and Restore

Automated Backup Script

#!/bin/bash
# vault-backup.sh

BACKUP_DIR="/tmp/vault-backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/vault-backup-$TIMESTAMP.snap"

# Create backup directory
mkdir -p $BACKUP_DIR

# Take snapshot
kubectl exec -n vault vault-0 -- vault operator raft snapshot save /tmp/backup.snap
kubectl cp vault/vault-0:/tmp/backup.snap $BACKUP_FILE

# Encrypt backup with GPG
gpg --symmetric --cipher-algo AES256 $BACKUP_FILE

# Upload to S3 (optional)
aws s3 cp $BACKUP_FILE.gpg s3://my-vault-backups/

# Clean up old backups (keep last 7 days)
find $BACKUP_DIR -name "vault-backup-*.snap*" -mtime +7 -delete

echo "Backup completed: $BACKUP_FILE.gpg"

Restore Process

#!/bin/bash
# vault-restore.sh

BACKUP_FILE="$1"

if [ -z "$BACKUP_FILE" ]; then
    echo "Usage: $0 <backup-file>"
    exit 1
fi

# Decrypt if needed
if [[ $BACKUP_FILE == *.gpg ]]; then
    gpg --decrypt $BACKUP_FILE > /tmp/restore.snap
    BACKUP_FILE="/tmp/restore.snap"
fi

# Copy to pod
kubectl cp $BACKUP_FILE vault/vault-0:/tmp/restore.snap

# Restore snapshot
kubectl exec -n vault vault-0 -- vault operator raft snapshot restore /tmp/restore.snap

echo "Restore completed from: $BACKUP_FILE"

Monitoring and Observability

Prometheus Metrics

# vault-prometheus.yaml
apiVersion: v1
kind: Service
metadata:
  name: vault-metrics
  namespace: vault
  labels:
    app: vault
spec:
  selector:
    app: vault
  ports:
    - name: metrics
      port: 8200
      targetPort: 8200
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vault
  namespace: vault
spec:
  selector:
    matchLabels:
      app: vault
  endpoints:
  - port: metrics
    path: /v1/sys/metrics
    params:
      format: ['prometheus']

Enable Telemetry in Vault

# Add to vault config
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname = true
}

Grafana Dashboard

{
  "dashboard": {
    "title": "Vault Metrics",
    "panels": [
      {
        "title": "Seal Status",
        "targets": [
          {
            "expr": "vault_core_unsealed"
          }
        ]
      },
      {
        "title": "Active Requests",
        "targets": [
          {
            "expr": "vault_route_request_duration_seconds_count"
          }
        ]
      },
      {
        "title": "Token Count",
        "targets": [
          {
            "expr": "vault_token_count"
          }
        ]
      }
    ]
  }
}

Security Hardening

Network Policies

# vault-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vault-network-policy
  namespace: vault
spec:
  podSelector:
    matchLabels:
      app: vault
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: default
    - namespaceSelector:
        matchLabels:
          name: vault
    ports:
    - protocol: TCP
      port: 8200
    - protocol: TCP
      port: 8201
  egress:
  - to:
    - namespaceSelector: {}
    ports:
    - protocol: TCP
      port: 443
    - protocol: TCP
      port: 8200

TLS Configuration

# Generate self-signed certificates
openssl req -x509 -newkey rsa:4096 -keyout vault-key.pem -out vault-cert.pem -days 365 -nodes \
  -subj "/CN=vault.vault.svc.cluster.local" \
  -addext "subjectAltName=DNS:vault,DNS:vault.vault.svc,DNS:vault.vault.svc.cluster.local,DNS:localhost,IP:127.0.0.1"

# Create Kubernetes secret
kubectl create secret tls vault-tls \
  --cert=vault-cert.pem \
  --key=vault-key.pem \
  -n vault

# Update Vault config for TLS
listener "tcp" {
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_cert_file = "/vault/tls/tls.crt"
  tls_key_file = "/vault/tls/tls.key"
}

RBAC Configuration

# vault-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vault-k8s-role
rules:
- apiGroups: [""]
  resources: ["serviceaccounts", "serviceaccounts/token"]
  verbs: ["get", "list", "watch", "create"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["clusterrolebindings"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-k8s-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vault-k8s-role
subjects:
- kind: ServiceAccount
  name: vault
  namespace: vault

Troubleshooting Guide

Common Issues

Issue: Vault is sealed after restart