Running Your Own Personal HashiCorp Vault on Local Kubernetes: A Security Engineer's Guide

Oct 21, 2025 min
#vault #kubernetes #security #devops #hashicorp

Introduction

Running HashiCorp Vault locally on Kubernetes is an excellent way to learn Vault’s capabilities, test credential management strategies, and develop secure applications without the complexity of a production deployment. This guide walks you through setting up a fully functional Vault instance on your local machine using Kubernetes, complete with auto-unseal, secret engines, and policy management.

Whether you’re evaluating Vault for your organization, developing security tools, or just want a secure local secret store, this setup provides a production-like environment for experimentation and development.

Prerequisites and Environment Setup

Required Tools

Before we begin, ensure you have the following tools installed:

# Check if tools are installed
docker --version          # Docker Desktop or Docker Engine
kubectl version --client  # Kubernetes CLI
helm version             # Helm 3.x
vault --version          # Vault CLI
# Install missing tools
# macOS
brew install kubectl helm vault
brew install --cask docker
# Linux
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
curl https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz | tar xz
wget https://releases.hashicorp.com/vault/1.15.0/vault_1.15.0_linux_amd64.zip

Kubernetes Cluster Options

Choose one of these local Kubernetes options:

Option 1: Docker Desktop (Recommended for beginners)

# Enable Kubernetes in Docker Desktop
# Settings → Kubernetes → Enable Kubernetes
# Wait for cluster to be ready
kubectl cluster-info

Option 2: kind (Kubernetes in Docker)

Create cluster with optimized configuration:

First, let’s set up a reusable kind configuration in your home directory that you can use for any Vault-related projects:

# Create kind config directory in home
mkdir -p ~/.config/kind

# Create a reusable Vault-optimized configuration
cat > ~/.config/kind/vault-cluster.yaml << 'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: vault-cluster
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 8200
    protocol: TCP
  extraMounts:
  - hostPath: ~/.config/vault-data
    containerPath: /vault-data
EOF
# Create the data directory in your home
mkdir -p ~/.config/vault-data

# Create the cluster using your personal configuration
kind create cluster --config ~/.config/kind/vault-cluster.yaml

# Verify cluster is ready
kubectl cluster-info --context kind-vault-cluster
kubectl get nodes

Benefits of this approach:

  • Reusable: Use the same config across multiple projects
  • Personal: Stored in your home directory, not tied to specific projects
  • Persistent: The ~/vault-data directory persists across cluster recreations
  • Organized: Keep all your kind configs in ~/.config/kind/

Configuration Management:

# List your kind configurations
ls ~/.config/kind/

# Create different configs for different purposes
cat > ~/.config/kind/multi-node-vault.yaml << 'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: vault-multi-node
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 8200
    protocol: TCP
- role: worker
- role: worker
EOF

# Use specific configurations as needed
kind create cluster --config ~/.config/kind/vault-cluster.yaml
kind create cluster --config ~/.config/kind/multi-node-vault.yaml

Cluster Configuration

For kind, use this configuration for better Vault performance:

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30000
    hostPort: 8200
    protocol: TCP
  extraMounts:
  - hostPath: ./vault-data
    containerPath: /vault-data

Basic Vault Installation

# Add HashiCorp Helm repository
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Create namespace
kubectl create namespace vault

# Install Vault in dev mode (quick start)
helm install vault hashicorp/vault \
  --namespace vault \
  --set "server.dev.enabled=true" \
  --set "injector.enabled=false" \
  --set "ui.enabled=true" \
  --set "ui.serviceType=NodePort" \
  --set "ui.serviceNodePort=30000"

# Wait for pod to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=vault -n vault --timeout=120s

# Check status
kubectl get all -n vault

Method 2: Manual Kubernetes Manifests

For more control, use custom manifests:

# vault-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: vault
---
# vault-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vault
  namespace: vault
spec:
  type: NodePort
  ports:
    - name: vault
      port: 8200
      targetPort: 8200
      nodePort: 30000
    - name: vault-internal
      port: 8201
      targetPort: 8201
  selector:
    app: vault
---
# vault-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.15.0
        ports:
        - containerPort: 8200
          name: vault
        - containerPort: 8201
          name: vault-internal
        env:
        - name: VAULT_DEV_ROOT_TOKEN_ID
          value: "root-token"
        - name: VAULT_DEV_LISTEN_ADDRESS
          value: "0.0.0.0:8200"
        - name: VAULT_ADDR
          value: "http://127.0.0.1:8200"
        volumeMounts:
        - name: vault-data
          mountPath: /vault/data
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
      volumes:
      - name: vault-data
        emptyDir: {}

Apply the manifests:

kubectl apply -f vault-namespace.yaml
kubectl apply -f vault-service.yaml
kubectl apply -f vault-deployment.yaml

Accessing Vault

# Port forward for CLI access
kubectl port-forward -n vault svc/vault 8200:8200 &

# Export Vault address
export VAULT_ADDR='http://127.0.0.1:8200'

# For dev mode, use the root token
export VAULT_TOKEN='root-token'

# Verify connection
vault status

# Access UI at http://localhost:8200/ui

Production-Like Setup with Raft Storage

For a more realistic setup, let’s configure Vault with Raft storage and auto-unseal:

Step 1: Create ConfigMap for Vault Configuration

# vault-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-config
  namespace: vault
data:
  vault.hcl: |
    ui = true
    
    listener "tcp" {
      address = "[::]:8200"
      cluster_address = "[::]:8201"
      tls_disable = 1
    }
    
    storage "raft" {
      path = "/vault/data"
      node_id = "vault-0"
    }
    
    seal "transit" {
      address = "http://vault-transit:8200"
      disable_renewal = "false"
      key_name = "autounseal"
      mount_path = "transit/"
      tls_skip_verify = "true"
    }
    
    service_registration "kubernetes" {
      namespace = "vault"
      pod_name = "vault-0"
    }
    
    api_addr = "http://vault-0.vault-internal:8200"
    cluster_addr = "https://vault-0.vault-internal:8201"

Step 2: Set Up Transit Vault for Auto-Unseal

# transit-vault.yaml
apiVersion: v1
kind: Service
metadata:
  name: vault-transit
  namespace: vault
spec:
  selector:
    app: vault-transit
  ports:
    - port: 8200
      targetPort: 8200
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-transit
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault-transit
  template:
    metadata:
      labels:
        app: vault-transit
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.15.0
        env:
        - name: VAULT_DEV_ROOT_TOKEN_ID
          value: "transit-root-token"
        - name: VAULT_DEV_LISTEN_ADDRESS
          value: "0.0.0.0:8200"
        ports:
        - containerPort: 8200
        command: ["vault", "server", "-dev", "-dev-root-token-id=transit-root-token"]

Step 3: StatefulSet for Vault with Raft

# vault-statefulset.yaml
apiVersion: v1
kind: Service
metadata:
  name: vault-internal
  namespace: vault
spec:
  clusterIP: None
  selector:
    app: vault
  ports:
    - name: vault
      port: 8200
    - name: vault-internal
      port: 8201
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
  namespace: vault
spec:
  serviceName: vault-internal
  replicas: 1
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - name: vault
        image: hashicorp/vault:1.15.0
        ports:
        - containerPort: 8200
          name: vault
        - containerPort: 8201
          name: vault-internal
        env:
        - name: VAULT_ADDR
          value: "http://127.0.0.1:8200"
        - name: VAULT_API_ADDR
          value: "http://vault-0.vault-internal:8200"
        - name: VAULT_CLUSTER_ADDR
          value: "https://vault-0.vault-internal:8201"
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: vault-data
          mountPath: /vault/data
        - name: vault-config
          mountPath: /vault/config
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        command: ["vault", "server", "-config=/vault/config/vault.hcl"]
      volumes:
      - name: vault-config
        configMap:
          name: vault-config
  volumeClaimTemplates:
  - metadata:
      name: vault-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

Step 4: Initialize and Unseal

# Apply all configurations
kubectl apply -f transit-vault.yaml
kubectl apply -f vault-config.yaml
kubectl apply -f vault-statefulset.yaml

# Wait for pods
kubectl wait --for=condition=ready pod -l app=vault-transit -n vault --timeout=120s
kubectl wait --for=condition=ready pod vault-0 -n vault --timeout=120s

# Configure transit vault for auto-unseal
kubectl exec -n vault deployment/vault-transit -- vault login transit-root-token
kubectl exec -n vault deployment/vault-transit -- vault secrets enable transit
kubectl exec -n vault deployment/vault-transit -- vault write -f transit/keys/autounseal

# Initialize main vault
kubectl exec -n vault vault-0 -- vault operator init -recovery-shares=5 -recovery-threshold=3

# Save the root token and recovery keys securely!

Configuring Vault for Development

Enable Essential Secret Engines

# Key-Value secrets engine
vault secrets enable -version=2 kv
vault secrets enable -path=secret kv-v2

# AWS secrets engine
vault secrets enable aws
vault write aws/config/root \
  access_key=$AWS_ACCESS_KEY_ID \
  secret_key=$AWS_SECRET_ACCESS_KEY \
  region=us-east-1

# Database secrets engine
vault secrets enable database

# PKI secrets engine
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki

# Transit for encryption as a service
vault secrets enable transit

Configure Authentication Methods

# Kubernetes auth
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

# AppRole auth
vault auth enable approle
vault policy write my-app - <<EOF
path "secret/data/my-app/*" {
  capabilities = ["read"]
}
EOF

vault write auth/approle/role/my-app \
  token_policies="my-app" \
  token_ttl=1h \
  token_max_ttl=4h

# Userpass for local testing
vault auth enable userpass
vault write auth/userpass/users/testuser \
  password=testpassword \
  policies=default

Create Development Policies

# Developer policy
vault policy write developer - <<EOF
# Read and write to secret/data/dev
path "secret/data/dev/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# List secret/metadata
path "secret/metadata/*" {
  capabilities = ["list"]
}

# Create and manage AWS credentials
path "aws/creds/developer" {
  capabilities = ["read"]
}

# Encrypt/decrypt with transit
path "transit/encrypt/app-key" {
  capabilities = ["update"]
}

path "transit/decrypt/app-key" {
  capabilities = ["update"]
}
EOF

# Security engineer policy
vault policy write security-engineer - <<EOF
# Full access to secret engine
path "secret/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage AWS roles
path "aws/roles/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage database connections
path "database/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Audit log access
path "sys/audit" {
  capabilities = ["read", "list"]
}

# Policy management
path "sys/policies/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOF

Integration Examples

Python Application with Vault

# vault_client.py
import hvac
import os
from kubernetes import client, config

class VaultClient:
    def __init__(self, vault_addr='http://localhost:8200'):
        self.vault_addr = vault_addr
        self.client = None
        self._authenticate()
    
    def _authenticate(self):
        """Authenticate based on environment"""
        # Try Kubernetes auth first
        if os.path.exists('/var/run/secrets/kubernetes.io/serviceaccount/token'):
            self._k8s_auth()
        # Fall back to token auth
        elif os.environ.get('VAULT_TOKEN'):
            self.client = hvac.Client(
                url=self.vault_addr,
                token=os.environ['VAULT_TOKEN']
            )
        else:
            raise Exception("No authentication method available")
    
    def _k8s_auth(self):
        """Authenticate using Kubernetes service account"""
        with open('/var/run/secrets/kubernetes.io/serviceaccount/token', 'r') as f:
            jwt = f.read()
        
        self.client = hvac.Client(url=self.vault_addr)
        self.client.auth.kubernetes.login(
            role='my-app',
            jwt=jwt
        )
    
    def get_secret(self, path):
        """Retrieve secret from KV v2"""
        response = self.client.secrets.kv.v2.read_secret_version(
            path=path,
            mount_point='secret'
        )
        return response['data']['data']
    
    def get_aws_creds(self, role='developer'):
        """Get temporary AWS credentials"""
        response = self.client.read(f'aws/creds/{role}')
        return {
            'access_key': response['data']['access_key'],
            'secret_key': response['data']['secret_key'],
            'session_token': response['data']['security_token']
        }
    
    def encrypt(self, plaintext, key_name='app-key'):
        """Encrypt data using Transit engine"""
        response = self.client.write(
            f'transit/encrypt/{key_name}',
            plaintext=plaintext
        )
        return response['data']['ciphertext']
    
    def decrypt(self, ciphertext, key_name='app-key'):
        """Decrypt data using Transit engine"""
        response = self.client.write(
            f'transit/decrypt/{key_name}',
            ciphertext=ciphertext
        )
        return response['data']['plaintext']

# Example usage
if __name__ == '__main__':
    vault = VaultClient()
    
    # Store a secret
    vault.client.secrets.kv.v2.create_or_update_secret(
        path='my-app/config',
        secret={'api_key': 'secret-value'},
        mount_point='secret'
    )
    
    # Retrieve the secret
    config = vault.get_secret('my-app/config')
    print(f"API Key: {config['api_key']}")
    
    # Get AWS credentials
    aws_creds = vault.get_aws_creds()
    print(f"AWS Access Key: {aws_creds['access_key']}")
    
    # Encrypt sensitive data
    encrypted = vault.encrypt('sensitive-data')
    print(f"Encrypted: {encrypted}")
    
    # Decrypt it back
    decrypted = vault.decrypt(encrypted)
    print(f"Decrypted: {decrypted}")

Kubernetes Pod with Vault Integration

# app-deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "my-app"
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/my-app/config"
        vault.hashicorp.com/agent-inject-template-config: |
          {{ with secret "secret/data/my-app/config" -}}
          export API_KEY="{{ .Data.data.api_key }}"
          export DB_PASSWORD="{{ .Data.data.db_password }}"
          {{- end }}
    spec:
      serviceAccountName: my-app
      containers:
      - name: app
        image: my-app:latest
        command: ['sh', '-c']
        args: ['source /vault/secrets/config && exec my-app']
        env:
        - name: VAULT_ADDR
          value: "http://vault.vault.svc:8200"

Shell Script Integration

#!/bin/bash
# vault-wrapper.sh - Wrapper script for using Vault credentials

# Setup
VAULT_ADDR="http://localhost:8200"
VAULT_TOKEN=${VAULT_TOKEN:-$(cat ~/.vault-token)}

# Function to get AWS credentials
get_aws_creds() {
    local role="${1:-developer}"
    vault read -format=json aws/creds/$role | jq -r '.data | 
        "export AWS_ACCESS_KEY_ID=\(.access_key)\n
         export AWS_SECRET_ACCESS_KEY=\(.secret_key)\n
         export AWS_SESSION_TOKEN=\(.security_token)"'
}

# Function to get secret
get_secret() {
    local path="$1"
    vault kv get -format=json secret/$path | jq -r '.data.data'
}

# Function to encrypt file
encrypt_file() {
    local file="$1"
    local key="${2:-app-key}"
    
    base64 < "$file" | vault write -format=json transit/encrypt/$key plaintext=- | \
        jq -r '.data.ciphertext' > "$file.enc"
}

# Function to decrypt file
decrypt_file() {
    local file="$1"
    local key="${2:-app-key}"
    
    vault write -format=json transit/decrypt/$key ciphertext=@"$file" | \
        jq -r '.data.plaintext' | base64 -d
}

# Example usage
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    # Get AWS credentials and export them
    eval $(get_aws_creds developer)
    
    # Get database password
    DB_PASSWORD=$(get_secret my-app/config | jq -r '.db_password')
    
    # Encrypt a file
    encrypt_file sensitive-data.txt
    
    # Run command with credentials
    aws s3 ls
fi

Backup and Restore

Automated Backup Script

#!/bin/bash
# vault-backup.sh

BACKUP_DIR="/tmp/vault-backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/vault-backup-$TIMESTAMP.snap"

# Create backup directory
mkdir -p $BACKUP_DIR

# Take snapshot
kubectl exec -n vault vault-0 -- vault operator raft snapshot save /tmp/backup.snap
kubectl cp vault/vault-0:/tmp/backup.snap $BACKUP_FILE

# Encrypt backup with GPG
gpg --symmetric --cipher-algo AES256 $BACKUP_FILE

# Upload to S3 (optional)
aws s3 cp $BACKUP_FILE.gpg s3://my-vault-backups/

# Clean up old backups (keep last 7 days)
find $BACKUP_DIR -name "vault-backup-*.snap*" -mtime +7 -delete

echo "Backup completed: $BACKUP_FILE.gpg"

Restore Process

#!/bin/bash
# vault-restore.sh

BACKUP_FILE="$1"

if [ -z "$BACKUP_FILE" ]; then
    echo "Usage: $0 <backup-file>"
    exit 1
fi

# Decrypt if needed
if [[ $BACKUP_FILE == *.gpg ]]; then
    gpg --decrypt $BACKUP_FILE > /tmp/restore.snap
    BACKUP_FILE="/tmp/restore.snap"
fi

# Copy to pod
kubectl cp $BACKUP_FILE vault/vault-0:/tmp/restore.snap

# Restore snapshot
kubectl exec -n vault vault-0 -- vault operator raft snapshot restore /tmp/restore.snap

echo "Restore completed from: $BACKUP_FILE"

Cloud Backup Solutions

For a personal Vault, backing up to the cloud provides durability and disaster recovery capabilities. Here are comprehensive solutions for different cloud providers:

AWS S3 Backup Strategy

Prerequisites Setup

# Create dedicated S3 bucket for Vault backups
aws s3 mb s3://your-vault-backups-$(uuidgen | tr '[:upper:]' '[:lower:]' | head -c 8)

# Enable versioning for backup protection
aws s3api put-bucket-versioning \
  --bucket your-vault-backups \
  --versioning-configuration Status=Enabled

# Configure server-side encryption
aws s3api put-bucket-encryption \
  --bucket your-vault-backups \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "alias/vault-backup-key"
        }
      }
    ]
  }'

# Set lifecycle policy to manage costs
aws s3api put-bucket-lifecycle-configuration \
  --bucket your-vault-backups \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "VaultBackupLifecycle",
        "Status": "Enabled",
        "Filter": {"Prefix": ""},
        "Transitions": [
          {
            "Days": 30,
            "StorageClass": "STANDARD_IA"
          },
          {
            "Days": 90,
            "StorageClass": "GLACIER"
          },
          {
            "Days": 365,
            "StorageClass": "DEEP_ARCHIVE"
          }
        ],
        "Expiration": {
          "Days": 2555
        }
      }
    ]
  }'

Enhanced Backup Script for AWS

#!/bin/bash
# vault-backup-aws.sh - Comprehensive AWS backup solution

set -euo pipefail

# Configuration
BACKUP_BUCKET="your-vault-backups"
BACKUP_PREFIX="vault-backups"
KMS_KEY_ID="alias/vault-backup-key"
BACKUP_DIR="/tmp/vault-backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
HOSTNAME=$(hostname)
RETENTION_DAYS=90

# Logging
LOG_FILE="/var/log/vault-backup.log"
exec 1> >(tee -a "$LOG_FILE")
exec 2>&1

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}

# Check prerequisites
check_prereqs() {
    log "Checking prerequisites..."
    
    # Check if kubectl can connect to cluster
    if ! kubectl cluster-info &>/dev/null; then
        log "ERROR: Cannot connect to Kubernetes cluster"
        exit 1
    fi
    
    # Check if Vault pod is running
    if ! kubectl get pod -n vault vault-0 &>/dev/null; then
        log "ERROR: Vault pod vault-0 not found"
        exit 1
    fi
    
    # Check AWS credentials
    if ! aws sts get-caller-identity &>/dev/null; then
        log "ERROR: AWS credentials not configured"
        exit 1
    fi
    
    # Check if bucket exists
    if ! aws s3api head-bucket --bucket "$BACKUP_BUCKET" &>/dev/null; then
        log "ERROR: Backup bucket $BACKUP_BUCKET not accessible"
        exit 1
    fi
}

# Create snapshot
create_snapshot() {
    log "Creating Vault snapshot..."
    
    mkdir -p "$BACKUP_DIR"
    
    # Check Vault status first
    local vault_status
    vault_status=$(kubectl exec -n vault vault-0 -- vault status -format=json 2>/dev/null || echo '{}')
    
    if [[ $(echo "$vault_status" | jq -r '.sealed // true') == "true" ]]; then
        log "WARNING: Vault is sealed, snapshot may be incomplete"
    fi
    
    # Create snapshot with retry logic
    local retry_count=0
    local max_retries=3
    
    while [ $retry_count -lt $max_retries ]; do
        if kubectl exec -n vault vault-0 -- vault operator raft snapshot save /tmp/backup.snap; then
            break
        fi
        
        retry_count=$((retry_count + 1))
        log "Snapshot creation failed, attempt $retry_count/$max_retries"
        sleep 10
    done
    
    if [ $retry_count -eq $max_retries ]; then
        log "ERROR: Failed to create snapshot after $max_retries attempts"
        exit 1
    fi
    
    # Copy snapshot from pod
    kubectl cp vault/vault-0:/tmp/backup.snap "$BACKUP_DIR/vault-snapshot-$TIMESTAMP.snap"
    
    # Verify snapshot file
    if [[ ! -f "$BACKUP_DIR/vault-snapshot-$TIMESTAMP.snap" ]] || [[ ! -s "$BACKUP_DIR/vault-snapshot-$TIMESTAMP.snap" ]]; then
        log "ERROR: Snapshot file is missing or empty"
        exit 1
    fi
    
    log "Snapshot created successfully: vault-snapshot-$TIMESTAMP.snap"
}

# Backup configuration
backup_config() {
    log "Backing up Vault configuration..."
    
    # Export Vault configuration
    kubectl get configmap -n vault vault-config -o yaml > "$BACKUP_DIR/vault-config-$TIMESTAMP.yaml"
    kubectl get secret -n vault vault-tls -o yaml > "$BACKUP_DIR/vault-tls-$TIMESTAMP.yaml" 2>/dev/null || true
    kubectl get statefulset -n vault vault -o yaml > "$BACKUP_DIR/vault-statefulset-$TIMESTAMP.yaml"
    
    # Create manifest archive
    tar -czf "$BACKUP_DIR/vault-manifests-$TIMESTAMP.tar.gz" -C "$BACKUP_DIR" \
        vault-config-$TIMESTAMP.yaml \
        vault-tls-$TIMESTAMP.yaml \
        vault-statefulset-$TIMESTAMP.yaml 2>/dev/null || true
    
    log "Configuration backup completed"
}

# Create metadata
create_metadata() {
    log "Creating backup metadata..."
    
    cat > "$BACKUP_DIR/metadata-$TIMESTAMP.json" << EOF
{
  "timestamp": "$TIMESTAMP",
  "hostname": "$HOSTNAME",
  "vault_version": "$(kubectl get pod -n vault vault-0 -o jsonpath='{.spec.containers[0].image}' | cut -d: -f2)",
  "kubernetes_version": "$(kubectl version --short --client | grep 'Client Version' | cut -d: -f2 | tr -d ' ')",
  "backup_type": "full",
  "files": [
    "vault-snapshot-$TIMESTAMP.snap",
    "vault-manifests-$TIMESTAMP.tar.gz"
  ],
  "checksum": "$(sha256sum "$BACKUP_DIR/vault-snapshot-$TIMESTAMP.snap" | cut -d' ' -f1)"
}
EOF
}

# Encrypt and upload to S3
upload_to_s3() {
    log "Encrypting and uploading to S3..."
    
    # Encrypt snapshot
    gpg --batch --yes --symmetric --cipher-algo AES256 \
        --passphrase-file <(echo "$GPG_PASSPHRASE") \
        "$BACKUP_DIR/vault-snapshot-$TIMESTAMP.snap"
    
    # Upload files to S3 with server-side encryption
    aws s3 cp "$BACKUP_DIR/vault-snapshot-$TIMESTAMP.snap.gpg" \
        "s3://$BACKUP_BUCKET/$BACKUP_PREFIX/snapshots/" \
        --server-side-encryption aws:kms \
        --ssekms-key-id "$KMS_KEY_ID" \
        --metadata "hostname=$HOSTNAME,timestamp=$TIMESTAMP,type=snapshot"
    
    aws s3 cp "$BACKUP_DIR/vault-manifests-$TIMESTAMP.tar.gz" \
        "s3://$BACKUP_BUCKET/$BACKUP_PREFIX/manifests/" \
        --server-side-encryption aws:kms \
        --ssekms-key-id "$KMS_KEY_ID"
    
    aws s3 cp "$BACKUP_DIR/metadata-$TIMESTAMP.json" \
        "s3://$BACKUP_BUCKET/$BACKUP_PREFIX/metadata/" \
        --server-side-encryption aws:kms \
        --ssekms-key-id "$KMS_KEY_ID"
    
    log "Upload completed successfully"
}

# Cleanup old backups
cleanup_old_backups() {
    log "Cleaning up old backups..."
    
    # Local cleanup
    find "$BACKUP_DIR" -name "vault-*" -mtime +7 -delete 2>/dev/null || true
    
    # S3 cleanup (older than retention period)
    local cutoff_date
    cutoff_date=$(date -d "$RETENTION_DAYS days ago" '+%Y%m%d')
    
    aws s3api list-objects-v2 \
        --bucket "$BACKUP_BUCKET" \
        --prefix "$BACKUP_PREFIX/" \
        --query "Contents[?LastModified<='${cutoff_date}T00:00:00.000Z'].Key" \
        --output text | while read -r key; do
            if [[ -n "$key" ]]; then
                aws s3 rm "s3://$BACKUP_BUCKET/$key"
                log "Deleted old backup: $key"
            fi
    done
}

# Verify backup integrity
verify_backup() {
    log "Verifying backup integrity..."
    
    # Download and verify the uploaded file
    aws s3 cp "s3://$BACKUP_BUCKET/$BACKUP_PREFIX/snapshots/vault-snapshot-$TIMESTAMP.snap.gpg" \
        "/tmp/verify-$TIMESTAMP.snap.gpg"
    
    # Compare checksums
    local original_checksum
    local uploaded_checksum
    
    original_checksum=$(sha256sum "$BACKUP_DIR/vault-snapshot-$TIMESTAMP.snap.gpg" | cut -d' ' -f1)
    uploaded_checksum=$(sha256sum "/tmp/verify-$TIMESTAMP.snap.gpg" | cut -d' ' -f1)
    
    if [[ "$original_checksum" == "$uploaded_checksum" ]]; then
        log "Backup integrity verified successfully"
        rm "/tmp/verify-$TIMESTAMP.snap.gpg"
    else
        log "ERROR: Backup integrity check failed"
        exit 1
    fi
}

# Send notification
send_notification() {
    local status=$1
    
    if [[ -n "${SLACK_WEBHOOK_URL:-}" ]]; then
        curl -X POST -H 'Content-type: application/json' \
            --data "{\"text\":\"Vault backup $status for $HOSTNAME at $TIMESTAMP\"}" \
            "$SLACK_WEBHOOK_URL" 2>/dev/null || true
    fi
    
    if [[ -n "${EMAIL_RECIPIENT:-}" ]]; then
        echo "Vault backup $status for $HOSTNAME at $TIMESTAMP" | \
            mail -s "Vault Backup $status" "$EMAIL_RECIPIENT" 2>/dev/null || true
    fi
}

# Main execution
main() {
    log "Starting Vault backup process..."
    
    # Check for required GPG passphrase
    if [[ -z "${GPG_PASSPHRASE:-}" ]]; then
        log "ERROR: GPG_PASSPHRASE environment variable not set"
        exit 1
    fi
    
    trap 'send_notification "FAILED"; exit 1' ERR
    
    check_prereqs
    create_snapshot
    backup_config
    create_metadata
    upload_to_s3
    verify_backup
    cleanup_old_backups
    
    send_notification "COMPLETED"
    log "Vault backup process completed successfully"
}

main "$@"

Google Cloud Storage Backup

#!/bin/bash
# vault-backup-gcp.sh - Google Cloud Storage backup solution

set -euo pipefail

# Configuration
GCS_BUCKET="your-vault-backups"
PROJECT_ID="your-project-id"
KMS_KEY="projects/$PROJECT_ID/locations/global/keyRings/vault-backup/cryptoKeys/vault-backup-key"
BACKUP_PREFIX="vault-backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Setup GCS bucket
setup_gcs() {
    # Create bucket with uniform bucket-level access
    gsutil mb -p "$PROJECT_ID" -l US gs://"$GCS_BUCKET" 2>/dev/null || true
    
    # Enable versioning
    gsutil versioning set on gs://"$GCS_BUCKET"
    
    # Set lifecycle policy
    cat > lifecycle.json << 'EOF'
{
  "rule": [
    {
      "action": {"type": "SetStorageClass", "storageClass": "NEARLINE"},
      "condition": {"age": 30}
    },
    {
      "action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
      "condition": {"age": 90}
    },
    {
      "action": {"type": "SetStorageClass", "storageClass": "ARCHIVE"},
      "condition": {"age": 365}
    },
    {
      "action": {"type": "Delete"},
      "condition": {"age": 2555}
    }
  ]
}
EOF
    
    gsutil lifecycle set lifecycle.json gs://"$GCS_BUCKET"
    rm lifecycle.json
}

# Upload with customer-managed encryption
upload_to_gcs() {
    log "Uploading to Google Cloud Storage..."
    
    # Upload snapshot with CMEK
    gsutil -o "GSUtil:encryption_key=$KMS_KEY" cp \
        "$BACKUP_DIR/vault-snapshot-$TIMESTAMP.snap.gpg" \
        "gs://$GCS_BUCKET/$BACKUP_PREFIX/snapshots/"
    
    # Upload manifests
    gsutil -o "GSUtil:encryption_key=$KMS_KEY" cp \
        "$BACKUP_DIR/vault-manifests-$TIMESTAMP.tar.gz" \
        "gs://$GCS_BUCKET/$BACKUP_PREFIX/manifests/"
    
    # Upload metadata
    gsutil -o "GSUtil:encryption_key=$KMS_KEY" cp \
        "$BACKUP_DIR/metadata-$TIMESTAMP.json" \
        "gs://$GCS_BUCKET/$BACKUP_PREFIX/metadata/"
    
    log "GCS upload completed"
}

# Include the main backup logic from AWS script...

Azure Blob Storage Backup

#!/bin/bash
# vault-backup-azure.sh - Azure Blob Storage backup solution

set -euo pipefail

# Configuration
STORAGE_ACCOUNT="yourvaultbackups"
CONTAINER_NAME="vault-backups"
RESOURCE_GROUP="vault-backup-rg"
KEY_VAULT_NAME="vault-backup-kv"
ENCRYPTION_KEY="vault-backup-key"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Setup Azure Blob Storage
setup_azure() {
    # Create resource group
    az group create --name "$RESOURCE_GROUP" --location eastus 2>/dev/null || true
    
    # Create storage account with encryption
    az storage account create \
        --name "$STORAGE_ACCOUNT" \
        --resource-group "$RESOURCE_GROUP" \
        --location eastus \
        --sku Standard_LRS \
        --encryption-services blob \
        --encryption-key-source Microsoft.Keyvault \
        --encryption-key-vault "https://$KEY_VAULT_NAME.vault.azure.net/" \
        --encryption-key-name "$ENCRYPTION_KEY" \
        2>/dev/null || true
    
    # Create container
    az storage container create \
        --name "$CONTAINER_NAME" \
        --account-name "$STORAGE_ACCOUNT" \
        --auth-mode login
    
    # Set lifecycle policy
    cat > lifecycle-policy.json << 'EOF'
{
  "rules": [
    {
      "name": "VaultBackupLifecycle",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["vault-backups/"]
        },
        "actions": {
          "baseBlob": {
            "tierToCool": {
              "daysAfterModificationGreaterThan": 30
            },
            "tierToArchive": {
              "daysAfterModificationGreaterThan": 90
            },
            "delete": {
              "daysAfterModificationGreaterThan": 2555
            }
          }
        }
      }
    }
  ]
}
EOF
    
    az storage account management-policy create \
        --account-name "$STORAGE_ACCOUNT" \
        --policy @lifecycle-policy.json \
        --resource-group "$RESOURCE_GROUP"
    
    rm lifecycle-policy.json
}

# Upload to Azure Blob Storage
upload_to_azure() {
    log "Uploading to Azure Blob Storage..."
    
    # Upload files with metadata
    az storage blob upload \
        --account-name "$STORAGE_ACCOUNT" \
        --container-name "$CONTAINER_NAME" \
        --name "snapshots/vault-snapshot-$TIMESTAMP.snap.gpg" \
        --file "$BACKUP_DIR/vault-snapshot-$TIMESTAMP.snap.gpg" \
        --metadata "hostname=$HOSTNAME" "timestamp=$TIMESTAMP" "type=snapshot" \
        --auth-mode login
    
    az storage blob upload \
        --account-name "$STORAGE_ACCOUNT" \
        --container-name "$CONTAINER_NAME" \
        --name "manifests/vault-manifests-$TIMESTAMP.tar.gz" \
        --file "$BACKUP_DIR/vault-manifests-$TIMESTAMP.tar.gz" \
        --auth-mode login
    
    az storage blob upload \
        --account-name "$STORAGE_ACCOUNT" \
        --container-name "$CONTAINER_NAME" \
        --name "metadata/metadata-$TIMESTAMP.json" \
        --file "$BACKUP_DIR/metadata-$TIMESTAMP.json" \
        --auth-mode login
    
    log "Azure upload completed"
}

# Include the main backup logic from AWS script...

Multi-Cloud Backup Strategy

#!/bin/bash
# vault-backup-multicloud.sh - Backup to multiple cloud providers

# Configuration for multi-cloud redundancy
BACKUP_PROVIDERS=("aws" "gcp" "azure")
BACKUP_DIR="/tmp/vault-backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Provider-specific configurations
declare -A PROVIDER_CONFIGS
PROVIDER_CONFIGS[aws]="bucket=your-vault-backups,region=us-east-1"
PROVIDER_CONFIGS[gcp]="bucket=your-vault-backups,project=your-project-id"
PROVIDER_CONFIGS[azure]="account=yourvaultbackups,container=vault-backups"

backup_to_provider() {
    local provider=$1
    log "Starting backup to $provider..."
    
    case $provider in
        "aws")
            source vault-backup-aws.sh
            upload_to_s3
            ;;
        "gcp")
            source vault-backup-gcp.sh
            upload_to_gcs
            ;;
        "azure")
            source vault-backup-azure.sh
            upload_to_azure
            ;;
    esac
    
    log "Backup to $provider completed"
}

# Execute parallel backups
main() {
    log "Starting multi-cloud backup process..."
    
    # Create snapshot once
    create_snapshot
    backup_config
    create_metadata
    
    # Backup to all providers in parallel
    for provider in "${BACKUP_PROVIDERS[@]}"; do
        backup_to_provider "$provider" &
    done
    
    # Wait for all uploads to complete
    wait
    
    log "Multi-cloud backup process completed"
}

Automated Scheduling with Cron

# Add to crontab for automated backups
# Edit with: crontab -e

# Daily backup at 2 AM
0 2 * * * /home/user/scripts/vault-backup-aws.sh >> /var/log/vault-backup.log 2>&1

# Weekly full backup to all clouds (Sunday 3 AM)
0 3 * * 0 /home/user/scripts/vault-backup-multicloud.sh >> /var/log/vault-backup.log 2>&1

# Hourly snapshot during business hours
0 9-17 * * 1-5 /home/user/scripts/vault-snapshot-only.sh >> /var/log/vault-backup.log 2>&1

Kubernetes CronJob for Automated Backups

# vault-backup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: vault-backup
  namespace: vault
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  timeZone: "America/New_York"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: vault-backup
          containers:
          - name: backup
            image: vault-backup:latest
            env:
            - name: AWS_REGION
              value: "us-east-1"
            - name: BACKUP_BUCKET
              value: "your-vault-backups"
            - name: GPG_PASSPHRASE
              valueFrom:
                secretKeyRef:
                  name: backup-secrets
                  key: gpg-passphrase
            - name: SLACK_WEBHOOK_URL
              valueFrom:
                secretKeyRef:
                  name: backup-secrets
                  key: slack-webhook
            volumeMounts:
            - name: backup-script
              mountPath: /scripts
            - name: aws-credentials
              mountPath: /root/.aws
            command: ["/scripts/vault-backup-aws.sh"]
          volumes:
          - name: backup-script
            configMap:
              name: backup-scripts
              defaultMode: 0755
          - name: aws-credentials
            secret:
              secretName: aws-credentials
          restartPolicy: OnFailure
      backoffLimit: 3
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1

Cloud Restore Procedures

AWS S3 Restore

#!/bin/bash
# vault-restore-aws.sh

BACKUP_BUCKET="your-vault-backups"
RESTORE_TIMESTAMP="$1"

if [[ -z "$RESTORE_TIMESTAMP" ]]; then
    echo "Usage: $0 <timestamp>"
    echo "Available backups:"
    aws s3 ls "s3://$BACKUP_BUCKET/vault-backups/snapshots/" | grep ".snap.gpg"
    exit 1
fi

# Download and decrypt
mkdir -p /tmp/restore
aws s3 cp "s3://$BACKUP_BUCKET/vault-backups/snapshots/vault-snapshot-$RESTORE_TIMESTAMP.snap.gpg" \
    "/tmp/restore/"

# Decrypt
gpg --batch --yes --decrypt \
    --passphrase-file <(echo "$GPG_PASSPHRASE") \
    "/tmp/restore/vault-snapshot-$RESTORE_TIMESTAMP.snap.gpg" > \
    "/tmp/restore/vault-snapshot-$RESTORE_TIMESTAMP.snap"

# Verify integrity
aws s3 cp "s3://$BACKUP_BUCKET/vault-backups/metadata/metadata-$RESTORE_TIMESTAMP.json" \
    "/tmp/restore/"

expected_checksum=$(jq -r '.checksum' "/tmp/restore/metadata-$RESTORE_TIMESTAMP.json")
actual_checksum=$(sha256sum "/tmp/restore/vault-snapshot-$RESTORE_TIMESTAMP.snap" | cut -d' ' -f1)

if [[ "$expected_checksum" != "$actual_checksum" ]]; then
    echo "ERROR: Checksum mismatch!"
    exit 1
fi

# Restore to Vault
kubectl cp "/tmp/restore/vault-snapshot-$RESTORE_TIMESTAMP.snap" \
    vault/vault-0:/tmp/restore.snap

kubectl exec -n vault vault-0 -- vault operator raft snapshot restore /tmp/restore.snap

echo "Restore completed successfully"

Disaster Recovery Runbook

# Vault Disaster Recovery Runbook

## Emergency Contacts
- Security Team: [email protected]
- On-call Engineer: +1-555-ONCALL

## Recovery Scenarios

### Scenario 1: Complete Cluster Loss
1. **Assessment** (5 minutes)
   - Verify Kubernetes cluster status
   - Check cloud provider status page
   - Confirm backup availability

2. **Environment Recreation** (15 minutes)
   ```bash
   # Recreate kind cluster
   kind create cluster --config ~/.config/kind/vault-cluster.yaml
   
   # Restore Vault manifests
   aws s3 cp s3://your-vault-backups/vault-backups/manifests/vault-manifests-<timestamp>.tar.gz .
   tar -xzf vault-manifests-<timestamp>.tar.gz
   kubectl apply -f vault-statefulset-<timestamp>.yaml

  1. Data Restoration (10 minutes)

    # Use latest backup
./vault-restore-aws.sh $(aws s3 ls s3://your-vault-backups/vault-backups/snapshots/ | tail -1 | awk '{print $4}' | sed 's/vault-snapshot-//;s/.snap.gpg//')

  2. Verification (10 minutes)

    • Test Vault accessibility
    • Verify secret engines
    • Confirm authentication methods

Scenario 2: Data Corruption

  1. Stop all write operations
  2. Identify last good backup
  3. Restore from point-in-time backup
  4. Verify data integrity

Scenario 3: Ransomware/Security Incident

  1. Isolate affected systems
  2. Use offline backup copies
  3. Rebuild from clean state
  4. Implement additional security measures

Recovery Testing

  • Monthly: Test backup restoration process
  • Quarterly: Full disaster recovery simulation
  • Annually: Cross-region recovery test

### Security Considerations for Cloud Backups

#### Encryption at Rest and in Transit
```bash
# Generate strong encryption key for backups
openssl rand -base64 32 > ~/.config/vault-backup-key

# Use strong GPG encryption
export GPG_PASSPHRASE=$(openssl rand -base64 32)

# Encrypt before upload
gpg --batch --yes --symmetric --cipher-algo AES256 \
    --passphrase-file <(echo "$GPG_PASSPHRASE") \
    vault-snapshot.snap

Access Control and Monitoring

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VaultBackupAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/VaultBackupRole"
      },
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::your-vault-backups/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        },
        "IpAddress": {
          "aws:SourceIp": ["YOUR_HOME_IP/32", "YOUR_VPN_CIDR"]
        }
      }
    }
  ]
}

Backup Verification and Testing

#!/bin/bash
# verify-backup-integrity.sh

# Monthly backup verification job
verify_monthly_backups() {
    local current_month=$(date +%Y%m)
    
    # List all backups for current month
    aws s3api list-objects-v2 \
        --bucket "$BACKUP_BUCKET" \
        --prefix "vault-backups/snapshots/vault-snapshot-${current_month}" \
        --query 'Contents[].Key' \
        --output text | while read -r backup_key; do
        
        local timestamp=$(basename "$backup_key" | sed 's/vault-snapshot-//;s/.snap.gpg//')
        
        # Download and verify each backup
        download_and_verify_backup "$timestamp"
        
        # Test restore in isolated environment
        test_restore_process "$timestamp"
    done
}

test_restore_process() {
    local timestamp=$1
    
    # Create temporary test environment
    kind create cluster --name vault-test-restore
    
    # Restore backup to test cluster
    ./vault-restore-aws.sh "$timestamp"
    
    # Verify Vault functionality
    kubectl exec -n vault vault-0 -- vault status
    kubectl exec -n vault vault-0 -- vault kv list secret/ || true
    
    # Cleanup
    kind delete cluster --name vault-test-restore
    
    log "Backup verification completed for $timestamp"
}

This comprehensive cloud backup solution provides:

  1. Multi-cloud redundancy for disaster recovery
  2. Strong encryption at rest and in transit
  3. Automated scheduling with Kubernetes CronJobs
  4. Integrity verification and monitoring
  5. Detailed disaster recovery procedures
  6. Security best practices for cloud storage

Choose the cloud provider that best fits your needs, or implement the multi-cloud strategy for maximum redundancy. The automated scripts handle encryption, upload, verification, and cleanup, ensuring your personal Vault data is safely backed up to the cloud.

~jared gore

Tags

#vault #kubernetes #security #devops #hashicorp