Core services I use daily: AKS (managed Kubernetes), ACR (container registry — stores Docker images that AKS pulls), Azure DevOps (CI/CD pipelines — builds images, deploys to AKS), Azure Key Vault (secrets/certs — mounted into pods via CSI driver), Virtual Network and NSGs (network isolation), Application Gateway (L7 load balancer with WAF, terminates SSL), Azure Monitor + Log Analytics (logs and metrics), Managed Identity (pod-to-Azure authentication without credentials). Integration flow: Developer commits code → Azure DevOps pipeline triggers → builds Docker image → pushes to ACR → deploys to AKS via helm upgrade → AKS pulls image from ACR using Managed Identity (no credentials stored) → pods access Key Vault secrets via Workload Identity → Application Gateway routes external traffic to pods → Azure Monitor collects all logs and metrics → alerts to PagerDuty.

Application Gateway is Azure Layer 7 load balancer. Features: SSL termination (offload TLS processing from pods), URL-based routing (/api → backend API pods, /app → frontend pods), WAF (Web Application Firewall — blocks OWASP Top 10 attacks), health probes (automatically removes unhealthy backends), session affinity, auto-scaling. AGIC (Application Gateway Ingress Controller): a controller pod running inside AKS. Watches Kubernetes Ingress resources. Translates them into Application Gateway configuration — creates listeners, routing rules, backend pools automatically. When you create a Kubernetes Ingress resource with class azure/application-gateway, AGIC automatically programs the Application Gateway. No manual App Gateway configuration needed. Benefit over NGINX Ingress: App Gateway is managed by Azure, auto-scales, includes WAF. NGINX needs you to manage the ingress controller pods. Setup: az aks enable-addons --addons ingress-appgw --appgw-id /subscriptions/.../applicationGateways/myAppGW. App Gateway must be in same VNet as AKS (or peered VNet).

NSG (Network Security Group) is a stateful firewall at the subnet or NIC level. Rules: priority (100-4096, lower = higher priority), source IP/range, destination IP/range, port, protocol, allow/deny. For AKS: Subnet NSG rules needed: allow port 443 inbound for HTTPS traffic to Application Gateway subnet. Allow port 10250 between nodes (kubelet communication). Allow port 2379-2380 for etcd. Allow 30000-32767 for NodePort services. Deny everything else inbound. Outbound: allow 443 to ACR, Key Vault, Azure APIs (use service tags: AzureContainerRegistry, AzureKeyVault, AzureMonitor). Deny outbound to internet from production nodes (use private endpoints for ACR and Key Vault). Best practice: manage NSGs in Terraform. Never manually modify NSGs in production — changes must go through Git PR process. az network nsg rule list --nsg-name myNSG --resource-group myRG shows current rules. NSG flow logs: enable for security audit — logs every allowed/denied connection to Log Analytics.

Azure DevOps Pipeline YAML flow: trigger: branches: include: [main] → pipeline starts on merge to main. Stage 1 Build: checkout code, dotnet build or mvn package or npm install && npm build, run unit tests, SonarQube analysis (fail if quality gate fails), docker build --build-arg BUILD_ID=$(Build.BuildId), trivy image scan (fail on CRITICAL), docker push to ACR with tag $(Build.BuildId). Stage 2 Deploy Dev: dependsOn: Build. uses KubernetesManifest task or helm upgrade --install --namespace dev --set image.tag=$(Build.BuildId) --atomic --timeout 5m. Run smoke tests. Stage 3 Deploy Staging: auto-deploy after dev succeeds. Stage 4 Deploy Production: environment: production with approval check — release manager must approve. Service connections: connection from Azure DevOps to AKS (using Managed Identity or Service Principal). Variables: non-secret in Variable Groups (Library), secrets from Azure Key Vault linked to Variable Group. Artifact: Build.BuildId is the versioned tag — every build gets unique ID, traceable from pipeline to running pod.

Variable Groups (non-sensitive): Pipelines → Library → Variable Groups. Store: environment names, registry URLs, cluster names, feature flags. Reference in pipeline: variables: group: myapp-variables. Azure Key Vault linked Variable Group (for secrets): create Variable Group → link to Azure Key Vault → select secrets. Secrets appear as pipeline variables but are fetched from Key Vault at runtime — never stored in Azure DevOps. Reference: $(secret-name). Pipeline YAML variables vs Variable Groups: pipeline-level variables for things specific to that pipeline. Variable Groups for shared config across multiple pipelines. Secret masking: Azure DevOps automatically masks secrets in pipeline logs — $(my-secret) prints as ***. Environment variables: use env: section in pipeline steps to inject as environment variables. Best practice: never hardcode any value that might change between environments. Use variables for everything: image tag, namespace, cluster name, replica count. Keep secrets exclusively in Key Vault — not even in Variable Group values, only Key Vault references.

Without Workload Identity: pods use Service Principal credentials stored as Kubernetes Secrets to authenticate to Azure. Those credentials can expire, can be leaked, and are a security risk. Workload Identity: federate a Kubernetes ServiceAccount with an Azure Managed Identity using OIDC. Pod uses the K8s SA → AKS injects a signed token → pod presents token to Azure AD → Azure AD validates via OIDC trust → issues short-lived Azure access token. No stored credentials anywhere. Setup: 1. Enable OIDC issuer on cluster: az aks update --enable-oidc-issuer --enable-workload-identity. 2. Create User Assigned Managed Identity: az identity create --name payment-identity --resource-group myRG. 3. Create Kubernetes ServiceAccount with annotation: kubectl annotate serviceaccount payment-sa azure.workload.identity/client-id=<identity-client-id>. 4. Create federated credential: az identity federated-credential create linking the K8s SA to the Managed Identity. 5. Grant Managed Identity access: az keyvault set-policy --name myVault --object-id <identity-object-id> --secret-permissions get list. In pod spec: serviceAccountName: payment-sa. The SDK (azure-identity DefaultAzureCredential) automatically uses the injected token.

Enable Container Insights: az aks enable-addons --addons monitoring --workspace-resource-id /subscriptions/.../workspaces/myLogAnalytics. What it collects automatically: node-level metrics (CPU, memory, disk, network), pod metrics (CPU, memory, restarts), container logs (stdout/stderr from all containers), Kubernetes events. Query with KQL in Log Analytics: KubePodInventory | where TimeGenerated > ago(1h) | where Namespace == "production" | where PodStatus == "Failed" | project TimeGenerated, PodName, ContainerStatus. ContainerLog | where LogEntry contains "ERROR" | summarize count() by Computer, bin(TimeGenerated, 5m). Alert rules: Alerts → New Alert Rule → use metric (e.g. Percentage CPU > 80%) or KQL query (e.g. failed pods count > 0 for 5 minutes). Action groups: send to email, SMS, PagerDuty, Slack webhook. Dashboards: Azure Monitor Workbooks provide pre-built Kubernetes dashboards. Custom Grafana dashboards: add Azure Monitor as Grafana data source, query ContainerInsights tables. Cost control: set Log Analytics data cap to avoid unexpected bills from verbose application logs.

AKS (Azure Kubernetes Service) is managed Kubernetes. Azure manages: control plane (API server, etcd, scheduler, controller manager) — free, zero maintenance. etcd backups — automatic. Control plane HA — built-in. Control plane upgrades — initiated by you, executed by Azure. You manage: worker nodes (VM size, count, OS updates), application deployments, networking config, security policies. Key benefit: eliminates 30-40% of ops effort spent managing control plane in self-managed K8s. You pay only for worker node VMs.

A node pool is a group of VMs with the same configuration. The system node pool runs Kubernetes system components (CoreDNS, metrics-server). User node pools run your workloads. Reasons for multiple pools: workload isolation (keep noisy batch jobs away from latency-sensitive APIs), cost optimisation (standard VMs for most workloads, GPU VMs only in the GPU pool), OS separation (Linux pool for containers, Windows pool for .NET apps), scaling independence (scale web pool to 20 nodes during peak, batch pool stays at 2), taint/toleration-based scheduling (taint gpu-pool=true:NoSchedule, only GPU workloads tolerate it).

AGIC makes Azure Application Gateway act as the Kubernetes Ingress controller. Architecture: Application Gateway (public IP, WAF, SSL termination) → AGIC controller (pod in AKS) → Routes to Kubernetes Services → Pods. Setup: 1. Create Application Gateway in the same VNet as AKS (or peered VNet). 2. Enable AGIC addon: az aks enable-addons --name myAKS --resource-group myRG --addons ingress-appgw --appgw-id /subscriptions/.../applicationGateways/myAppGW. 3. Create Ingress resource with class annotation: kubernetes.io/ingress.class: azure/application-gateway. 4. For SSL: upload certificate to App Gateway, reference in Ingress: appgw.ingress.kubernetes.io/ssl-certificate: my-cert. AGIC watches Ingress resources and automatically updates App Gateway backend pools, HTTP settings, and routing rules. One App Gateway can serve multiple applications using path-based routing.

Two approaches: cert-manager (automated) or manual. cert-manager approach (recommended for production): Install cert-manager helm chart. Create ClusterIssuer pointing to Let's Encrypt: apiVersion: cert-manager.io/v1, kind: ClusterIssuer, spec.acme.server: https://acme-v02.api.letsencrypt.org/directory. Create Certificate resource or add cert-manager annotations to Ingress: cert-manager.io/cluster-issuer: letsencrypt-prod. cert-manager automatically requests the cert, stores it as a K8s Secret, and renews it before expiry. Manual approach: create TLS secret: kubectl create secret tls myapp-tls --cert=tls.crt --key=tls.key -n production. Reference in Ingress: spec.tls: - hosts: [app.example.com], secretName: myapp-tls. For App Gateway: upload PFX certificate to Key Vault, reference via AGIC annotation. For NGINX ingress: the secret is used directly. Monitor expiry: kubectl get certificate -A shows READY status and expiry dates.

The complete pipeline for each app: Terraform manages the infrastructure layer: AKS cluster, ACR, Key Vault, networking. All in code, state in Azure Blob. Jenkins CI pipeline: trigger on Git commit → checkout code → docker build → trivy scan → docker push to ACR → update Helm values file with new image tag → commit back to Git → ArgoCD detects the change. Helm chart per application: values.yaml contains image tag, replica count, resource limits, environment variables. Each app has its own Helm release. kubectl apply or helm upgrade --install --atomic --timeout 5m --namespace app-namespace. For 10 apps: define each as a separate Helm release. Use ArgoCD App-of-Apps pattern: one root ArgoCD Application points to a directory of 10 Application manifests. ArgoCD syncs all 10. Namespace isolation: each app gets its own namespace. Network policies restrict cross-namespace communication. RBAC per namespace: dev team only accesses their app namespace.

Pre-upgrade: az aks get-upgrades --name myAKS --resource-group myRG shows available versions. Read the release notes for breaking changes. Test in dev/staging first. Raise Change Request, get approvals, schedule maintenance window. Verify application backups and rollback plan. Upgrade process: Step 1 — upgrade control plane: az aks upgrade --name myAKS --resource-group myRG --kubernetes-version 1.29.0 --control-plane-only. Verify control plane: az aks show shows new version, kubectl get nodes shows control plane upgraded. Step 2 — upgrade node pools one at a time: az aks nodepool upgrade --name userpool --cluster-name myAKS --resource-group myRG --kubernetes-version 1.29.0. Kubernetes cordons and drains each node, creates a new node with the new version, moves pods. Step 3 — validate: kubectl get nodes all nodes show new version, kubectl get pods -A all pods running, run smoke tests against all critical endpoints. Monitor for 30 minutes before marking complete. Important: minor version skew of 1 is allowed (control plane 1.29, nodes 1.28). Never skip minor versions.

Audit from cluster side: kubectl get rolebindings,clusterrolebindings -A -o wide shows every binding — who has what access where. For a specific user: kubectl auth can-i --list --as=user@company.com --namespace production shows everything they can do. For a group: kubectl auth can-i --list --as-group=devteam. Check if specific action allowed: kubectl auth can-i delete pods --as=user@company.com -n production. Audit from Azure AD side: az role assignment list --all shows all Azure RBAC assignments. For AKS specifically: az aks show shows AAD integration status, admin groups. Regular access review process: quarterly, export all RoleBindings and ClusterRoleBindings, review with team leads, remove any stale accounts or over-privileged roles. Automate detection: Azure AD access reviews can be scheduled to require each user to re-confirm their access. Enable AKS audit logging: all kubectl commands logged to Log Analytics. Query: AzureActivity | where OperationNameValue contains "kubectl" | project Caller, OperationNameValue, ResultType.

Zero trust means: never trust, always verify. No implicit trust based on network location. Every pod, every request must be explicitly authorised. Layers: 1. Network: Kubernetes NetworkPolicies — deny all traffic by default, explicitly allow only what is needed. apiVersion: networking.k8s.io/v1, kind: NetworkPolicy, spec.podSelector: {}, spec.policyTypes: [Ingress, Egress], spec.ingress: [] (deny all). Then add explicit allow policies for required paths. Use Cilium for L7 policies (allow only GET /api/v1/orders not all HTTP). 2. Identity: Workload Identity for pod-to-Azure authentication. Each service has its own Managed Identity. Payment service can only access payment Key Vault secrets. 3. mTLS between services: Istio or Linkerd encrypts all pod-to-pod traffic and mutually authenticates every call. 4. Image security: only images from ACR with valid scan results allowed (Azure Policy). 5. Runtime: Falco detects anomalous behaviour (shell in container, unexpected network call). 6. RBAC: least privilege for every service account. No default service account has meaningful permissions. Result: even if one pod is compromised, it cannot reach other services, cannot access credentials it was not explicitly granted, and the anomaly is detected.

You cannot resize an existing node pool in-place — the VM size is fixed at creation. The process is blue-green node pool replacement: Step 1: create a new node pool with the larger VM size: az aks nodepool add --name newpool --cluster-name myAKS --resource-group myRG --node-vm-size Standard_D8s_v3 --node-count 3. Step 2: taint the old node pool to prevent new pods scheduling on it: kubectl taint nodes -l agentpool=oldpool dedicated=old:NoSchedule. Step 3: cordon all old nodes: kubectl cordon node --selector agentpool=oldpool. Step 4: drain old nodes one at a time: kubectl drain nodename --ignore-daemonsets --delete-emptydir-data. Pods move to the new larger nodes. Step 5: verify all pods running on new nodes: kubectl get pods -o wide. Step 6: delete old node pool: az aks nodepool delete --name oldpool. For immediate relief while waiting: add more nodes (scale out) to distribute load: az aks nodepool scale --name existingpool --node-count 5. Check which pods are causing pressure: kubectl top nodes, kubectl top pods -A --sort-by=cpu.

Pod restarts cause brief downtime unless handled properly. Prevention strategies: Multiple replicas: minimum 2 replicas per service. One pod restarting leaves the other handling traffic. Kubernetes automatically routes traffic away from the restarting pod. PodDisruptionBudget: ensures at least 1 replica stays running during voluntary disruptions (node drain, upgrades): kubectl create pdb myapp-pdb --selector=app=myapp --min-available=1. Readiness probes: pod only receives traffic when the readiness probe passes. During restart and initialisation, traffic is not sent to the pod. livenessProbe restarts pods that are stuck but not serving. startupProbe gives slow-starting apps (Java) time to initialise before liveness checks begin. PreStop hook: add lifecycle.preStop.exec.command to gracefully drain connections before the pod terminates: sleep 15 to allow load balancer to drain. Resource limits: set appropriate memory limits to prevent OOMKilled. Set CPU requests correctly to prevent CPU throttling causing slowness that triggers liveness probe. For detecting the cause: kubectl get events -n production --sort-by=lastTimestamp shows restart reasons. kubectl logs mypod --previous shows what happened before the last restart.

AKS AAD integration creates a unified identity layer. How it works: Enable on cluster: az aks update --name myAKS --resource-group myRG --enable-aad --enable-azure-rbac --aad-admin-group-object-ids . Authentication flow: developer runs az aks get-credentials → kubectl command sent to AKS API server → API server redirects to Azure AD for authentication → developer authenticates with their corporate credentials (MFA supported) → Azure AD returns a token → AKS validates the token → checks Kubernetes RBAC permissions for the user/group. Define permissions in code (Terraform): create Azure RBAC role assignments: azurerm_role_assignment resource with role_definition_name "Azure Kubernetes Service RBAC Reader" scoped to the namespace. AD groups map to K8s roles: devteam@company.com group bound to edit ClusterRole in dev namespace only. ops@company.com group bound to cluster-admin ClusterRole. Why AD groups over individual users: when engineer joins the team → add to AD group → automatically gets all permissions. When they leave → remove from AD group → access revoked everywhere. No manual K8s RBAC changes needed. Audit: every API server call is logged with the Azure AD identity of the caller.

Step 1: create a Role (namespace-scoped) or ClusterRole (cluster-wide). For read-only pods in production namespace: apiVersion: rbac.authorization.k8s.io/v1, kind: Role, metadata.name: pod-reader, metadata.namespace: production, rules: - apiGroups: [""], resources: ["pods", "pods/log"], verbs: ["get", "list", "watch"]. Step 2: bind to an AD group with RoleBinding: kind: RoleBinding, metadata.namespace: production, subjects: - kind: Group, name: "support-team@company.com" (this is the AD group objectId or email), roleRef.kind: Role, roleRef.name: pod-reader. Step 3: verify: kubectl auth can-i get pods --as-group=support-team@company.com -n production → yes. kubectl auth can-i delete pods --as-group=support-team@company.com -n production → no. In Terraform (recommended — no manual kubectl): resource "kubernetes_role" and resource "kubernetes_role_binding". Store all RBAC definitions in a GitOps repo — any change goes through PR review and audit trail. Principle: production gets READ ONLY for most engineers. Only the deployment service account and ops team have write access.

Pod-level security: SecurityContext on every pod — runAsNonRoot: true (no root processes), readOnlyRootFilesystem: true (container cannot write to its own filesystem), allowPrivilegeEscalation: false, drop all capabilities then add only what is needed (capabilities.drop: [ALL], capabilities.add: [NET_BIND_SERVICE] if needed). Container image security: use distroless or minimal base images. Trivy or Grype scan in CI pipeline — fail build on CRITICAL/HIGH CVEs. Only pull from trusted registry (ACR) enforced via Azure Policy. Tag immutability on ACR — no overwriting image tags. Network security: NetworkPolicy deny-all default, explicit allow for required paths only. Network policies enforced by Cilium or Azure CNI. Secrets: never in environment variables directly, never in ConfigMaps. Use CSI Secrets Store driver to mount Key Vault secrets as files. Workload Identity for pod-to-Azure authentication. Admission control: Azure Policy for Kubernetes enforces: required resource limits, banned privileged containers, required security context fields, approved image registries only. Runtime: Falco for anomaly detection. These controls together mean a compromised pod has no root access, no writable filesystem, no ability to reach other services or Azure resources it was not explicitly granted, and any anomalous behaviour is detected within seconds.

Architecture I have implemented at HPE for a telecom production platform. Network: private AKS cluster (API server not reachable from internet), Azure CNI networking (pods get VNet IPs directly, no NAT), separate subnets for system node pool and user node pools, private endpoints for all PaaS services (ACR, Key Vault, Azure SQL, Storage). Identity: managed identity for the cluster, workload identity for pod-level Azure access (payment pods get their own identity to access Key Vault payment secrets, order pods get a different identity), zero service principal secrets. Node pools: system pool (3 nodes, Standard_D2s_v3, always on), user pool (auto-scale 2-20, Standard_D4s_v3), GPU pool (optional, only when needed for ML). Security: Azure Policy add-on enforces: no privileged containers, required resource limits, approved registry only (ACR). Defender for Containers for runtime security. Network policies (Calico) for pod-to-pod traffic control. Monitoring: Azure Monitor + Container Insights, Prometheus for custom metrics, Grafana dashboards, Alert rules for: node pressure, pod crash rate, certificate expiry. Disaster recovery: secondary cluster in paired region, Azure Traffic Manager for DNS failover, ArgoCD managing both clusters from same Git.

First: check why it is stuck. kubectl get nodes shows which nodes are on old version vs new. The upgrade process is: cordon node (stop new pods), drain node (move existing pods), delete old node, create new node, uncordon. Stuck at 60% usually means: one node cannot be drained. kubectl describe node stucknode -- look at Events. Common causes: Pod with no PodDisruptionBudget that cannot be evicted (kubectl drain fails with "cannot delete pods not managed by RC/RS/DaemonSet/Job" -- these are orphaned pods, manually delete them), PDB set too strictly (minAvailable = replicas, so drain cannot remove any pod -- temporarily relax PDB or add a replica), DaemonSet pods (kubectl drain --ignore-daemonsets handles these), local storage (kubectl drain --delete-emptydir-data). If drain keeps timing out: az aks nodepool upgrade --no-wait and manually drain the stuck node. Last resort: delete the stuck node manually (az vmss delete-instances) -- AKS will replace it with a new upgraded node. Never let an upgrade run unattended on production. Always monitor each node drain manually the first time you run an upgrade on a new cluster.

Pod: the smallest deployable unit in Kubernetes. Contains one or more containers that share the same network namespace and storage. Pods are ephemeral — if a Pod dies, it is gone. No self-healing. ReplicaSet: ensures a specified number of Pod replicas are always running. If a Pod dies, ReplicaSet creates a new one. But ReplicaSet alone has no rolling update capability. Deployment: manages ReplicaSets. Provides declarative updates — you tell it what you want, it figures out how to get there. Rolling updates: create a new ReplicaSet with the new version, scale it up while scaling down the old. Rollback: switch back to the previous ReplicaSet. In practice: you never create ReplicaSets directly. You always create Deployments, which manage ReplicaSets, which manage Pods.

HPA automatically scales the number of pod replicas based on observed CPU utilization or custom metrics. How it works: HPA controller queries metrics API every 15 seconds. If current metric > target: calculate desired replicas = ceil(current * (current_metric / target_metric)). Apply scale up/down with cooldown periods to prevent thrashing. Basic HPA on CPU: apiVersion: autoscaling/v2, kind: HorizontalPodAutoscaler, spec.scaleTargetRef.name: payment-api, spec.minReplicas: 2, spec.maxReplicas: 20, spec.metrics: type: Resource, resource.name: cpu, resource.target.type: Utilization, resource.target.averageUtilization: 70. Custom metrics HPA (Kafka lag): metrics: type: External, external.metric.name: kafka_consumer_lag, external.metric.target.value: 1000. This scales pods when Kafka consumer lag exceeds 1000 messages. Prerequisites: metrics-server installed (for CPU/memory), Prometheus adapter or KEDA (for custom metrics). Always set resource requests — HPA cannot calculate utilization percentage without them.

External traffic flow in AKS: 1. DNS resolves app.company.com to Azure Load Balancer public IP. 2. Azure Load Balancer receives traffic on port 443, forwards to NGINX Ingress Controller pod (NodePort or LoadBalancer service). 3. NGINX Ingress Controller reads Ingress rules — matches hostname and path, selects backend Service. 4. Service (ClusterIP) has a virtual IP. kube-proxy maintains iptables rules that distribute traffic to matching pod IPs. 5. Request reaches the pod. Pod-to-pod communication: pods get IPs from the pod CIDR. Azure CNI: pods get IPs from the actual VNet subnet — each pod is a first-class citizen in the network. Flannel/Calico: overlay network, pods get virtual IPs, NAT at the node boundary. Service discovery: pods reach other services by DNS name: payment-api.production.svc.cluster.local. CoreDNS resolves this to the ClusterIP. Key insight: the Service IP is virtual — it never actually appears on any network interface. It is implemented entirely via iptables/eBPF rules on each node.

OOMKilled means the container exceeded its memory limit and the Linux OOM killer terminated it. Diagnose: kubectl describe pod mypod — look for Last State: Terminated Reason: OOMKilled. kubectl top pod mypod --containers shows current memory usage. Check if memory limit is set too low: kubectl get pod mypod -o jsonpath=.spec.containers[0].resources. Check application logs before the kill: kubectl logs mypod --previous. Check if it is a memory leak: watch -n5 kubectl top pod mypod — if memory grows continuously without levelling off, it is a leak. Fix options: Increase memory limit: edit the Deployment resources.limits.memory. But first check if the increase is justified — if the app is well-behaved and just needs more memory, increase it. If memory is growing continuously, investigate the code — memory leak in the application. JVM applications: set explicit heap size with -Xmx flag to control maximum heap. Set limits 20-30% above maximum heap to account for JVM overhead (metaspace, thread stacks, native memory). Python applications: check for unbounded in-memory caches, accumulating lists. Add Prometheus memory metrics to track allocation over time.

PersistentVolume (PV): a piece of storage provisioned in the cluster. Can be backed by Azure Disk, Azure Files, NFS, or other storage. Has capacity, access mode, and reclaim policy. Like a physical disk that exists independently of any pod. PersistentVolumeClaim (PVC): a request for storage by a user/pod. Specifies: how much storage (5Gi), access mode (ReadWriteOnce, ReadWriteMany), and optionally a StorageClass. Kubernetes binds a PVC to a matching PV. Pod uses the PVC as a volume. StorageClass: defines the type of storage to dynamically provision. Azure Disk StorageClass provisions a new Azure Managed Disk when a PVC is created. No pre-provisioning needed. Access modes: ReadWriteOnce (one node at a time — Azure Disk), ReadWriteMany (multiple nodes simultaneously — Azure Files, NFS), ReadOnlyMany. StatefulSet + PVCs: volumeClaimTemplates in StatefulSet creates one PVC per pod replica. mongo-0 gets its own disk, mongo-1 gets its own disk. They are not shared. Lifecycle: pods are deleted, PVCs remain (data preserved). PVCs are deleted, PVs are released (reclaim policy: Retain = keep data, Delete = delete the disk).

Network policies use pod/namespace label selectors to control traffic. Default K8s: all pods can talk to all pods (no isolation). Default deny all: apiVersion: networking.k8s.io/v1, kind: NetworkPolicy, metadata.namespace: production, spec.podSelector: {} (matches all pods), spec.policyTypes: [Ingress, Egress] with empty rules. Now no pod can receive or send traffic. Explicit allow — only payment-api can talk to postgres: spec.podSelector.matchLabels: app=postgres, ingress.from.podSelector.matchLabels: app=payment-api, ports: 5432. Only ingress controller can receive external traffic: spec.podSelector.matchLabels: app=nginx-ingress, ingress.from.namespaceSelector.matchLabels: kubernetes.io/metadata.name=ingress-nginx. Allow DNS (critical — without this pods cannot resolve names): spec.podSelector: {}, egress.ports: 53/UDP and 53/TCP. Allow monitoring (Prometheus scrape): egress from monitoring namespace to all pods on port 9090. For L7 policies (HTTP path/method level): use Cilium CiliumNetworkPolicy — allows only POST to /api/v1/payment from order-service. Standard NetworkPolicy cannot do this.

Deployment: manages stateless pods. Pods are interchangeable, get random names (app-7d9f-xyz), can be scheduled on any node. Rolling updates replace pods one at a time. Use for: web APIs, microservices, anything stateless. StatefulSet: manages stateful pods needing stable identity. Pods get ordered stable names (mongo-0, mongo-1, mongo-2) that persist across restarts. Each pod gets its own PVC via volumeClaimTemplates. Pods start/stop in order. DNS: pod-0.service.namespace. Use for: databases, message brokers, any app that knows its own identity. DaemonSet: ensures one pod runs on every node (or selected nodes). When new node added, DaemonSet pod automatically scheduled on it. When node removed, pod garbage collected. Use for: log collectors (Fluentd, Promtail), monitoring agents (Datadog agent, node-exporter), network plugins (Cilium, Calico), security agents (Falco). The key distinction: Deployment = any pod anywhere, StatefulSet = specific pod with stable identity, DaemonSet = one per node.

Readiness probe: determines when the pod is ready to RECEIVE TRAFFIC. Until readiness probe passes, the pod is removed from the Service endpoints. Traffic is not sent to it. Use for: apps that take time to load config, warm up caches, or establish database connections. If readiness fails: pod stays Running but gets no traffic. Kubernetes does NOT restart it. Liveness probe: determines if the container is ALIVE (not stuck in a deadlock, infinite loop, or corrupted state). If liveness fails: Kubernetes restarts the container. Use for: catching application freezes that do not crash the process. Startup probe: used for slow-starting containers (Java apps, legacy apps). Liveness and readiness probes are disabled until the startup probe succeeds. Prevents liveness from killing a slow-starting pod before it finishes initialising. Without startup probe: a Java app taking 90 seconds to start would be killed by liveness (default 30s timeout) before it could serve traffic. Types: httpGet (HTTP 200-399 = success), tcpSocket (connection accepted = success), exec (command exit code 0 = success). Always tune: initialDelaySeconds, periodSeconds, failureThreshold to match your app's actual startup and response characteristics.

ConfigMap: stores non-sensitive configuration as key-value pairs. Environment variables, config files, command-line arguments. Not encrypted at rest by default. Use for: application config (log level, feature flags, database hostnames). kubectl create configmap app-config --from-literal=LOG_LEVEL=INFO --from-file=app.properties. Secret: stores sensitive data, base64 encoded (not encrypted by default in etcd — enable etcd encryption for true security). Use for: passwords, tokens, certificates, API keys. kubectl create secret generic db-creds --from-literal=password=mypassword. Better: use Azure Key Vault with CSI Secrets Store driver — secrets are NOT stored in etcd at all, fetched from Key Vault and mounted as files. Mount options: as environment variables (env.valueFrom.configMapKeyRef), as volume files (mount config as /etc/config/app.properties), as volume for secrets (mount at /var/secrets/). Best practice: never put secrets in environment variables (visible in kubectl describe pod, ps aux output). Always mount as files. Use Workload Identity to fetch from Key Vault dynamically.

Deployment rollback: kubectl rollout undo deployment/myapp -n production. Kubernetes rolls back to the previous ReplicaSet. Check history first: kubectl rollout history deployment/myapp shows all revisions with CHANGE-CAUSE annotations (add --record to deployments). Roll back to specific version: kubectl rollout undo deployment/myapp --to-revision=3. Monitor rollback: kubectl rollout status deployment/myapp --watch. Helm rollback: helm rollback myapp [REVISION] -n production. helm history myapp shows all releases. After rollback: investigate why the new version failed BEFORE attempting another deployment. Root cause analysis: kubectl logs from the failed pods, kubectl describe pod for events, check monitoring dashboards for the exact minute deployment started. Prevention: use --atomic in helm upgrade (auto-rollback if deployment fails within timeout). Use ArgoCD — OutOfSync detection and one-click rollback via UI. Blue-green: instant rollback by switching traffic back to blue. Canary: if error rate increases at 10% traffic, never increase to 30% — auto-rollback. Key point: rollback is not failure. Rollback + investigation + fix + re-deploy is the right pattern. Never leave a broken version deployed.

Minute 0-2 — assess without touching anything: kubectl get nodes (are nodes healthy?), kubectl get pods -A | grep -v Running (full picture), kubectl top nodes (CPU/memory pressure?), kubectl get events -A --sort-by=.lastTimestamp | tail -30 (what just happened?). Minute 2-5 — identify trigger: if nodes show MemoryPressure or DiskPressure → eviction is expected, kubelet protecting the node, fix: add nodes or reduce pod density. If sudden → check what changed, was there a deployment 30 min before? Minute 5-10 — triage by severity: are any critical services (payment, auth) affected? If yes → declare incident immediately, do not solo-debug. Minute 10-20 — fix based on root cause: resource pressure → kubectl cordon the struggling node, drain pods to healthy nodes. Quota issue → check ResourceQuota. Node NotReady → check kubelet logs on the node. Key principle: never touch a production cluster blindly. Understand before acting. Communicate blast radius within 5 minutes of being paged.

Multi-AZ worker nodes — minimum 3 nodes across 3 availability zones so one zone failure does not bring the cluster down. Pod Anti-Affinity rules — ensure no two replicas of the same service land on the same node. topologySpreadConstraints with maxSkew:1. Resource requests and limits on every pod — prevents noisy neighbours starving other services. HPA on every stateless service — scale based on CPU (70% threshold) or custom metrics (Kafka lag, queue depth). PodDisruptionBudgets — minAvailable:1 ensures at least one pod always running during node drains. ReadinessProbes — pod only gets traffic when genuinely ready. LivenessProbes — restart pods that are alive but not serving. Cluster Autoscaler — nodes scale with pod demand. Pre-production load testing — verify the HPA and CA work before peak traffic. At HPE I managed a 50+ pod OpenShift cluster serving 3 telecom provisioning systems at 99.9% SLA using exactly this pattern.

RBAC has four objects. Role: namespace-scoped, defines which verbs (get/list/watch/create/delete) on which resources (pods/deployments/secrets). ClusterRole: same but cluster-wide. RoleBinding: attaches a Role to a subject (User, Group, ServiceAccount) in a namespace. ClusterRoleBinding: attaches ClusterRole cluster-wide. Production pattern at HPE: developers got a Role in their team namespace with get/list/watch on pods, deployments, services — no create/delete in production. The deployment service account got create/update on deployments only. No human had cluster-admin except the platform team. Integration with Azure AD: we used AKS AAD integration — engineers authenticated with their corporate credentials, mapped to K8s groups via ClusterRoleBindings. The developer AD group mapped to view ClusterRole. The release team AD group mapped to a custom deploy Role. Audit: kubectl get rolebindings,clusterrolebindings -A shows the entire access matrix.

Drift occurs when infrastructure changes outside Terraform — someone modifies a security group manually, an auto-healing process changes a VM, or an Azure Policy enforces a setting. Detection: terraform plan compares state file (what Terraform thinks exists) with actual cloud resources via API calls. Differences are shown as changes. Run terraform plan regularly in a read-only CI job — alert if plan output is non-empty. Fix options: 1. Re-apply Terraform to restore desired state: terraform apply accepts the plan. Overwrites the manual change. 2. Import the manual change into state: terraform import azurerm_virtual_machine.web /subscriptions/.../virtualMachines/myVM — updates state to reflect reality, then update the .tf code to match. 3. Use lifecycle ignore_changes: lifecycle { ignore_changes = [tags] } — tells Terraform to ignore specific attributes that change outside its control (e.g. auto-managed tags). Prevention: enforce "no console changes" policy. Use Azure Policy or SCPs (AWS) to block manual infrastructure changes in production. All changes through Terraform, all Terraform through CI/CD pipeline with PR review.

Terraform workspaces allow multiple state files for the same configuration in the same directory. Default workspace is always "default". Create: terraform workspace new dev. Switch: terraform workspace select production. Reference in code: terraform.workspace variable. Example: resource_group_name = "rg-${terraform.workspace}". Use workspaces when: environments are structurally identical (same resources, just different names/sizes), small team, simple infrastructure. Limitations: all workspaces share the same backend config, no separation of access controls between workspaces. Separate directories + separate state files when: environments differ significantly (production has more redundancy, WAF, DR), different teams manage different environments, need different IAM permissions for production vs dev, want blast-radius isolation. At HPE I used separate directories per environment with separate state files in separate storage containers. Production state container had strict IAM — only CI/CD service principal could write. Developers could plan against dev but not apply to production. This is the enterprise pattern.

A stuck apply is worse than a failed apply — the state might be partially modified and locked. Step 1: check if it is genuinely stuck or just slow. AKS cluster creation takes 10-15 minutes. Azure SQL creation takes 5 minutes. Wait and check Azure Portal — is the resource being created? Step 2: check the state lock. If another process holds the lock, your apply waits. Check Azure Storage → terraform.tfstate.lock.info file. If the locking process crashed, the lock is orphaned. Force-unlock: terraform force-unlock LOCK_ID. Only do this if you are CERTAIN no other process is applying. Step 3: if a specific resource is stuck: check Azure Activity Log for the resource — Azure might be returning an error that Terraform is retrying. Sometimes Azure has transient issues. Step 4: if apply was interrupted: run terraform plan again — it shows what changed and what remains. The state file reflects what was successfully created before the interruption. Run apply again to complete the remaining resources. Prevention: set -parallelism=5 (default 10) to reduce concurrent API calls and avoid rate limiting. Use timeouts in resource configs: timeouts { create = "30m" }.

Terraform CODE in Git: version controlled, PR reviewed, all changes audited. One repo per major infrastructure domain or per cloud (infra-azure/, infra-aws/). Branching: main = production, feature branches for changes, PR required to merge. Never commit .tfstate or .tfvars with secrets to Git. Add to .gitignore: *.tfstate, *.tfstate.backup, .terraform/, *.tfvars (if containing secrets). Terraform STATE in remote backend: Azure: Azure Storage Account, Blob Container, terraform.tfstate file. backend "azurerm" { resource_group_name = "tfstate-rg", storage_account_name = "tfstateXXX", container_name = "tfstate", key = "production.tfstate" }. AWS: S3 bucket with versioning enabled. State locking: AzureRM backend uses Azure Blob lease for locking — prevents two applies running simultaneously. Separate state per environment: dev has its own tfstate, staging has its own, production has its own. This means a failed plan in staging cannot corrupt production state. Access control: production state storage account locked to CI/CD pipeline service principal and senior engineers only. Developers can read staging state, not production.

Static analysis tools for Terraform security and compliance scanning: tfsec: scans Terraform code for security misconfigurations. Runs in CI pipeline before terraform plan. Example findings: storage account allows public access, security group allows 0.0.0.0/0 on port 22, AKS cluster has RBAC disabled. Checkov: policy-as-code scanner. Checks against CIS benchmarks for Azure, AWS, GCP. Outputs pass/fail per check with remediation advice. Pipeline integration: - stage: Security_Scan steps: - script: pip install checkov && checkov -d . --framework terraform --compact --quiet. Fail the pipeline if HIGH severity issues found. tflint: linting and provider-specific rules. Catches syntax errors, deprecated attributes, invalid resource configurations. Infracost: calculates cost of terraform plan output — see the cost impact before applying. Workflow: git push → Terraform lint (tflint) → Security scan (tfsec + Checkov) → terraform fmt check → terraform validate → terraform plan (in CI) → review plan output → manual approval → terraform apply (in CD). All scan results stored as pipeline artifacts for audit.

Multiple defence layers: Layer 1 — lifecycle block: lifecycle { prevent_destroy = true } on critical resources (databases, VNets, AKS clusters). Terraform errors if it tries to destroy. Requires explicit removal of the block before any destruction. Layer 2 — state separation: production has its own state file. Developers cannot run terraform apply in production — only the CI/CD pipeline can. Layer 3 — plan review gate: CI pipeline runs terraform plan and posts output as PR comment. Any destroy shows up highlighted. Required reviewer must approve. No auto-apply of destroys. Layer 4 — workspace/folder protection: production Terraform files in a separate directory. Branch protection rules prevent direct commits. Layer 5 — IAM restrictions: production service principal only has Contributor on the specific resource group, not subscription-wide. Cannot delete resource groups (Owner role needed). Layer 6 — Azure resource locks: az lock create --name "production-lock" --lock-type CanNotDelete --resource-group production-rg. Even Owner role cannot delete locked resources without first removing the lock — generates an audit trail. Layer 7 — backup: Azure Backup enabled on all databases. Even if destroyed, recovery is possible. Defence in depth: no single layer fails catastrophically.

Step 1: do not apply. Read every destroy line carefully. Terraform destroys when: a resource was renamed (sees old name as deleted, new as created — use moved block to fix without recreation). A required attribute changed that forces replacement (some attributes like AKS node pool OS disk type cannot change in-place). The resource was imported outside Terraform and the state does not match. Step 2: for each destroy, determine if it is intentional. If database or network resource: almost certainly a mistake. If an old unused resource: may be fine. Step 3: fix the cause not the symptom. Renamed resource: add moved { from = aws_instance.old; to = aws_instance.new } block — Terraform updates state without destroying. Attribute change forcing replacement: check if you can use lifecycle { ignore_changes = [disk_type] } or if a blue-green replacement is acceptable. Step 4: add lifecycle { prevent_destroy = true } on all production databases, networks, and stateful resources. This makes Terraform error if it ever tries to destroy them, requiring explicit removal of the block first. Step 5: in CI/CD, require manual approval for any plan containing destroy. Never auto-apply destroys.

Structure I have used in production at HPE. Three layers: Layer 1 — modules/ directory. Reusable components: modules/aks-cluster/, modules/networking/, modules/monitoring/. Each module has variables.tf (inputs), main.tf (resources), outputs.tf (exposed values). No environment-specific code here. Layer 2 — environments/ directory. environments/dev/, environments/staging/, environments/production/. Each environment directory calls the modules with environment-specific variable values. environments/production/main.tf: module "aks" { source = "../../modules/aks-cluster"; node_count = 5; vm_size = "Standard_D4s_v3" }. Layer 3 — separate state per environment. Each environment has its own backend.tf pointing to a different storage container: dev state in dev-tfstate storage, production state in prod-tfstate storage with stricter access controls. Multi-region: create a subdirectory per region: environments/production/eastus/, environments/production/westeurope/. Each has its own state. Cross-region resources (like Azure Traffic Manager) go in environments/production/global/. Key rule: production state is read-only for most engineers. Only the pipeline service principal and a small group can run terraform apply in production.

Parallel stages run simultaneously in a Declarative Pipeline. stages { stage("Parallel Tests") { parallel { stage("Unit Tests") { steps { sh "mvn test -Dtest=Unit*" } }; stage("Integration Tests") { steps { sh "mvn test -Dtest=Integration*" } }; stage("Security Scan") { steps { sh "trivy image myapp:${env.BUILD_NUMBER}" } } } } }. The three stages run at the same time. If any fails, the parallel block fails. Limit with agent { label "docker" } per stage if they need different environments. Resource considerations: each parallel stage runs on a separate agent if available. Ensure Jenkins has enough agents. For matrix builds (test against multiple JDK versions): matrix { axes { axis { name "JAVA_VERSION"; values "11", "17", "21" } }; stages { stage("Build") { steps { sh "java -version" } } } }. Performance impact: parallel execution reduced our pipeline from 45 minutes to 12 minutes at HPE. Unit tests (5 min) + integration tests (8 min) + security scan (6 min) running in parallel = 8 minutes total instead of 19 minutes sequential.

Shared Libraries allow you to define reusable pipeline code that multiple Jenkinsfiles can import. Without them: every project has its own Jenkinsfile copying the same boilerplate. If you need to change the security scan step, you update 20 Jenkinsfiles. With Shared Library: one change propagates to all pipelines. Structure: vars/ directory contains global functions as Groovy files. Call: @Library("my-shared-lib") import. vars/dockerBuild.groovy: def call(String imageName, String tag) { sh "docker build -t ${imageName}:${tag} ." }. In Jenkinsfile: @Library("platform-library") _ → dockerBuild("myapp", env.BUILD_NUMBER). Steps I put in shared library at HPE: dockerBuildAndPush (standard build, scan, push), helmDeploy (standard helm upgrade with retry and rollback), slackNotify (standard Slack alerts on success/failure), sonarAnalysis (standard SonarQube quality gate). Configure in Jenkins: Manage Jenkins → Configure System → Global Pipeline Libraries → Add library with Git URL. Libraries versioned with Git tags — teams can pin to specific versions or use main for latest.

Architecture: Jenkins controllers (masters) with distributed agents. For scale: do NOT share one Jenkins master for all teams. Use one master per business unit or pipeline type. Better: Jenkins Kubernetes Plugin — spin up fresh pods as Jenkins agents, delete after build. No persistent agents to maintain. Each build gets a clean environment. Kubernetes manifest per build type: agent { kubernetes { yaml pod-spec } } in Jenkinsfile. Job organization: GitHub Organization Plugin — automatically creates jobs for every repo in a GitHub org. Developers do not register their pipelines manually. Folder Structure: top-level folders per team. Each team has own credentials, own slaves, own shared library. RBAC: Jenkins Matrix-based security. Operations team has full access. Developers have access to their folder only. Release managers have access to production environments. Scaling agents: Jenkins scales agent pods on Kubernetes automatically. Peak builds: 50 concurrent pods. Idle: 0 pods. No idle compute cost. Monitoring: Jenkins metrics plugin exposes Prometheus metrics. Alert on: executor saturation (all agents busy), long queue times, failed builds rate. Infrastructure as code: use Jenkins Configuration as Code (JCasC) plugin — entire Jenkins config as YAML in Git.

Declarative pipeline with SonarQube: pipeline { agent any; environment { SONAR_TOKEN = credentials("sonar-token") }; stages { stage("Checkout") { steps { checkout scm } }; stage("SonarQube Analysis") { steps { withSonarQubeEnv("SonarQube") { sh "mvn sonar:sonar -Dsonar.projectKey=myapp -Dsonar.host.url=http://sonar:9000 -Dsonar.login=$SONAR_TOKEN" } } }; stage("Quality Gate") { steps { timeout(time: 5, unit: "MINUTES") { waitForQualityGate abortPipeline: true } } }; stage("Build Docker") { when { expression { return currentBuild.result == null } }; steps { sh "docker build -t myapp:${BUILD_NUMBER} ." } } } }. The waitForQualityGate step polls SonarQube until the quality gate result is available. If the gate fails (coverage below threshold, too many bugs, security hotspots), the pipeline aborts — the Docker image is never built, nothing gets deployed. Quality gate conditions you define in SonarQube: code coverage > 80%, no new CRITICAL or BLOCKER issues, duplicated lines < 5%, security rating A. For new code: Sonar also checks only the changed lines (new code gate) so legacy code does not block new features.

Jenkins Credentials Store: Manage Jenkins → Credentials → System → Global credentials. Store: Username/Password, Secret Text, SSH Keys, Certificates. Reference in pipeline: environment { DB_PASS = credentials("db-password") } or withCredentials([string(credentialsId: "api-token", variable: "TOKEN")]) { sh "curl -H Authorization:Bearer $TOKEN" }. Jenkins masks credentials in build logs automatically. Better approach — Azure Key Vault integration: Jenkins Azure Key Vault plugin. Configure vault URL and credentials in Jenkins. Reference in pipeline: withAzureKeyvault(credentialID: "my-vault", secrets: [[envVariable: "DB_PASS", name: "database-password", secretType: "Secret"]]) { sh "connect to db with $DB_PASS" }. Secrets fetched at runtime, never stored in Jenkins. Best practices: use service principals with minimum permissions, rotate credentials regularly, never hardcode in Jenkinsfile, use short-lived tokens where possible, audit credential access logs. What NOT to do: never echo $SECRET in scripts (even though masked, avoid the habit), never write secrets to files that get archived as build artifacts, never pass secrets as build parameters (they appear in build history).

I have reduced a 45-minute pipeline to 12 minutes. The analysis first — instrument each stage to find where time goes: add timestamps to each stage, run the pipeline and read the logs. Usually 80% of the time is in 20% of stages. Common findings: sequential tests that could run in parallel (unit + integration + security scan all running one after the other), no caching (Maven downloads 500MB of dependencies every run), no Docker layer caching (rebuilding base image from scratch), waiting for environment to spin up. Fixes applied: parallel stages — wrap independent stages in parallel { stage("Unit Tests") stage("Security Scan") stage("Lint") }. Caching — for Maven: mount .m2 as a persistent volume on the Jenkins agent. For npm: cache node_modules. For Docker: use --cache-from with a cached layer image in ACR/ECR. Agent pools — pre-warmed agents eliminate 2-3 minutes of agent startup time. Selective testing — only run full test suite on main branch; on feature branches run only tests related to changed files. Incremental builds — if only docs changed, skip the build stage entirely. Result at HPE: 70% pipeline time reduction, development team satisfaction improved significantly.

This is why we have runbooks and do not depend on a single Jenkins master. Immediate response: check if it is a service crash or a full node failure. Restart the Jenkins service first — most outages are service-level: systemctl restart jenkins. If node failure: failover to the Jenkins secondary if you have HA configured. If no HA: check if all pipelines are stored in Jenkinsfiles in Git (they should be). The pipelines themselves are not lost. Recovery steps: spin up a new Jenkins instance (we had an AMI/VM image with Jenkins pre-installed and configured). Restore the configuration from backup — CASC (Configuration as Code) plugin means the entire Jenkins config is a YAML file in Git. Restore jobs from Git — all Jenkinsfiles are in source control. Restore credentials from the secrets manager — never store credentials only in Jenkins. Within 45 minutes: new Jenkins instance running, CASC applied, all jobs re-registered, credentials from Key Vault re-injected. Preventive action after: implement Jenkins HA (active-passive with EFS/Azure Files for shared storage). Lesson learned: always use CASC + Jenkinsfiles in Git. Never depend on the Jenkins UI for configuration.

BuildKit is the next-generation Docker build engine, enabled by default in Docker 23+. Advantages over classic builder: parallel stage execution — in multi-stage builds, independent stages build simultaneously instead of sequentially, significantly faster. Better caching: RUN --mount=type=cache,target=/root/.m2 mounts a persistent cache directory that survives across builds. Maven downloads once, reuses forever. Cache export: --cache-to type=registry exports build cache to registry for CI/CD to reuse between pipeline runs. Secret mounting: RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm install — secrets available during build but not baked into the image and not visible in docker history. SSH forwarding: RUN --mount=type=ssh git clone git@github.com:private/repo — SSH key available during build without being in the image. Syntax: DOCKER_BUILDKIT=1 docker build . (or set in daemon.json). Better output: structured progress output, per-step timing. Use BuildKit in CI: DOCKER_BUILDKIT=1 docker build --cache-from type=registry,ref=myacr.azurecr.io/myapp:cache --cache-to type=registry,ref=myacr.azurecr.io/myapp:cache,mode=max .

Docker uses disk in several places: images, containers (writable layer), volumes, build cache. Check usage: docker system df shows breakdown. Images too large: docker image ls --format "table {{.Repository}} {{.Tag}} {{.Size}}" | sort -k3 -hr. Remove unused: docker image prune -a removes all images not used by any container. Or selective: docker rmi image-id. Stopped containers using disk: docker container prune removes all stopped containers. Volumes: docker volume prune removes unused volumes (CAREFUL — check if any volume has data you need). Build cache: docker builder prune clears build cache (can be large — 10-20GB on active build machines). Nuclear option: docker system prune --all --volumes removes everything unused. In production CI agents: add to cron: 0 2 * * * docker system prune -f (weekly cleanup). Set daemon.json image GC: log-driver: json-file with max-size and max-file to prevent container logs filling disk. Monitoring: set up Prometheus node-exporter to alert when /var/lib/docker filesystem exceeds 80%. In Kubernetes: eviction threshold eviction-hard: imagefs.available<15% controls when kubelet starts removing images.

# Stage 1: Build dependencies FROM python:3.11-slim AS builder WORKDIR /app # Install deps first (layer caching - deps change less often than code) COPY requirements.txt . RUN pip install --no-cache-dir --user -r requirements.txt # Stage 2: Runtime image FROM python:3.11-slim # Security: create non-root user RUN useradd --create-home --no-log-init appuser WORKDIR /app # Copy only installed packages from builder COPY --from=builder /root/.local /home/appuser/.local # Copy application code COPY --chown=appuser:appuser . . # Switch to non-root user USER appuser # Environment ENV PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 \ PATH=/home/appuser/.local/bin:$PATH EXPOSE 5000 # Health check HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ CMD curl -f http://localhost:5000/health || exit 1 # Use exec form (not shell form) for proper signal handling ENTRYPOINT ["python3", "-m", "gunicorn"] CMD ["--bind", "0.0.0.0:5000", "--workers", "4", "app:app"]

Each instruction in a Dockerfile creates a layer. Docker caches each layer. When rebuilding: if a layer and all layers before it are unchanged, Docker uses the cache. If any layer changes, that layer and ALL subsequent layers are rebuilt. Optimisation rule: put things that change LESS OFTEN earlier in the Dockerfile. Wrong order (slow): COPY . . then RUN pip install. Every code change invalidates the pip install layer — downloads all packages again. Correct order (fast): COPY requirements.txt . then RUN pip install then COPY . . (code). pip install only reruns when requirements.txt changes. For Docker builds in CI: use --cache-from to reuse cache from ACR: docker build --cache-from myacr.azurecr.io/myapp:cache --build-arg BUILDKIT_INLINE_CACHE=1 -t myapp:${BUILD_ID} . and push the cache image: docker push myacr.azurecr.io/myapp:cache. BuildKit (enabled by default in Docker 23+) has improved cache logic including parallel layer building and external cache sources. Result: a correctly ordered Dockerfile with registry cache cuts CI build time from 8 minutes to 90 seconds.

2.1GB is a serious problem in production. I would show the concrete impact: Pull time — on a cold node, pulling 2.1GB takes 3-5 minutes. During a spike, K8s schedules pods on new nodes. They cannot start for 5 minutes. Your HPA is useless during that window. Storage — if you run 20 services on one node, 2.1GB per service = 42GB just in images. Attack surface — a 2.1GB image likely includes a full OS, package manager, compiler, and tools. Every one of those is a potential CVE. Fix I would propose: multi-stage build. Builder stage: full JDK/Maven/Node — installs, compiles, builds. Final stage: FROM eclipse-temurin:17-jre-alpine (100MB base) COPY --from=builder just the JAR. Result: 2.1GB → 180MB in one afternoon. Other reductions: use alpine or distroless base images, remove build tools (apt-get install X && ... && apt-get purge X in one RUN layer), use .dockerignore to exclude tests, docs, node_modules. I have reduced images from 1.8GB to 220MB multiple times. The effort is 2 hours, the benefit lasts the lifetime of the service.

What NOT to do: never put secrets in ENV in the Dockerfile (they appear in docker inspect, docker history, and any image layer). Never bake secrets into the image with COPY or ARG. Never store them in docker-compose.yml committed to Git. Production approaches: Kubernetes Secrets mounted as files — the container reads from /var/secrets/db-password at startup. Better than env vars because they do not appear in ps aux output. External secrets manager — HashiCorp Vault, Azure Key Vault, AWS Secrets Manager. The container authenticates using its workload identity (Workload Identity/IRSA/Managed Identity) and fetches secrets at runtime. The CSI Secrets Store driver mounts secrets directly into pods as files without storing them in etcd. Docker Swarm — docker secret create stores secrets encrypted, mounted at /run/secrets/ inside the container. Rule of thumb: if the secret could be seen by running docker inspect or kubectl describe pod, it is too exposed. Secrets should be fetched at runtime using the container identity, never baked in.

Ansible roles are a structured way to organise playbooks into reusable components. A role has a standard directory structure: roles/nginx/tasks/main.yml (tasks to execute), roles/nginx/handlers/main.yml (handlers like restart nginx), roles/nginx/templates/nginx.conf.j2 (Jinja2 templates), roles/nginx/vars/main.yml (role-specific variables), roles/nginx/defaults/main.yml (default values, lowest priority), roles/nginx/files/ (static files to copy), roles/nginx/meta/main.yml (role dependencies). Use in playbook: roles: - nginx - { role: application, version: "2.1.0" }. Benefits: reusability — one nginx role used by 20 different playbooks. Separation of concerns — nginx config is in the nginx role, not mixed with application config. Testing — roles can be tested independently with Molecule. Distribution — roles can be shared via Ansible Galaxy. galaxy.yml for publishing. Molecule for testing: molecule test runs the role in a Docker container and verifies it. At HPE I created roles for: base-hardening (SSH config, firewall, fail2ban), application-deploy, monitoring-agent, certificate-renewal. Each team consumed these roles without needing to understand their internals.

Static inventory files break when servers come and go (cloud auto-scaling). Dynamic inventory scripts or plugins query the cloud API at runtime. Azure dynamic inventory: install azure.azcollection.azure_rm inventory plugin. Create azure_rm.yml: plugin: azure.azcollection.azure_rm, include_vm_resource_groups: [production-rg]. ansible-inventory -i azure_rm.yml --list returns all Azure VMs. Group by tag: keyed_groups: key: tags.Environment (creates groups like tag_Environment_production). Target all production servers: ansible -i azure_rm.yml tag_Environment_production -m ping. AWS dynamic inventory: aws_ec2 plugin. Create aws_ec2.yml: plugin: amazon.aws.aws_ec2, regions: [eu-west-1], filters: instance-state-name: [running]. keyed_groups: key: tags.Role → groups webservers, databases from EC2 tags. Kubernetes dynamic inventory: for managing K8s nodes with Ansible. In CI/CD: Ansible Tower/AWX stores dynamic inventory configurations. Credentials for cloud APIs configured in Tower. Teams run playbooks against live inventory without managing any host files.

--- - name: Install and configure Nginx on web servers hosts: webservers become: yes vars: nginx_port: 80 tasks: - name: Install Nginx apt: name: nginx state: present update_cache: yes when: ansible_os_family == "Debian" - name: Install Nginx yum: name: nginx state: present when: ansible_os_family == "RedHat" - name: Start and enable Nginx service: name: nginx state: started enabled: yes - name: Configure Nginx template: src: templates/nginx.conf.j2 dest: /etc/nginx/nginx.conf backup: yes notify: Reload Nginx - name: Open port 80 in firewall ufw: rule: allow port: "{{ nginx_port }}" proto: tcp when: ansible_os_family == "Debian" handlers: - name: Reload Nginx service: name: nginx state: reloaded. Key points: become: yes for sudo. when conditions handle different OS families. template module uses Jinja2 template for config. notify triggers the handler only if the task changed — handler runs once at the end even if notified multiple times. Idempotent: running this 100 times always results in the same state.

Ansible Vault encrypts sensitive data (passwords, API keys, certificates) within YAML files or as standalone encrypted strings. The encrypted files can safely be committed to Git — without the vault password they are unreadable. Encrypt a file: ansible-vault encrypt group_vars/production/secrets.yml. Edit encrypted file: ansible-vault edit group_vars/production/secrets.yml — opens in editor, saves re-encrypted. Encrypt a single string (inline): ansible-vault encrypt_string "my-db-password" --name db_password → outputs encrypted YAML value to paste into a vars file. Decrypt for viewing: ansible-vault decrypt secrets.yml (temporary, for debugging). Use in playbook: the vault-encrypted vars are automatically decrypted when running with the vault password. Run playbook: ansible-playbook site.yml --vault-password-file ~/.vault_pass or --ask-vault-pass for interactive prompt. In CI/CD: store vault password in Jenkins/Azure DevOps as a secret credential. Pass as --vault-password-file /dev/stdin with the credential piped in. Multiple vault IDs: ansible-vault encrypt --vault-id prod@prompt secrets.yml allows different passwords for different environments. Best practice: never store the vault password in the same repo as the encrypted files.

UNREACHABLE means Ansible cannot connect via SSH (or WinRM) to those hosts. It does not mean the playbook task failed. Diagnose: check if the hosts are actually up: ping hostname. Check SSH manually: ssh user@hostname. Common causes: host is down or being rebooted (wait and retry), SSH key changed or expired (update inventory credentials), firewall blocking SSH port 22 (check NSG/security group), wrong username or Python interpreter path, bastion/jump host required but not configured. Handle gracefully in playbook: serial: 5 processes 5 hosts at a time. max_fail_percentage: 20 — if more than 20% of hosts fail, abort. This prevents cascading failures. Retry failed hosts: Ansible creates a retry file (site.retry) listing failed hosts. Re-run: ansible-playbook site.yml --limit @site.retry to only target the failed hosts. ignore_unreachable: true in play or task to continue with remaining hosts even if some are unreachable — use carefully and log which hosts were skipped. After fix: ansible all -m ping --limit problematic_hosts verifies connectivity before re-running the full playbook.

Configuration drift means servers that should be identical have diverged -- someone ran a manual command, a package auto-updated, a file was edited. Detection: ansible all -m setup --tree /tmp/facts collects current state of all servers. Compare against expected state. Use ansible-lint to validate playbooks. The idempotent approach: write your playbooks with state: present/absent/latest so running them multiple times always produces the same result. Run playbooks in check mode first: ansible-playbook site.yml --check --diff shows exactly what would change without making changes. This is your drift detection. Fix: ansible-playbook site.yml --limit drifted_servers applies the correct state. For scheduled drift detection: run the playbook in check mode via cron or Ansible Tower/AWX every 6 hours. If check mode finds differences, send an alert. Decide: auto-remediate (re-run without --check) or require manual approval. At HPE we ran check mode daily and auto-remediated non-critical drift (package versions) but required human approval for file changes in /etc/.

The danger: 25 servers are in the new state, 25 are in the old state. Your environment is inconsistent. Recovery steps: do not just re-run the full playbook blindly. Step 1: identify exactly which servers failed. The Ansible output shows the exact hosts. ansible-playbook site.yml --limit "failed_servers" --start-at-task "task that failed". Step 2: understand WHY it failed. Check the error message. Common causes: SSH connection timeout (network issue, not a playbook problem -- retry), package manager locked (another process running apt/yum -- wait and retry), disk full (needs manual intervention before retry), wrong variable value (fix the variable first). Step 3: use --start-at-task to resume from where it failed -- do not re-run from the beginning on servers where steps already completed. Step 4: for the 25 servers that completed successfully, run the playbook in --check mode to verify they are in the correct state. Step 5: after full recovery, run the complete playbook against all 50 servers in check mode to confirm consistency. Prevention: use serial: 10% in the playbook so Ansible processes servers in batches. If the first 10% fail, it stops before affecting all servers.

Alertmanager receives alerts from Prometheus and handles routing, grouping, silencing, and notification. Config: global.smtp_from / slack_api_url. route: group_by: [alertname, namespace]. group_wait: 30s (wait to collect related alerts). group_interval: 5m (how often to re-send grouped alerts). repeat_interval: 4h (re-alert if not resolved). receiver: slack-production. routes: - match: severity=critical, receiver: pagerduty. Inhibition rules: critical alert inhibits warning — if payment-api-down fires, suppress all individual pod alerts for payment-api (noise reduction). Receiver config for Slack: - name: slack-production, slack_configs: url: $SLACK_WEBHOOK, channel: #alerts-production, text: {{ range .Alerts }}{{ .Annotations.description }}{{ end }}. PagerDuty: service_key: $PAGERDUTY_KEY, severity: critical. Best practices: critical alerts = PagerDuty (wakes someone up). Warning = Slack (visible but no wake-up). Alert on symptoms not causes: alert on "high error rate" not "CPU high" (CPU high is a cause, user impact is the symptom). Dead man switch: always-firing alert ensures Alertmanager itself is working. If it goes silent, something is broken.

Grafana connects to Prometheus as a data source. Add data source: Configuration → Data Sources → Add Prometheus → URL: http://prometheus:9090. For Kubernetes monitoring use kube-prometheus-stack (Prometheus Operator + pre-built Grafana dashboards): helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack. Includes 20+ pre-built dashboards: Kubernetes Cluster Overview (nodes, pods, namespaces), Node Exporter Full (CPU, memory, disk, network per node), Kubernetes Pods (per-pod CPU, memory, restarts), Kubernetes Deployments. Custom dashboard for your application: Add panel → PromQL query → select visualisation type. Panel for HTTP error rate: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service). Organise dashboards: folder per team (platform, payments, orders). Dashboard variables: $namespace dropdown lets user filter all panels. Variables: Settings → Variables → type=query → query: label_values(kube_pod_info, namespace). Now every panel uses $namespace variable. Export dashboards as JSON → store in Git → GitOps deploy via ConfigMap → Grafana sidecar picks them up automatically.

CPU throttling (most important): rate(container_cpu_throttled_seconds_total[5m]) > 0.1 — pods being CPU throttled, indicates limits are too low. OOMKilled rate: increase(kube_pod_container_status_restarts_total[1h]) > 3 and kube_pod_container_status_last_terminated_reason == "OOMKilled". Memory pressure: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1 — node has less than 10% memory available. Pod pending: kube_pod_status_phase{phase="Pending"} > 0 for more than 5 minutes — pods waiting for resources. HTTP error rate per service: sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) > 0.01. P99 latency: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)). Certificate expiry: (x509_cert_expiry - time()) / 86400 < 30 — certificates expiring in less than 30 days. Disk pressure: node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.15 — less than 15% disk free. These 8 queries cover the most common production alert scenarios.

Install Datadog Agent as DaemonSet via Helm: helm repo add datadog https://helm.datadoghq.com. values.yaml: datadog.apiKey from environment variable (DD_API_KEY), datadog.clusterName: myaks-production, clusterAgent.enabled: true, datadog.kubelet.tlsVerify: false (for AKS), datadog.logs.enabled: true, datadog.logs.containerCollectAll: true, datadog.apm.enabled: true. helm install datadog datadog/datadog -f values.yaml -n datadog. What Datadog collects automatically: node CPU/memory/disk metrics, pod metrics (CPU, memory, restarts), container logs from all pods, Kubernetes events. Add application-level monitoring: annotate pods for Datadog log collection: ad.datadoghq.com/container.logs: [{"source":"python","service":"payment-api"}]. For APM: add DD_AGENT_HOST (via downward API) and dd-trace library to application. Dashboards: Datadog has pre-built Kubernetes dashboards. Custom dashboards: drag-and-drop with your custom metrics. Monitors (alerts): create from any metric with threshold, anomaly detection, or forecast alerts. Integration with PagerDuty: one-click in Datadog integrations. Cost control: use DogStatsD for custom metrics instead of APM where possible, configure log exclusion filters to avoid ingesting noisy/irrelevant logs.

Minute 0-2: check the Grafana dashboard. HTTP error rate panel shows which service and which endpoint is returning 503s. Error rate chart shows when it started -- was there a deployment? Minute 2-5: narrow the blast radius. Is it one service or multiple? kubectl get pods -n production -- are pods crashing? If CrashLoopBackOff: the service itself is broken. If pods are running: might be a downstream dependency (database, external API). Minute 5-10: Prometheus queries to isolate. sum(rate(http_requests_total{status="503"}[5m])) by (service, endpoint) -- which specific endpoint? Check the upstream: if payment-api returns 503, check if it can reach the database: probe_success metric from Blackbox Exporter. Check latency: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) by (service) -- is the 99th percentile exploding? This points to slowness not crashes. Minute 10+: if database connection pool exhausted -- increase pool size or scale the service. If a dependency is down -- circuit breaker should kick in, check if it is working. Always: communicate status every 10 minutes to stakeholders even if "still investigating". Silent incident response makes stakeholders nervous.

SLA is the contractual commitment to customers. SLO is the internal engineering target (usually stricter than SLA). SLI is the measurement. At HPE for the telecom provisioning platform: SLI choices -- availability (percentage of requests returning 2xx/3xx, measured per 1-minute window), latency (percentage of requests completing under 500ms), error rate (percentage of requests returning 5xx). SLO values -- availability: 99.9% (allows 43.8 minutes downtime per month), latency: 95% of requests under 500ms, 99% under 2 seconds. SLA to business -- 99.5% availability monthly. Error budget -- if SLO is 99.9% availability over 30 days, the error budget is 43.8 minutes. If we use more than that: freeze new deployments until next month, focus engineering on reliability. Implementation in Prometheus: recording rule calculates rolling 30-day availability. Alert fires when burn rate exceeds 2x (fast burn = will exhaust budget in 2 weeks) or 1x for 6 hours (slow burn). The multi-window alerting (5-minute window + 1-hour window must both exceed threshold) reduces alert noise dramatically. Grafana dashboard shows: current SLO compliance, error budget remaining, burn rate. Reviewed weekly in engineering meeting.

Datadog APM (Application Performance Monitoring) traces requests as they flow through your services — showing latency, errors, and bottlenecks. To instrument Python: pip install ddtrace. Run: ddtrace-run python app.py. This auto-instruments Flask, Django, requests, psycopg2 etc. In code: from ddtrace import tracer; with tracer.trace("operation.name") as span: span.set_tag("user.id", user_id). Key concepts: Trace = end-to-end request. Span = single operation within a trace. Service Map = visual of all services and their dependencies. Flame graph = shows exactly where time is spent within a request. Configure via DD_SERVICE, DD_ENV, DD_VERSION env vars for unified service tagging.

A Datadog Monitor watches a metric or log query and alerts when a threshold is breached. Real example — CPU alert on Kubernetes node: Go to Monitors → New Monitor → Metric. Query: avg:system.cpu.user{kube_cluster_name:prod} by {host}. Set threshold: Alert when > 85% for 5 minutes. Warning at 70%. Configure notification: "@slack-devops-alerts High CPU on {{host.name}} — current: {{value}}%". Add escalation: if unresolved after 30 minutes, page on-call via PagerDuty. Types of monitors: Metric monitor (threshold on time-series), Log monitor (alert on log pattern count), APM monitor (P99 latency > 500ms), Composite monitor (CPU AND memory both high), Anomaly monitor (ML-based deviation from baseline).

Install via Helm: helm repo add datadog https://helm.datadoghq.com; helm install datadog-agent datadog/datadog --set datadog.apiKey=YOUR_KEY --set datadog.clusterName=prod. The DaemonSet deploys one agent per node. What to monitor: Node level — CPU, memory, disk, network per node. Pod level — container CPU/memory limits vs requests, OOMKills, restarts. Control plane — API server latency, etcd health, scheduler queue. Application — custom metrics via StatsD or DogStatsD. Logs — collect with datadog.logs.enabled=true, then use Log Patterns to detect anomalies. Key dashboards: Kubernetes Overview (built-in), Node health, Deployment status. SLO tracking: create SLOs in Datadog based on APM error rate or uptime monitors — links directly to error budget reporting.

Three pillars of observability — each answers a different question. Metrics: What is happening? Numeric, time-series data. CPU=85%, request rate=1200/min, error rate=0.3%. Low storage cost, fast to query, ideal for dashboards and alerts. Logs: Why did it happen? Unstructured text events. "ERROR: database connection failed at 14:32:05". High detail but expensive to store and slow to search. Use for debugging specific incidents. Traces: How did it happen? Distributed request flow across services. Shows that a slow API call spent 800ms waiting on database query. Links logs and metrics to a specific request. In practice: Alert triggers from a Metric threshold. You open the Dashboard, correlate with Logs to find the error message. You open the Trace to see which service call caused the slowdown. Together they give full context — metrics alone cannot tell you WHY, logs alone cannot show you WHERE across services.

To handle unpredictable traffic spikes, I would design an auto-scaling solution using AWS Auto Scaling groups. First, I would create an Auto Scaling group for the web application instances, defining the minimum, maximum, and desired capacity. I would set up scaling policies based on CloudWatch metrics, such as CPU utilization or request count. For example, if the average CPU utilization exceeds 70% for 5 consecutive minutes, I would trigger the scaling out policy to add more instances. Conversely, if the CPU utilization drops below 20% for 10 minutes, I would trigger the scaling in policy to terminate instances. I would also use Elastic Load Balancing (ELB) to distribute incoming traffic across multiple instances to ensure even load distribution and high availability.

To optimize costs, I would implement the following strategies: - Right-Sizing: Analyze the usage of EC2 instances and other resources, and right-size them to the most cost-effective instance types based on actual usage. - Reserved Instances (RIs) and Savings Plans: Purchase Reserved Instances or Savings Plans for predictable workloads to get significant discounts compared to On-Demand pricing. - Spot Instances: Use Spot Instances for fault-tolerant and flexible workloads to take advantage of the lower prices. - Auto Scaling: Implement Auto Scaling to ensure resources are only used when needed, scaling in during off-peak times. - Storage Optimization: Use S3 lifecycle policies to move infrequently accessed data to cheaper storage classes (e.g., S3 Infrequent Access or Glacier). Delete unused EBS volumes, snapshots, and obsolete data. - Monitoring and Alerts: Set up CloudWatch alarms to monitor resource usage and receive alerts for any anomalies or over-utilization. - Cost Explorer and Budgets: Use AWS Cost Explorer and Budgets to analyze spending patterns and set up budgets and alerts to stay within budget.

The disaster recovery plan would include: - Backup and Restore: Regularly back up data using AWS Backup or automated scripts to S3, and enable versioning and cross-region replication for S3 buckets. - Pilot Light: Maintain a minimal version of the environment always running in another region. In case of a disaster, scale up the resources to handle production traffic. - Warm Standby: Run a scaled-down version of the fully functional environment in another region. In case of a disaster, scale up to full capacity. - Multi-Region Active-Active: Deploy the application across multiple regions in an active-active configuration. Use Route 53 for DNS failover to route traffic to healthy regions. - Data Replication: Use services like RDS with Multi-AZ or Aurora Global Database for database replication across regions. Use DynamoDB global tables for multi-region replication of NoSQL data. - Regular Testing: Regularly test the disaster recovery plan by simulating failover and recovery scenarios to ensure that RTO (Recovery Time Objective) and RPO (Recovery Point Objective) are met.

To ensure compliance, I would: - Encryption: Enable encryption at rest and in transit for all sensitive data using AWS Key Management Service (KMS) and SSL/TLS. Use AWS Certificate Manager for managing SSL/TLS certificates. - Identity and Access Management (IAM): Implement the principle of least privilege by creating granular IAM policies and roles. Use IAM roles for EC2 instances and other services to avoid hardcoding credentials. - Logging and Monitoring: Enable CloudTrail for auditing API calls, configure CloudWatch Logs for logging application and system logs, and use AWS Config for resource configuration tracking and compliance auditing. - Network Security: Use VPC with subnets, security groups, and NACLs to control inbound and outbound traffic. Implement AWS WAF and Shield for web application protection. - Compliance Programs: Leverage AWS compliance programs and services like AWS Artifact to access audit reports and compliance documentation for regulatory requirements. - Automated Security Assessments: Use AWS Trusted Advisor, Inspector, and Macie to regularly scan for vulnerabilities and data leaks, and ensure compliance with security best practices. - Periodic Audits and Penetration Testing: Conduct regular security audits and penetration testing to identify and mitigate potential vulnerabilities.

For minimal downtime migration of a large database, I would recommend: - AWS Database Migration Service (DMS): Use AWS DMS to perform the migration. Set up the source and target endpoints and create a replication task to perform continuous data replication from the on-premises database to the AWS target database. - Pre-Migration Planning: Conduct a thorough assessment of the source database, including schema, data size, and dependencies. Plan the migration strategy, downtime window, and rollback procedures. - Schema Conversion: Use AWS Schema Conversion Tool (SCT) to convert the source database schema to the target database schema (e.g., from Oracle to Aurora PostgreSQL). - Continuous Data Replication: Enable continuous data replication using DMS to keep the source and target databases in sync. This minimizes downtime by ensuring that only the final incremental changes need to be applied during the cutover. - Testing: Perform multiple migration tests in a staging environment to ensure data consistency, performance, and application compatibility. - Cutover: Schedule the cutover during a maintenance window. Perform a final sync using DMS, switch the application to the new database, and validate the data integrity and application functionality.

To implement a multi-account strategy, I would: - AWS Organizations: Use AWS Organizations to create and manage multiple AWS accounts. Create Organizational Units (OUs) for different projects, departments, or environments (e.g., development, staging, production). - Service Control Policies (SCPs): Implement SCPs to enforce governance and control access at the organizational level. Define policies to restrict or allow specific actions based on organizational requirements. - Cross-Account Access: Set up IAM roles for cross-account access, allowing users and services in one account to access resources in another account securely. Use AWS Resource Access Manager (RAM) to share resources like VPCs and subnets across accounts. - Billing and Cost Management: Use consolidated billing to aggregate usage and simplify billing management. Allocate budgets and set up cost allocation tags to track expenses per account, project, or department. - Centralized Logging and Monitoring: Use AWS CloudTrail, CloudWatch, and AWS Config across all accounts to centralize logging, monitoring, and compliance tracking. Aggregate logs to a central S3 bucket or use AWS CloudWatch Logs Insights for cross-account log analysis. - Security and Compliance: Implement a centralized security model using AWS Security Hub, GuardDuty, and Macie to monitor and enforce security best practices across all accounts.

To design a highly available and fault-tolerant architecture: - Multi-AZ Deployment: Deploy the application across multiple Availability Zones (AZs) to ensure high availability and fault tolerance. Use services like RDS with Multi-AZ for database deployments. - Elastic Load Balancing (ELB): Use ELB to distribute incoming traffic across multiple EC2 instances running in different AZs. This ensures that traffic is balanced and that the application remains available even if an instance or AZ fails. - Auto Scaling: Configure Auto Scaling groups to automatically adjust the number of EC2 instances based on demand. This ensures that the application can handle traffic spikes and maintain performance. - Data Replication and Backup: Use S3 for data storage with versioning and cross-region replication enabled. Regularly back up data using AWS Backup and implement snapshot policies for EBS volumes and RDS instances. - Route 53: Use Route 53 for DNS management with health checks and failover routing policies to route traffic to healthy endpoints and provide seamless failover in case of an AZ or region failure. - Serverless Services: Where applicable, use serverless services like AWS Lambda, DynamoDB, and S3 to reduce the risk of infrastructure-related failures and increase scalability.

To modernize the legacy monolithic application: - Assessment and Planning: Conduct a thorough assessment of the existing monolithic application to understand its components, dependencies, and data flow. Identify the logical boundaries and the potential microservices that can be created. - Containerization: Start by containerizing the existing monolith using Docker. Deploy the containerized application on Amazon ECS or EKS to manage the containers. - Service Decomposition: Gradually decompose the monolith into microservices. Identify and extract functionalities into independent services. Use AWS Lambda for stateless, event-driven microservices or ECS/EKS for stateful services. - Service Communication: Use Amazon API Gateway to manage API requests and service-to-service communication. Implement inter-service communication using RESTful APIs, gRPC, or AWS App Mesh for service mesh architecture. - Data Management: Decouple the data layer by using purpose-built databases for each microservice. Use DynamoDB, RDS, or Aurora depending on the data requirements. Implement data synchronization and eventual consistency where necessary. - CI/CD Pipeline: Set up a CI/CD pipeline using AWS CodePipeline, CodeBuild, and CodeDeploy to automate the build, test, and deployment processes for the microservices. - Monitoring and Logging: Implement centralized logging and monitoring using AWS CloudWatch, X-Ray, and Elasticsearch Service to trace requests, monitor performance, and troubleshoot issues.

To create a hybrid cloud environment: - Networking: Establish a secure network connection between the on-premises data center and AWS using AWS Direct Connect or VPN. This ensures low latency and secure communication. - Identity and Access Management: Implement federated access by integrating on-premises Active Directory with AWS IAM. Use AWS Single Sign-On (SSO) or IAM roles with SAML for seamless identity management. - Data Integration: Use AWS DataSync or Storage Gateway to integrate on-premises storage with AWS storage services like S3. For databases, use AWS DMS to replicate data between on-premises databases and AWS databases. - Hybrid Services: Leverage hybrid services like AWS Outposts to run AWS infrastructure and services on-premises for workloads that require low latency or data residency compliance. - Workload Distribution: Use AWS Elastic Beanstalk, ECS, or EKS to deploy and manage workloads across both on-premises and AWS environments. Implement load balancing and auto-scaling to manage the distribution of workloads. - Monitoring and Management: Use AWS Systems Manager to manage and monitor both on-premises and AWS resources. Implement CloudWatch and AWS Config for centralized monitoring, logging, and compliance tracking.

For a serverless application, I would use the following architecture and services: - Compute: Use AWS Lambda to run the application code without managing servers. Write functions to handle individual tasks and trigger them based on events. - API Gateway: Use Amazon API Gateway to create and manage RESTful APIs. Integrate API Gateway with Lambda functions to handle incoming requests and responses. - Storage: Use Amazon S3 for storing static assets like images, videos, and documents. Use DynamoDB for a serverless, managed NoSQL database solution for storing application data. - Authentication and Authorization: Use Amazon Cognito for user authentication, authorization, and user management. Integrate Cognito with API Gateway to secure the APIs. - Messaging and Integration: Use Amazon SNS and SQS for message publishing and queuing. Use AWS Step Functions to coordinate and manage complex workflows and state machines. - Monitoring and Logging: Use AWS CloudWatch for monitoring application performance and setting up alarms. Use CloudWatch Logs to aggregate and analyze logs from Lambda functions. - CI/CD: Use AWS CodePipeline, CodeBuild, and CodeDeploy to automate the build, test, and deployment processes for the serverless application. Each of these scenarios covers critical aspects of an AWS Cloud Solution Architect role, helping candidates demonstrate their practical knowledge and problem-solving abilities in real-world situations.

EC2 instance types (e.g., t2, m5, c5)?

Amazon S3 handle object versioning?

RDS Multi-AZ and Read Replicas.

Security Groups and NACLs (Network ACLs)?

Route 53 failover routing** and how to configure it.

Auto Scaling Groups** work with **Elastic Load Balancers.

load balancers in AWS (ALB, NLB, CLB)?

highly available architecture** across multiple regions?

IAM roles, policies, and users?

grant temporary access to an S3 bucket?

AWS Key Management Service (KMS)** and how to encrypt EBS volumes.

ECS (Elastic Container Service)** differ from **EKS (Elastic Kubernetes Service)?

Fargate,** and how does it simplify container management?

DynamoDB Global Tables** and how they achieve cross-region replication.

CloudWatch differ from CloudTrail?

alarms and automated scaling** using CloudWatch?

AWS Savings Plans** to reduce costs?

Reserved Instances** and **Spot Instances.

AWS billing and cost allocation?

High availability means the system continues operating despite failures. Multi-AZ architecture: deploy across at least 2 AZs. ELB distributes traffic across AZs. RDS Multi-AZ: synchronous replication, automatic failover in 60-120 seconds. EC2 Auto Scaling Groups: span multiple AZs, replace unhealthy instances automatically. EKS: worker nodes across 3 AZs, pod anti-affinity ensures replicas spread across AZs. Route 53 health checks: route traffic away from unhealthy endpoints. S3: 11 nines durability, multi-AZ storage automatically. For stateless services: easy — add more instances across AZs. For stateful services: harder — use managed services (RDS, ElastiCache) which handle replication. DR beyond single region: Route 53 latency routing or failover routing to secondary region. RDS read replicas in secondary region promoted to primary if main region fails. RTO target: 15 minutes. RPO target: 5 minutes. Test DR quarterly: simulate region failure, verify traffic routes correctly, verify database failover completes.

Step 1: check node group status. aws eks describe-nodegroup --cluster-name myEKS --nodegroup-name myNodeGroup. Status DEGRADED or CREATE_FAILED shows the reason. Step 2: check EC2 instances in the Auto Scaling Group. aws autoscaling describe-auto-scaling-groups. Are instances launching? If launching but not joining: Step 3: check EC2 instance system logs. aws ec2 get-console-output --instance-id i-xxxx shows bootstrap errors. Common causes: IAM instance role missing AmazonEKSWorkerNodePolicy or AmazonEC2ContainerRegistryReadOnly policies. VPC DNS not enabled (enableDnsHostnames must be true). Security Group blocking port 443 to EKS cluster endpoint. Wrong AMI — must use EKS-optimised AMI for the correct K8s version. Bootstrap script error — check /var/log/cloud-init-output.log on the node. Step 4: if node joins but shows NotReady: kubectl describe node nodename — look at Conditions. Usually: kubelet not started (systemctl status kubelet), CNI plugin not installed, or disk pressure. kubectl get events -A shows cluster-level events that may explain the issue.

EKS: managed Kubernetes. Full K8s feature set. Control plane managed by AWS ($0.10/hr). You manage worker nodes. Use when: team has K8s expertise, complex microservices needing K8s features (RBAC, network policies, custom schedulers), multi-cloud portability required, existing Kubernetes workloads. ECS: AWS-proprietary container orchestrator. Simpler than K8s. No control plane cost. Two modes: EC2 (you manage servers) and Fargate (fully serverless). Use when: team is new to containers, pure AWS environment, want simplicity over flexibility, Fargate avoids node management entirely. Lambda: serverless functions. No infrastructure. Pay per execution (per 100ms). Scale to zero. Cold start latency (100ms-3s). Use when: event-driven processing (S3 uploads, SQS messages, API calls), short-lived tasks (under 15 minutes), variable traffic with periods of zero load. Decision framework: Is the workload containerised and long-running? → EKS or ECS. Is it event-driven and short? → Lambda. Does the team know Kubernetes? → EKS. Does the team want simplicity? → ECS Fargate. At HPE we used EKS for the main microservices platform (K8s expertise existed) and Lambda for event-driven integrations (S3 processing, webhook handlers).

GitHub Actions is CI/CD built directly into GitHub — no server to manage, no plugins to update, no agents to configure. You write YAML workflows stored in .github/workflows/. When an event happens (push, PR, schedule), GitHub automatically runs the workflow on hosted servers (runners). Key differences from Jenkins: GitHub manages all infrastructure — you never think about agents going offline or Jenkins master running out of memory. GitHub Actions uses OIDC for cloud authentication, so no long-lived secrets stored anywhere. The free tier gives 2000 minutes/month which is enough for most projects. Jenkins gives more control and is better for complex pipelines with many dependencies, but GitHub Actions wins for simplicity and GitHub-integrated projects.

Jobs run in parallel by default on separate runner machines. Each job gets a fresh VM — nothing carries over between jobs unless you explicitly pass outputs or use artifacts. Steps run in sequence inside a job on the same machine. If a step fails, the job fails and remaining steps are skipped (unless you add if: always()). Common pattern: quality job (tests, SonarQube) runs in parallel with security job (Trivy, Checkov). Both must pass before the build job starts. Build job produces a Docker image, uploads it as an artifact. Deploy job downloads that artifact and deploys. The needs keyword creates a dependency graph between jobs.

Three layers. Repository secrets (Settings → Secrets → Actions) for most secrets — these are encrypted and masked in logs. Environment secrets for environment-specific values (prod DB host vs dev DB host). The production environment can require review approval before secrets are accessible, preventing accidental prod deployments. OIDC for cloud providers — instead of storing AWS/Azure keys as long-lived secrets, configure GitHub as an OIDC provider in your cloud account. The workflow requests a short-lived token at runtime using id-token: write permission. No credentials stored anywhere. Rotation happens automatically. At HPE: we moved from AWS access keys in secrets to OIDC — eliminated an entire class of credential rotation toil and security risk.

Systematic approach: First, enable debug logging by adding ACTIONS_RUNNER_DEBUG=true as a repository secret and re-running. The detailed logs usually reveal the exact problem. Common causes: 1) Environment difference — local machine has tools installed globally (kubectl, helm, node) that the runner does not have. Fix: add explicit setup steps (actions/setup-node, actions/setup-java). 2) Secret not set — the workflow references a secret that exists in your environment but not in GitHub. Check Settings → Secrets. 3) Permissions — runner cannot push to registry or create releases. Check permissions block in workflow YAML. 4) Path difference — workflow runs from repo root, local you run from a subdirectory. Use working-directory: ./subfolder. 5) Token scope — GITHUB_TOKEN has limited permissions. Check if you need additional permissions.

Use GitHub Environments with protection rules. Create three environments in repo Settings → Environments: staging (no approvals, auto-deploy), uat (1 required reviewer), production (2 required reviewers + branch protection). Each environment can have its own secrets (prod DB password different from staging). In the workflow, set environment: production on the deployment job. GitHub automatically pauses the workflow and sends notifications to required reviewers. If they approve, the workflow continues. If they reject or timeout (configurable), the workflow fails. This gives a complete audit trail — who approved what deployment, when, and from which commit. No Slack messages that disappear, no email threads. All in GitHub history.

Reusable workflows are workflow files that other workflows can call — like functions. You define them once with workflow_call trigger and inputs/secrets parameters. Any other workflow calls them with uses: org/repo/.github/workflows/deploy.yml. Use them when: multiple repos do the same deployment pattern (all microservices deploy to Kubernetes the same way), you want to enforce standards (all deployments must include security scan before push), or you want to simplify individual repo workflows (50-line workflow becomes 10 lines calling reusable workflows). Difference from composite actions: reusable workflows run as separate jobs. Composite actions are steps within a job. Reusable workflows are better for complete deployment flows; composite actions for step-level reuse.

Measure first: Actions → billing shows minutes per workflow. Four main optimisations. One: caching — use actions/cache for node_modules, Maven .m2, pip cache. Saves 2-5 minutes per run. Use hashFiles() to invalidate cache only when dependency files change. Two: parallel vs sequential — check if jobs that run sequentially could run in parallel (needs removed). Three: conditional jobs — skip deployment jobs on PR (if: github.event_name == push). Skip tests on documentation-only changes using paths-ignore or path filters. Four: self-hosted runners — for heavy workloads, run on your own server. No minute counting. Your server, your hardware. Good for Docker builds with large images (10+ GB) where network bandwidth to GitHub runners is the bottleneck.

OIDC (OpenID Connect) lets GitHub Actions authenticate to cloud providers without stored credentials. Traditional approach: store AWS access key ID and secret in GitHub secrets. Problem: long-lived credentials, manual rotation, risk of exposure. OIDC approach: GitHub acts as an identity provider. When a workflow runs, GitHub mints a short-lived JWT token signed by GitHub's private key. Your cloud provider (AWS, Azure, GCP) is configured to trust GitHub's identity provider. The workflow exchanges the GitHub JWT for a cloud-specific credential. The cloud credential expires after 1 hour. Setup: AWS IAM console → Identity providers → Add GitHub as OIDC provider. Create an IAM role with condition that allows only your specific repo/branch. Add permissions: id-token: write to workflow. Use aws-actions/configure-aws-credentials action. Result: no secrets to rotate, no stored credentials, automatic short-lived tokens per run.

Merge: creates a new merge commit combining two branches. Preserves the exact history of both branches including when the feature branch diverged. Non-destructive -- existing commits are never changed. Rebase: moves the entire feature branch to start from the tip of the target branch. Rewrites commit history to appear linear. The feature commits get new SHA hashes. When to use merge: on main/release branches -- never rebase shared branches because rewriting history breaks everyone else who has pulled them. Long-lived feature branches where the history of when the branch diverged is important. When to use rebase: on your local feature branch before creating a PR (interactive rebase to clean up "WIP commit 1", "fix typo", "fix the fix" into meaningful commits). To bring your feature branch up to date with main before merging. Golden rule: never rebase commits that have been pushed to a shared remote branch. If you need to clean up a pushed branch (only yours, no one else has pulled it): git push --force-with-lease. --force-with-lease is safer than --force: it fails if anyone else has pushed since your last pull.

This is a security incident. Speed matters -- bots scan GitHub in under 60 seconds. Step 1 (immediate): revoke the secret. Go to wherever the API key was issued (AWS Console, GitHub Settings, Stripe Dashboard) and invalidate it immediately. Even if you delete it from Git, assume it was already harvested. A new key is your actual protection. Step 2: remove from Git history. If just pushed: git rebase -i HEAD~N to remove the commit, then git push --force-with-lease. For commits deeper in history: git filter-branch or git filter-repo (safer and faster). git filter-repo --path-glob "*.env" --invert-paths removes files. BFG Repo Cleaner: bfg --delete-files .env. Step 3: force all collaborators to re-clone or rebase -- the rewritten history means their local copies have the old commits. Step 4: GitHub has a secret scanning feature -- go to Security tab, check if GitHub already detected and alerted. Step 5: audit access logs. If the key gave cloud access: check CloudTrail (AWS) or Activity Log (Azure) for any actions using that key in the last 24 hours. Prevention: git-secrets pre-commit hook, GitHub Actions secret scanning, .env in .gitignore from day one, secrets manager for all credentials.

OpenShift IS Kubernetes (uses the same API server, etcd, kubelet) but adds enterprise features on top. Key additions: Security Context Constraints (SCC): more granular than K8s Pod Security Standards. Controls UID, capabilities, SELinux context per pod. Default in OpenShift: pods cannot run as root — more secure by default than vanilla K8s. Routes: OpenShift-native Ingress. Simpler than K8s Ingress with annotations. Integrated with HAProxy. TLS termination built-in. Operator Framework: pre-installed Operator Lifecycle Manager. Install complex applications (databases, monitoring) via OperatorHub — one click to install and manage. ImageStreams: abstraction over container images. Track image changes, trigger automatic redeploys when a base image updates. Built-in registry: integrated container registry, no ACR/ECR needed for internal images. OpenShift Console: richer UI than K8s dashboard. Built-in monitoring stack (Prometheus + Grafana via OpenShift Monitoring Operator). Developer experience: oc new-app deploys an application from Git with S2I (source-to-image) without writing a Dockerfile. When to choose OpenShift over AKS: heavily regulated industries (banking, healthcare) needing stricter defaults. Teams wanting an integrated platform (registry, monitoring, CI/CD). On-premise data centre deployments.

SCC is OpenShift equivalent of Kubernetes Pod Security Standards (PSS) but more powerful and granular. SCCs control what a pod is allowed to do at the OS level: which UIDs it can run as, which capabilities it can request, whether it can use host networking/PID/IPC, whether it can mount host paths, what SELinux context it runs under. Standard SCCs: restricted (most secure, no root, no host access -- default for user workloads), anyuid (allow running as any UID including root), privileged (full host access -- only for infrastructure pods like node agents). How pods get SCCs: the pod service account is assigned an SCC via RBAC. oc adm policy add-scc-to-user anyuid -z my-service-account. Common problem: a containerised application that was written to run as root fails in OpenShift with restricted SCC because OpenShift assigns a random high UID. Fix: update the Dockerfile to use a non-root user, or grant anyuid SCC to the service account (less secure but sometimes necessary for third-party apps). K8s PSS comparison: PSS has three levels (privileged, baseline, restricted) applied at namespace level. Less granular than SCC but standardised across clouds. OpenShift 4.x supports both SCC (backward compatibility) and PSS.

import subprocess, json, os, sys def terraform_plan(env, working_dir): """Run terraform plan and return exit code + output""" result = subprocess.run( ["terraform", "plan", "-var-file", f"envs/{env}.tfvars", "-out", f"/tmp/{env}.plan", "-json"], cwd=working_dir, capture_output=True, text=True ) if result.returncode not in [0, 2]: # 0=no changes, 2=changes planned raise RuntimeError(f"terraform plan failed: {result.stderr}") # Parse JSON output to extract resource changes for line in result.stdout.split("\n"): if line.strip(): try: event = json.loads(line) if event.get("type") == "change_summary": changes = event.get("changes", {}) print(f"Plan: +{changes.get("add",0)} ~{changes.get("change",0)} -{changes.get("remove",0)}") # Alert if destroys detected if changes.get("remove", 0) > 0: notify_slack(f"WARNING: Plan includes {changes["remove"]} destroys in {env}") except json.JSONDecodeError: pass return result.returncode # Run: python3 deploy.py production tf_dir = "/infra/environments/production" rc = terraform_plan("production", tf_dir) if rc == 2: # changes to apply if env == "production": wait_for_approval() # check approval ticket subprocess.run(["terraform", "apply", f"/tmp/{env}.plan"], cwd=tf_dir, check=True)

import subprocess, json, re from datetime import datetime, timedelta from collections import Counter def get_pod_logs(namespace, label_selector, since_hours=1): """Get logs from all pods matching a label""" pods_result = subprocess.run( ["kubectl", "get", "pods", "-n", namespace, "-l", label_selector, "-o", "json"], capture_output=True, text=True, check=True ) pods = json.loads(pods_result.stdout)["items"] all_logs = [] for pod in pods: pod_name = pod["metadata"]["name"] log_result = subprocess.run( ["kubectl", "logs", pod_name, "-n", namespace, f"--since={since_hours}h"], capture_output=True, text=True ) for line in log_result.stdout.split("\n"): if line: all_logs.append({"pod": pod_name, "line": line}) return all_logs def analyse_errors(logs): errors = [l for l in logs if "ERROR" in l["line"] or "FATAL" in l["line"]] # Group by error pattern patterns = Counter() for e in errors: # Extract error type (first word after ERROR:) m = re.search(r"ERROR:?\s*(\w+)", e["line"]) if m: patterns[m.group(1)] += 1 print(f"Total errors: {len(errors)}") for pattern, count in patterns.most_common(5): print(f" {pattern}: {count}") logs = get_pod_logs("production", "app=payment-api") analyse_errors(logs)

import subprocess, json, sys def check_pods(namespace): result = subprocess.run( ["kubectl","get","pods","-n",namespace,"-o","json"], capture_output=True, text=True ) if result.returncode != 0: print(f"Error: {result.stderr}"); sys.exit(1) pods = json.loads(result.stdout)["items"] if not pods: print(f"No pods in {namespace}"); return True failed = [] for pod in pods: name = pod["metadata"]["name"] phase = pod["status"].get("phase", "Unknown") if phase != "Running": failed.append(f"{name}: {phase}") if failed: print(f"FAILED pods in {namespace}:") for f in failed: print(f" {f}") return False else: print(f"All {len(pods)} pods Running in {namespace}") return True if not check_pods("production"): sys.exit(1) # fail the CI/CD step

import psutil, requests, os, time SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK_URL"] # never hardcode THRESHOLD = 80 HOSTNAME = os.uname().nodename def send_alert(cpu): payload = { "text": f":red_circle: *CPU Alert* on `{HOSTNAME}`", "attachments": [{ "color": "danger", "fields": [{ "title": "CPU Usage", "value": f"{cpu:.1f}% (threshold: {THRESHOLD}%)", "short": True }] }] } resp = requests.post(SLACK_WEBHOOK, json=payload) resp.raise_for_status() print(f"Alert sent: CPU={cpu:.1f}%") while True: cpu = psutil.cpu_percent(interval=5) if cpu > THRESHOLD: send_alert(cpu) time.sleep(60) # check every minute

from azure.identity import DefaultAzureCredential from azure.mgmt.containerservice import ContainerServiceClient import os # DefaultAzureCredential works with: managed identity (in Azure), # service principal env vars, Azure CLI login (local dev) credential = DefaultAzureCredential() subscription_id = os.environ["AZURE_SUBSCRIPTION_ID"] client = ContainerServiceClient(credential, subscription_id) print("AKS Clusters:") for cluster in client.managed_clusters.list(): print(f" {cluster.name}: {cluster.location} | " f"K8s: {cluster.kubernetes_version} | " f"Nodes: {sum(p.count for p in cluster.agent_pool_profiles)}") # Or using raw requests + token: token = credential.get_token("https://management.azure.com/.default") headers = {"Authorization": f"Bearer {token.token}"} url = f"https://management.azure.com/subscriptions/{subscription_id}/providers/Microsoft.ContainerService/managedClusters?api-version=2024-01-01" import requests resp = requests.get(url, headers=headers) for cluster in resp.json()["value"]: print(cluster["name"])

Production scripts must be resilient. Error handling pattern: import time, functools, logging logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") def retry(max_attempts=3, delay=5, exceptions=(Exception,)): """Decorator for automatic retry with exponential backoff""" def decorator(func): @functools.wraps(func) def wrapper(*args, **kwargs): for attempt in range(1, max_attempts+1): try: return func(*args, **kwargs) except exceptions as e: if attempt == max_attempts: logging.error(f"{func.__name__} failed after {max_attempts} attempts: {e}") raise wait = delay * (2 ** (attempt-1)) # exponential backoff logging.warning(f"Attempt {attempt} failed: {e}. Retrying in {wait}s...") time.sleep(wait) return wrapper return decorator @retry(max_attempts=3, delay=10) def deploy_to_kubernetes(manifest_path): result = subprocess.run(["kubectl","apply","-f",manifest_path], capture_output=True, text=True, check=True) return result.stdout # Always log start, end, and key decisions # Always send alerts on final failure (Slack, PagerDuty, email) # Never silently swallow exceptions in production

#!/bin/bash set -euo pipefail SERVICES=( "payment-api:production:8080:/health" "order-api:production:8080:/health" "inventory:production:8080:/actuator/health" ) FAILED=0 for svc in "${SERVICES[@]}"; do IFS=":" read -r name namespace port path <<< "$svc" # Port-forward in background kubectl port-forward "svc/$name" "1${port}:${port}" \ -n "$namespace" &>/dev/null & PF_PID=$! sleep 2 # Wait for port-forward # Health check HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \ "http://localhost:1${port}${path}" \ --connect-timeout 5 --max-time 10) kill $PF_PID 2>/dev/null || true if [[ "$HTTP_CODE" == "200" ]]; then echo "✅ $name: HEALTHY (HTTP $HTTP_CODE)" else echo "❌ $name: UNHEALTHY (HTTP $HTTP_CODE)" FAILED=$((FAILED+1)) fi done if [[ $FAILED -gt 0 ]]; then echo "ALERT: $FAILED service(s) unhealthy" exit 1 fi echo "All services healthy"

#!/bin/bash set -euo pipefail # ── Logging functions ────────────────────────────────── log() { echo "[$(date "+%Y-%m-%d %H:%M:%S")] INFO $*"; } warn() { echo "[$(date "+%Y-%m-%d %H:%M:%S")] WARN $*" >&2; } error() { echo "[$(date "+%Y-%m-%d %H:%M:%S")] ERROR $*" >&2; exit 1; } # ── Retry function ───────────────────────────────────── retry() { local max_attempts="$1"; shift local delay="$1"; shift local cmd="$@" local attempt=1 while [[ $attempt -le $max_attempts ]]; do if eval "$cmd"; then return 0 fi warn "Attempt $attempt/$max_attempts failed. Retrying in ${delay}s..." sleep "$delay" attempt=$((attempt + 1)) delay=$((delay * 2)) # exponential backoff done error "$cmd failed after $max_attempts attempts" } # ── Cleanup trap ─────────────────────────────────────── cleanup() { log "Cleaning up temp files..." rm -f /tmp/deploy-$$.* # Kill any background processes jobs -p | xargs -r kill 2>/dev/null || true } trap cleanup EXIT # ── Usage ────────────────────────────────────────────── retry 3 5 kubectl rollout status deployment/myapp -n production retry 3 10 helm upgrade --install myapp ./chart --atomic

#!/bin/bash set -euo pipefail # exit on error, undefined vars, pipe failures # Variables from CI/CD environment APP_NAME="${APP_NAME:?APP_NAME required}" IMAGE_TAG="${IMAGE_TAG:?IMAGE_TAG required}" NAMESPACE="${NAMESPACE:-production}" ACR_NAME="${ACR_NAME:?ACR_NAME required}" IMAGE="${ACR_NAME}.azurecr.io/${APP_NAME}:${IMAGE_TAG}" echo "Deploying ${APP_NAME}:${IMAGE_TAG} to ${NAMESPACE}" # Update the deployment image kubectl set image deployment/${APP_NAME} \ ${APP_NAME}=${IMAGE} \ -n ${NAMESPACE} # Wait for rollout kubectl rollout status deployment/${APP_NAME} \ -n ${NAMESPACE} \ --timeout=300s if [ $? -eq 0 ]; then echo "Deployment successful" # Verify pods are running kubectl get pods -n ${NAMESPACE} -l app=${APP_NAME} else echo "Deployment failed - rolling back" kubectl rollout undo deployment/${APP_NAME} -n ${NAMESPACE} exit 1 fi

#!/bin/bash # Find errors from last hour in application logs LOG_FILE="/var/log/app/application.log" ONE_HOUR_AGO=$(date -d "1 hour ago" "+%Y-%m-%d %H:%M:%S") # Count errors in last hour ERROR_COUNT=$(awk -v cutoff="$ONE_HOUR_AGO" \ '$1" "$2 >= cutoff && /ERROR/ {count++} END {print count+0}' \ "$LOG_FILE") echo "Errors in last hour: $ERROR_COUNT" # Extract unique error messages echo "Top error messages:" grep "ERROR" "$LOG_FILE" | \ awk '{for(i=4;i<=NF;i++) printf $i" "; print ""}' | \ sort | uniq -c | sort -rn | head -10 # Check pod logs directly from Kubernetes # Get logs from all pods matching a label kubectl get pods -n production -l app=payment-api -o name | \ xargs -I {} kubectl logs {} -n production --since=1h | \ grep ERROR | \ sort | uniq -c | sort -rn | head -20 # Alert if error count exceeds threshold if [ "$ERROR_COUNT" -gt 100 ]; then echo "ALERT: High error rate - $ERROR_COUNT errors in last hour" # Add Slack/PagerDuty notification here fi

set -e: exit immediately if any command returns non-zero exit code. Without this: a failed command is silently ignored and the script continues in a broken state. With -e: if kubectl apply fails, the script stops immediately — you know about it. set -u: treat unset variables as errors. Without this: if $APP_NAME is undefined, $APP_NAME expands to empty string silently. kubectl set image deployment/ would run with an empty deployment name. With -u: script exits with "unbound variable" error — you know a required variable was not set. set -o pipefail: in a pipeline (cmd1 | cmd2 | cmd3), the exit code of the whole pipeline is the exit code of the LAST command. Without pipefail: kubectl get pods | grep ERROR | sort — if kubectl fails, grep and sort might still succeed, and the overall exit code is 0 (success!). With pipefail: if any command in the pipeline fails, the whole pipeline is considered failed. Together set -euo pipefail means: the script is strict. Any error, undefined variable, or pipe failure stops the script immediately. This is the production standard. Use it on the very first line of every script after the shebang.

#!/bin/bash set -euo pipefail NAMESPACE="production" # Method 1: check exit code if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then echo "Namespace $NAMESPACE already exists" else echo "Creating namespace $NAMESPACE" kubectl create namespace "$NAMESPACE" # Add labels for network policy kubectl label namespace "$NAMESPACE" env=production team=platform fi # Method 2: using --dry-run (idempotent approach) kubectl create namespace "$NAMESPACE" \ --dry-run=client -o yaml | kubectl apply -f - # Idempotent: if exists, no-op. If not, creates it. # This is the preferred production approach # Method 3: wait for namespace to be Active kubectl wait --for=jsonpath={.status.phase}=Active \ namespace/$NAMESPACE --timeout=30s echo "Namespace $NAMESPACE is Active and ready"

At HPE on the telecom provisioning platform (OCP cluster serving 3 network provisioning systems). Alert at 2 AM: all provisioning requests failing, 100% error rate. Impact: new SIM cards and number activations for thousands of customers failing. My process: Minute 0 — acknowledged alert, opened incident channel, messaged stakeholders "Investigating outage in provisioning platform". Minute 2 — kubectl get pods -A: 40% of pods in CrashLoopBackOff across all namespaces. kubectl get nodes: all nodes showing MemoryPressure. kubectl top nodes: memory at 98-100% on every node. Minute 5 — root cause: a recent configuration change to the Oracle database connection pool settings had removed the pool size limit. Each pod was creating 200 connections instead of 20. The connection pool memory overhead OOMKilled the pods. Pods restarted, created more connections, more OOM. A feedback loop. Minute 8 — mitigation: kubectl scale deployment oracle-adapter --replicas=1 in each namespace. Reduced connection load immediately. Nodes stabilised. Minute 15 — services restored. Root cause fix: rolled back the ConfigMap change. Pods restarted cleanly with correct pool settings. Post-incident: added a Prometheus alert for database connection count. Added ResourceQuota for memory per namespace. Added PodDisruptionBudget. Documented in runbook. Changed review process for database config changes.

Management does not care about deployment frequency or MTTR in isolation. They care about business outcomes. Translation: Deployment frequency 30→ per day: "We now deliver features to customers 5x faster. Competitors take 2 weeks. We take 2 days." MTTR 4 hours→ 20 minutes: "Average revenue lost per incident dropped from ₹2L to ₹17K based on transaction volume during downtime." Change failure rate 15%→ 3%: "Rollbacks in production dropped from 18 per quarter to 4. Each rollback previously caused 2-hour customer impact." Pipeline automation: "Manual deployment process took 6 hours of engineer time. Now fully automated. Saves 180 engineer-hours per month = ₹3.6L monthly at loaded cost." Infrastructure cost: "Right-sizing and autoscaling reduced Azure spend by 35%. Saving ₹18L annually." Security: "Trivy scanning caught 47 CRITICAL CVEs before they reached production this quarter." Present as a quarterly Business Review: show trend graphs (4 quarters of DORA metrics), show cost trend (infrastructure spend over time), show security posture (CVEs caught vs missed). Ask management: "What do you want us to improve next quarter? Faster feature delivery, fewer incidents, or lower cost?" Let them prioritise. This creates partnership instead of reporting.

On-call is a system, not a heroic individual. Good on-call has: Clear escalation: L1 (automated recovery), L2 (on-call engineer), L3 (senior + management). Alert quality: every alert is actionable. No alert fires without a clear runbook. No flapping alerts. We reviewed and culled alerts quarterly — removed 40% that were noise. Runbooks: for every alert, a runbook with: what it means, initial triage commands, common causes and fixes, escalation criteria. New team member should be able to follow it at 2 AM. Rotation fairness: maximum 1 week per rotation. No more than 2 incidents per night (if exceeded, root cause must be fixed before next rotation). Post-incident process: every P1 gets a blameless RCA within 24 hours. Action items must be implemented within 1 sprint. Repeat incidents get root cause eliminated (not just fixed). At HPE: we had a certificate expiry alert that kept firing quarterly. Instead of just renewing the cert each time, I implemented cert-manager for automatic renewal. The alert never fired again. On-call health: monitor on-call burden. Track: alerts per week per engineer, sleep-affecting alerts, time to resolve. If any metric exceeds threshold: stop feature work, fix the reliability problem first.

Architecture: each microservice has its own pipeline but all use the same template. Jenkins shared library (or Azure DevOps template) defines the standard stages. Stage 1 — CI (runs on every PR): checkout, unit tests, code coverage check, SonarQube quality gate (abort if fails), Trivy image scan (abort on CRITICAL), docker build, push to ACR with PR tag. Stage 2 — Dev deploy (auto on merge to develop): helm upgrade --install --atomic --namespace dev --set image.tag=${BUILD_ID}. Run smoke tests. Stage 3 — Staging deploy (auto on merge to main): deploy to staging, run integration tests, performance test (k6 or JMeter), security scan against running app (DAST). Stage 4 — Production deploy (manual approval): raise CR automatically, wait for approval, deploy during maintenance window, helm upgrade --atomic --timeout 10m, automated smoke tests, monitor for 30 minutes, close CR. Canary for critical services: Argo Rollouts with 10% → analysis → 30% → analysis → 100% pattern. Each microservice has its own ArgoCD Application. The App-of-Apps root application manages all 10. Any deployment visible to the whole team in ArgoCD UI. Rollback: one click in ArgoCD or helm rollback.

Defence in depth — security at every stage: Stage 1 — Developer workstation: pre-commit hooks run dockerfile-lint (Hadolint) and basic secrets detection (detect-secrets). Catches obvious issues before they reach CI. Stage 2 — CI on every commit: Trivy scans the built image for OS CVEs, language CVEs, and IaC misconfigurations. FAIL on CRITICAL, alert on HIGH. Checkov scans Terraform and Kubernetes YAML for misconfigurations (public storage, no resource limits, privileged containers). Semgrep static analysis on application code. Stage 3 — Registry: ACR Content Trust with Notary — only signed images can be pulled by AKS. Azure Defender for Containers — continuous monitoring of images in ACR, alerts on new CVEs. Tag immutability — :latest cannot be overwritten. Stage 4 — Cluster admission: Azure Policy for Kubernetes (OPA Gatekeeper) enforces at admission time: images must come from ACR only, must have passed Trivy scan, must not run as root, must have resource limits. Stage 5 — Runtime: Falco DaemonSet detects runtime anomalies — shell spawned in container, unexpected network connection, sensitive file access. Result: a CVE in a base image is caught in CI before deployment, or if it emerges post-deployment, Defender alerts and Falco detects exploitation attempts.

This is about speed, communication, and systematic thinking under pressure. Minute 0 — acknowledge: respond to the PagerDuty alert within 3 minutes. Open incident channel. "Acknowledged, investigating, will update in 10 minutes." Never go silent. Minute 1-3 — understand blast radius: how many users affected? Is it all traffic or specific region/service? Check the monitoring dashboard first — HTTP error rate spike. When did it start? Correlate with recent deployments. Minute 3-8 — isolate: kubectl get pods -n production — any CrashLoopBackOff? kubectl get events -n production --sort-by=lastTimestamp — what happened recently? Check the service that is 503ing: is it the pod that is broken or its dependency? Check: was there a deployment in the last 30 minutes? Check downstream: if payment-api is 503, is it because the database is unreachable? Minute 8-15 — fix: if bad deployment → kubectl rollout undo deployment/payment-api. If database connection issue → check connection pool, restart pods with --max-unavailable=0. If node pressure → cordon the node, drain pods to healthy nodes. Minute 15 — communicate: "Root cause identified: deployment at 14:32 introduced a misconfigured database URL. Rolled back at 14:47. Services restored. Monitoring." Post-incident (24 hours): full RCA, timeline, what we missed, what we are changing.

First question: is there a way to deploy without the pipeline? I always keep an emergency deployment runbook. Then triage: flaky test (skip with justification, document, fix after release), broken tool -- SonarQube down, registry unreachable (bypass with tech lead approval), actual code issue (delay the release -- no release is worth pushing broken code). I communicate immediately to all stakeholders: "Pipeline issue detected at X, investigating, update in 15 minutes." Never go silent. If bypassing: I need written confirmation from tech lead level that the decision was made and risk accepted. I create an immediate ticket to fix within 24 hours. The key is: I own the outcome either way. If I delay, I own the delay. If I bypass and something breaks, I own that too. Own your decisions clearly.

Resistance comes from fear: fear of more work, fear of breaking things, fear of losing control. I never lead with process or tools. I lead with pain. First, I listen to understand what actually hurts. Long release cycles? Too many production incidents? Weekend deployments taking 6 hours? Then I show a small win. Take the most painful thing and fix it. If deployments take 6 hours manually, automate one step -- not everything, just one. Show the team their deployment time dropped to 4 hours. Now they want more. At HPE, the team resisted automated testing because "it takes time to write tests." I took the most common production bug from the last quarter, wrote a test that would have caught it, and showed it. The next incident happened and the test caught it before prod. The team started writing tests themselves. The pattern: show the pain, fix one piece, celebrate the win, let the team pull you forward. Mandate from above creates compliance. Demonstrated value creates ownership. Ownership is what you actually want.

People problem disguised as a technical problem. Technical fix: remove direct prod access -- all deployments go through the pipeline, no exceptions. Implement audit logging so every manual action is visible to everyone. People fix: understand WHY they are bypassing. Always a reason. Usually: pipeline is too slow (45 minutes is unacceptable), too many false failures (flaky tests blocking good code), bureaucratic approvals for trivial changes. At HPE a developer was bypassing because our pipeline took 45 minutes. We parallelised the test stages, added caching, optimised the Docker build. Pipeline went to 12 minutes. Bypassing stopped completely. Fix the real problem. If direct access is needed for genuine emergencies: create a documented emergency process with required ticket, manager approval, and mandatory post-incident review. This channels the behaviour constructively rather than just blocking it. The last resort: if bypasses continue after the real problem is fixed, it is now a performance issue handled by management.

GitOps is an operational model where Git is the single source of truth for infrastructure and application state. Every change to a production system happens through a Git commit and PR review — no direct kubectl apply, no console clicks, no shell scripts run manually. ArgoCD implements GitOps by continuously watching a Git repository. Every 3 minutes (default), ArgoCD fetches the desired state from Git and compares it with the actual state of the Kubernetes cluster. If they differ (sync needed), ArgoCD can automatically apply the Git state to the cluster. The critical shift: developers push to Git. ArgoCD pulls and applies. Nobody needs direct cluster credentials. Every deployment is a commit with author, timestamp, and review. Rollback is git revert — takes 2 minutes instead of 30. Drift detection: if someone manually kubectl applies something, ArgoCD marks the app as OutOfSync and can revert it automatically with selfHeal: true. At HPE: ArgoCD manages all telecom platform deployments. When a microservice update causes issues, rollback is git revert the image tag change. ArgoCD applies it within 3 minutes.

App of Apps is a pattern where a parent ArgoCD Application watches a directory in Git that contains other Application YAML files. The parent app manages the child apps. When you commit a new Application YAML file to that directory, ArgoCD automatically creates and manages that new application. Use App of Apps when: you have many services (10+) and want to manage them consistently, you want a single sync to bootstrap an entire environment, or you want ArgoCD to manage its own applications (self-managing). The directory structure: argocd/production/ contains: sro-app.yaml, com-app.yaml, monitoring.yaml. The parent app watches that directory. Adding a new service is just adding a new YAML file. Promoted from HPE design: we bootstrap staging with one parent app sync that creates 15 child applications in the right order using sync waves. Complete environment up in under 10 minutes.

OutOfSync means the actual cluster state differs from what is in Git. Step 1: identify what is different. argocd app diff app-name shows the exact diff — like git diff but for Kubernetes resources. Step 2: determine cause. Common causes: someone ran kubectl apply manually (config drift), Kubernetes mutated a resource (annotations, status fields), the app changed its own config at runtime (bad practice), or a new commit to Git changed something. Step 3: if the diff is intentional Git change — sync it. argocd app sync app-name. Step 4: if the diff is manual drift — the Git version is correct, revert the manual change. With selfHeal: true, ArgoCD does this automatically. Without it: argocd app sync --force. Step 5: if neither — investigate what changed. kubectl describe resource shows last-applied-configuration. Check audit logs. Step 6: prevent recurrence. Enable selfHeal: true in sync policy. Restrict direct kubectl access using Kubernetes RBAC or admission webhooks. Rule: in production, Git is always right. Manual changes should be exceptions with immediate follow-up to update Git.

ArgoCD can manage multiple Kubernetes clusters from one ArgoCD instance. Register each cluster: argocd cluster add context-name. Each Application's destination.server points to the cluster API endpoint. For environment promotion: three clusters (dev, staging, prod), separate Application objects pointing to each cluster, same Git repo but different branches or folder paths (environments/dev/, environments/staging/, environments/prod/). Promotion flow: merge PR to dev branch → ArgoCD syncs dev cluster. After QA approval: PR to update image tag in environments/staging/ → ArgoCD syncs staging cluster. After approval: PR to update environments/prod/ → ArgoCD syncs prod cluster. Each environment has its own ArgoCD Project with RBAC. Dev cluster: auto-sync allowed for all. Prod cluster: sync windows restrict deployments to business hours, requires manual sync approval. ApplicationSets (ArgoCD feature): generate Applications for multiple clusters from one template — useful when you have 10+ clusters following the same pattern (multi-region or multi-tenant).

The GitOps dilemma: everything is in Git, but secrets cannot be in Git (even encrypted, it is risky). Solutions: Sealed Secrets (Bitnami): kubeseal encrypts a Secret with the cluster's public key. Only the cluster can decrypt it. The SealedSecret (encrypted) is safe to commit to Git. ArgoCD syncs SealedSecrets, the controller decrypts them into real Secrets. Works fully GitOps — no external secret store required. External Secrets Operator: defines ExternalSecret resources in Git that reference secrets in an external store (Azure Key Vault, AWS Secrets Manager, HashiCorp Vault). The operator fetches and creates Kubernetes Secrets. ArgoCD syncs the ExternalSecret definition (safe to commit), the operator handles the actual secrets. Best for enterprise: centralised secret rotation, audit trail, compliance. Vault Agent Injector: HashiCorp Vault sidecar injected into pods fetches secrets at runtime, writes to in-memory filesystem. Application reads from /vault/secrets/. No Kubernetes Secrets at all. Most secure but most complex. My production recommendation: External Secrets Operator + Azure Key Vault. Secrets defined in code (the ExternalSecret manifest), stored in Key Vault, rotated in Key Vault. ArgoCD monitors the ExternalSecret, ESO syncs changes. Full GitOps, no secrets in Git, centralised management.

Both are CNCF GitOps tools. I have used ArgoCD in production. ArgoCD: multi-cluster management from one UI, excellent visibility into sync status, RBAC built into the UI, supports Helm/Kustomize/plain YAML, App-of-Apps pattern for bootstrapping entire clusters. Best when: team needs visibility, multiple clusters to manage, non-platform engineers need to see deployment status. FluxCD: CLI-first, no UI (Weave GitOps adds one), modular controllers, built-in image automation (auto-commit new image tags to Git), better multi-tenancy with namespace scoping. Best when: platform team manages it exclusively, you want Kubernetes-native approach, you need image automation without extra tools. Choose ArgoCD when: product team wants self-service deploys with visibility. Choose FluxCD when: pure GitOps, platform-team-managed, CLI workflow preferred. In practice: many enterprises use ArgoCD for application delivery and FluxCD for cluster bootstrapping. They can coexist. My recommendation for your context: ArgoCD — the UI is critical for communicating deployment status to non-platform stakeholders.

Step 1: read the sync error in the ArgoCD UI. Application → Sync Status → click the error. Common errors: "namespace not found" (the target namespace does not exist — create it or add CreateNamespace sync option), "resource already exists" (manual resource in the cluster not in Git — delete the manual resource or use kubectl apply --server-side), RBAC error (ArgoCD service account lacks permission to create this resource type). Step 2: check the diff. ArgoCD shows exactly what differs between Git and cluster. Sometimes it is a field being continuously mutated by a controller (like lastAppliedConfiguration annotation) — use ignoreDifferences to tell ArgoCD to ignore that field. Step 3: check the application logs. kubectl logs -n argocd deploy/argocd-application-controller | grep my-app. Step 4: verify the Git repository is accessible. ArgoCD repository page shows connection status. Expired credential, wrong branch name, or private repo without SSH key all cause sync failure. Step 5: if Helm-based app: render the manifests locally first. helm template my-release ./chart -f values.yaml. If template fails locally, it will fail in ArgoCD too. Fix the template, push to Git, ArgoCD syncs.

Helm is the package manager for Kubernetes. Without Helm: to deploy an application you have multiple YAML files (Deployment, Service, Ingress, ConfigMap, Secret) and you manually edit each file for different environments (change image tag, change replica count, change hostname). This is error-prone and hard to version. Helm solves this with charts: a chart packages all the YAML files as templates with variables. You provide values (replicaCount: 3, image.tag: v1.2.0) and Helm renders the final YAML. The same chart deploys to dev, staging, and prod with different values files. Upgrade is one command: helm upgrade --install. Rollback is one command: helm rollback. History is tracked: helm history shows every deployment. This is why Helm is the standard way to deploy applications on Kubernetes — every major tool (Prometheus, ArgoCD, nginx ingress, cert-manager) is distributed as a Helm chart.

helm install fails if the release already exists (the name is taken). helm upgrade --install (also called atomic upsert) installs the release if it does not exist, upgrades it if it does. This makes it idempotent — safe to run multiple times, perfect for CI/CD pipelines where you don't know if this is first deployment or an upgrade. Use helm upgrade --install with --atomic flag in production: if the upgrade fails (pods don't become ready within timeout), Helm automatically rolls back to the previous revision. Without --atomic, a failed upgrade leaves the release in a FAILED state and you must manually rollback. Also use --wait with --timeout 5m so the command waits until all pods are running before returning success. This ensures your CI/CD pipeline knows if the deployment actually worked.

Layered values files. Base values.yaml contains defaults that work for any environment (usually dev-sized). Environment-specific files override only what differs. helm upgrade --install myapp ./chart -f values.yaml -f values-production.yaml — later files override earlier ones. Production values-production.yaml only needs to specify what is different: larger resource limits, pinned image tag instead of latest, external database instead of local PostgreSQL, correct hostname, production-specific annotations. This keeps values-production.yaml small and focused. For image tags: never put the image tag in Git-committed values files for production. Pass it at deploy time: --set image.tag=$CI_COMMIT_SHA. This way Git does not have a rapidly-changing file, and every deploy has a unique traceable tag. Secret management: never put secrets in values files committed to Git. Use helm secrets plugin for encrypted values or pass secrets via --set from CI/CD secret variables.

First: if you used --atomic, Helm already rolled back automatically. helm history shows: revision 5 failed, rolled back to revision 4 is active. Check what the failure was: helm status myapp shows the error. Check pod events: kubectl describe pod and kubectl logs for the failed pods. Second: if you did NOT use --atomic and the release is in FAILED state: helm rollback myapp 4 to roll back to last known good revision. This takes seconds. Third: investigate the failure. helm template myapp ./chart --values values.yaml > rendered.yaml and compare with working version. kubectl apply --dry-run=client -f rendered.yaml catches most issues. Common failure causes: image does not exist in registry (wrong tag, registry authentication), resource limits too low (pod OOMKilled), health check misconfigured (readiness probe fails, Helm times out waiting for ready). Prevention: always use --atomic --wait --timeout 10m. Add staging deployment before prod. Use helm diff (plugin) to preview changes before upgrading. Add smoke tests as a post-upgrade step in CI/CD.

A Helm chart is a directory with this structure: Chart.yaml — metadata: chart name, version, appVersion (the app image tag), description, dependencies list. values.yaml — default configuration values. Any value here can be overridden at install time. templates/ directory — Kubernetes YAML files with Go template syntax. _helpers.tpl — named template functions reused across manifests (define once, include everywhere). templates/deployment.yaml, service.yaml, ingress.yaml, configmap.yaml — standard K8s resources. templates/NOTES.txt — displayed to user after helm install, typically shows how to access the app. charts/ — sub-charts (dependencies). .helmignore — files to exclude from packaging (like .git, README). When you run helm install, Helm renders all templates/ files by combining the template syntax with values (from values.yaml overridden by your -f flags or --set), producing plain Kubernetes YAML, then applies it to the cluster. The key design principle: templates should be generic, values should be environment-specific. Your templates/deployment.yaml should work for dev, staging, and production — the differences come entirely from values files.

Environment-specific values override the defaults in values.yaml. Structure: values.yaml (base defaults — used in development), values-staging.yaml (staging overrides), values-production.yaml (production overrides). Install command: helm install myapp ./mychart -f values-production.yaml. The -f flag merges production values ON TOP of the base values.yaml. Only the keys present in values-production.yaml override the defaults. Using --set for single overrides: helm upgrade myapp ./mychart --set image.tag=v2.1.0. In CI/CD: helm upgrade --install myapp ./mychart -f values-${ENVIRONMENT}.yaml --set image.tag=${IMAGE_TAG} --atomic --wait. The --atomic flag: if the upgrade fails, automatically rolls back to the previous release. The --wait flag: waits for all pods to be Running and ready before returning success. Production best practice: never use helm install (fails if already exists) — always use helm upgrade --install (upgrades or installs). Never use --set for many values in CI/CD — use -f with a committed values file so changes are tracked in Git. ArgoCD with Helm: ArgoCD can manage Helm releases — you commit values file changes to Git, ArgoCD detects the change and runs helm upgrade automatically.

First: understand the state. helm status myapp shows the release status: deployed, failed, pending-upgrade, or superseded. helm history myapp shows all revisions with their status. If upgrade failed: helm rollback myapp 5 (roll back to revision 5 — the last working one). Helm rollback re-applies the previous revision manifests. Check: kubectl get pods after rollback — are pods healthy? If rollback also fails (rare but happens with CRDs or irreversible changes): helm status shows you the release is stuck. To force it: helm upgrade myapp ./mychart --force --atomic reuses the chart but force-replaces pods. Nuclear option: helm uninstall myapp then helm install — but this causes downtime. Debugging failed upgrade: helm upgrade myapp ./mychart --debug 2>&1 | tee upgrade.log shows you exactly what Kubernetes returned. Look for: invalid YAML, missing required values, resource quota exceeded, PVC not found. Prevention: always use --atomic on upgrades in CI/CD. It automatically rolls back on failure, so a failed pipeline never leaves the cluster in a broken intermediate state. Also: run helm template . -f values-prod.yaml before the upgrade to catch template rendering errors before touching the cluster.

Helm dependencies (subcharts) allow one chart to include others. Defined in Chart.yaml under dependencies. Example: your application chart depends on PostgreSQL and Redis. Chart.yaml: dependencies: - name: postgresql, version: "12.x.x", repository: https://charts.bitnami.com/bitnami. - name: redis, version: "17.x.x", repository: https://charts.bitnami.com/bitnami. Run helm dependency update to download them into charts/ directory. Values for subcharts are namespaced: postgresql.auth.password: "mypassword" in your values.yaml configures the postgresql subchart. Two strategies for microservices: Strategy 1 (Umbrella chart): one parent chart with all microservices as subcharts. One helm upgrade deploys everything. Simple but: one bad subchart can block the whole deployment. Strategy 2 (Independent charts per service): each microservice has its own chart, deployed independently by ArgoCD. More resilient — payment service chart can deploy without touching order service. This is the preferred approach for large systems. At HPE: each microservice (TeMIP, UOC, UTM, UCA) had its own Helm chart managed as an independent ArgoCD Application. The charts were stored in a dedicated Helm charts Git repo, versioned separately from application code.

Check the release status: helm status my-release -n production. If it shows "failed" or "pending-upgrade", Helm has left the release in a bad state. Check history: helm history my-release -n production shows all revisions with their status and timestamps. Rollback: helm rollback my-release [REVISION] -n production. Example: helm rollback my-release 3 rolls back to revision 3. This redeploys the previous chart version with the previous values. If rollback also fails: this usually means the cluster state is inconsistent (partially applied resources). Manually check: kubectl get all -l app.kubernetes.io/instance=my-release -n production. Delete the broken resources manually if needed, then run helm rollback. If the release is stuck in "pending-upgrade": helm history shows it as pending forever, new upgrades fail with "cannot re-use a name that is still in use." Force unlock: kubectl delete secret -n production sh.helm.release.v1.my-release.v[N] deletes the stuck history entry. Then retry the upgrade. Prevention: use helm upgrade --atomic which automatically rolls back if any deployment fails its readiness check within --timeout. Add --timeout 5m --wait to all production upgrades.

Bicep is Microsoft's domain-specific language for Azure infrastructure. It compiles to ARM JSON templates but has much simpler syntax. ARM templates: verbose JSON, hard to read, difficult to maintain (100 lines in Bicep = 300+ lines in ARM JSON). Bicep: clean declarative syntax, type safety, IDE autocomplete, native Azure support. Terraform: multi-cloud (works on AWS, Azure, GCP), larger community, better module ecosystem, but requires separate installation and understanding of HCL. When to use Bicep: Azure-only infrastructure, want native Azure tooling, team is already in Azure DevOps ecosystem. When to use Terraform: multi-cloud, need complex state management across providers, team already knows Terraform. Many teams use both: Terraform for infrastructure (AKS cluster, networking), Bicep for Azure-specific services (Azure DevOps settings, Azure-specific policies).

Azure Policy for Kubernetes uses OPA Gatekeeper to enforce rules at admission time — before resources are created. Built-in policies enforce: containers must not run as root, images must come from allowed registries (ACR only), pods must have CPU/memory limits, no privileged containers, required labels on all resources. Assign policies: az policy assignment create --name require-resource-limits --policy-definition-id /providers/Microsoft.Authorization/policyDefinitions/... --scope /subscriptions/.../resourceGroups/.../providers/Microsoft.ContainerService/managedClusters/myAKS. Effect: Audit (logs violations but allows), Deny (blocks non-compliant resources). Custom policies: write ConstraintTemplate (defines the constraint schema as Rego code) and Constraint (instance of that template). Example: require all pods to have team label. Constraint: spec.match.kinds: [Pods], spec.parameters.requiredLabels: [team]. Test: kubectl apply pod without team label → admission rejected with policy violation message. Policy compliance: Azure Portal shows all non-compliant resources and remediation suggestions.

An Operator is a Kubernetes controller that automates operational tasks for a specific application. It extends the K8s API with Custom Resource Definitions (CRDs) and watches those resources to take action. Built-in controllers handle generic workloads (Deployment, StatefulSet). Operators handle application-specific logic. Example: PostgreSQL Operator manages: creating a database cluster (CRD: kind: PostgresCluster), taking backups on schedule, performing failover when primary dies, scaling read replicas, rotating passwords. Without operator: ops team manually handles all of this. With operator: it is automated and self-healing. When to create an operator: you have a stateful application with complex day-2 operations that cannot be modelled with standard K8s resources. Examples: database operators (CloudNativePG, MongoDB Community), messaging operators (Strimzi for Kafka), certificate operators (cert-manager). Build with: Operator SDK (Go or Ansible based). For most teams: use an existing operator from OperatorHub rather than building one. Building is complex — Go programming required for most.

Systematic network debugging: Step 1: verify basic pod connectivity. kubectl exec pod-a -- wget -qO- http://pod-b-ip:port. If this fails: Step 2: check NetworkPolicies. kubectl get networkpolicies -n namespace. If any policies exist, they might be blocking. Temporarily delete the policy to test: if communication works without the policy, the policy is the culprit. Step 3: check Service. kubectl get svc myservice. kubectl describe svc myservice — check if Endpoints field has pod IPs. If Endpoints is empty: label selector on Service does not match pod labels. kubectl get pod --show-labels. Step 4: check CoreDNS. kubectl exec pod-a -- nslookup myservice.namespace.svc.cluster.local. If DNS fails: kubectl get pods -n kube-system | grep coredns. Check CoreDNS logs. Step 5: check kube-proxy (or Cilium). kubectl get pods -n kube-system | grep kube-proxy. If using Cilium: cilium connectivity test runs a full connectivity validation. Step 6: use ephemeral debug container: kubectl debug pod-a -it --image=nicolaka/netshoot -- bash. netshoot has all networking tools (tcpdump, curl, netstat, dig).

Data sources allow Terraform to READ information about existing resources without managing them. You use data sources when: the resource was created outside this Terraform code (created manually, by another team, or by a different Terraform state), you need to reference an existing resource by name or tag, you want to look up dynamic values (latest AMI ID, current subscription ID). Examples: data "azurerm_resource_group" "existing" { name = "shared-rg" } — reference an existing resource group created by another team. data "azurerm_client_config" "current" {} — get current subscription ID and tenant ID. data "azurerm_key_vault" "vault" { name = "company-vault"; resource_group_name = "platform-rg" } — look up Key Vault to get its ID. Then reference: azurerm_key_vault_secret.app_secret { key_vault_id = data.azurerm_key_vault.vault.id }. Data sources fetch at plan time — they show the current value in the plan output. If the referenced resource does not exist, the plan fails. Common use: look up the existing VNet created by the network team, then create subnets in it using this Terraform code.

Step 1: check if webhook is reaching Jenkins. GitHub → repo Settings → Webhooks → click the webhook → Recent Deliveries. Green tick = delivered. Red X = failed (check the error — usually SSL cert issue or Jenkins URL unreachable). Step 2: check Jenkins receive the webhook. Manage Jenkins → System Log → look for webhook-related log entries. Jenkins Bitbucket Plugin or GitHub Plugin logs the trigger. Step 3: check GitHub plugin configuration. Manage Jenkins → Configure System → GitHub section → GitHub Servers → test connection. Verify the GitHub API URL and credentials are correct. Step 4: check the job trigger configuration. The Multibranch pipeline or job must have "GitHub hook trigger for GITScm polling" checked. Or use the generic webhook trigger plugin. Step 5: check if Jenkins is behind a firewall. GitHub webhooks come from GitHub IP ranges — ensure your firewall allows inbound from GitHub IPs on port 443 or 8080. GitHub publishes its IP ranges: curl https://api.github.com/meta. Step 6: fallback — use polling. Add H/5 * * * * to "Poll SCM" — Jenkins polls every 5 minutes as fallback. Less efficient but more reliable than webhooks through complex firewalls.

Trivy is the most commonly used scanner (open source, fast, accurate). Integration in Jenkins/Azure DevOps: after docker build, before docker push: trivy image --severity CRITICAL,HIGH --exit-code 1 myapp:${BUILD_NUMBER}. --exit-code 1 makes the pipeline fail if CRITICAL or HIGH CVEs are found. Output formats: table (human readable), json (for parsing), sarif (for GitHub Security tab). Trivy scans: OS packages (apt, yum), language packages (pip, npm, maven, gem), Dockerfile misconfigurations, secrets accidentally embedded in images. In Azure DevOps: task type: Docker, command: run, image: aquasec/trivy:latest, options: --rm, containerCommand: image --exit-code 1 --severity CRITICAL,HIGH myregistry.azurecr.io/myapp:$(Build.BuildId). ACR Defender for Containers: scans images pushed to ACR automatically. Alerts on new CVEs even for images already in registry. Baseline: do not fail on MEDIUM — too many false positives. CRITICAL always fail. HIGH fail unless explicitly suppressed with documented justification. Scan the base image too: FROM python:3.11-slim inherits all python:3.11-slim vulnerabilities. Rebuild when base image has new security patches.

Ad-hoc commands: run a single Ansible module once without writing a file. ansible all -m ping, ansible webservers -m shell -a "free -m", ansible databases -m service -a "name=mysql state=restarted". Use for: quick one-off tasks, checking system state, emergency actions, exploring what Ansible can do. Playbooks: YAML files defining multiple tasks in sequence. Reusable, version-controlled, idempotent. Use for: everything that happens more than once, automation that runs on schedule, configuration management, deployment automation. The rule: if you find yourself running the same ad-hoc command more than twice, make it a playbook task. Ad-hoc commands are not idempotent by default — running ansible all -m shell -a "useradd newuser" twice creates the user twice and errors. A playbook task with user: name=newuser state=present is idempotent — second run is a no-op. Debugging: ansible all -m setup collects facts (OS, memory, disk, IP) as ad-hoc. ansible all -m command -a "cat /etc/os-release" checks OS versions. These are naturally ad-hoc because you run them occasionally for diagnostics.

OpenTelemetry is the CNCF standard for distributed tracing, metrics, and logs. It provides vendor-neutral instrumentation. Architecture: application code instrumented with OTel SDK (or auto-instrumented via agent) → sends telemetry to OTel Collector → Collector processes and exports to backend (Jaeger, Zipkin, Azure Monitor, Datadog). Python auto-instrumentation: pip install opentelemetry-instrument-flask opentelemetry-exporter-otlp. OTEL_SERVICE_NAME=payment-api OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 opentelemetry-instrument python app.py. This automatically instruments Flask HTTP requests, SQLAlchemy DB calls, Redis calls — no code changes needed. OTel Collector deployment in Kubernetes: DaemonSet or Deployment with OTel Collector. Receives traces from pods, batches, exports to Jaeger. Collector config: receivers: otlp (grpc port 4317), processors: batch, exporters: jaeger + prometheus. Context propagation: each service passes trace-id and span-id in HTTP headers (W3C TraceContext). Downstream services create child spans under the same trace. You see the entire request journey across all services in one Jaeger trace view. At HPE: added OTel to 5 Java microservices using the Java auto-instrumentation agent (one JVM flag, no code changes). Reduced P99 latency debugging from hours to minutes.

ELK = Elasticsearch (storage and search) + Logstash or Fluentd (log processing) + Kibana (visualisation). For Kubernetes: use Fluent Bit (lightweight) or Fluentd as the log collector DaemonSet instead of Logstash. Architecture: Fluent Bit DaemonSet on every node → reads /var/log/containers/*.log → adds K8s metadata (pod name, namespace, labels) → sends to Elasticsearch. Fluent Bit config: [INPUT] Name tail, Path /var/log/containers/*.log. [FILTER] Name kubernetes, Match kube.*, Kube_URL https://kubernetes.default.svc:443. [OUTPUT] Name es, Match *, Host elasticsearch, Index k8s-logs. Elasticsearch: deploy via ECK (Elastic Cloud on Kubernetes) operator or Helm. Index templates define mappings. ILM (Index Lifecycle Management) policies: hot (recent logs, fast SSDs) → warm (week old, standard disk) → cold (month old, cold storage) → delete (after 90 days). Kibana: create index pattern k8s-logs-*. Discover: search and filter logs. Dashboard: visualise error rates, namespace activity. Alerts: Kibana Watcher or Elastic alerting rules. For high volume: use Logstash between Fluent Bit and Elasticsearch for complex parsing/enrichment.

IAM least privilege: grant only the permissions actually needed, nothing more. Principles: start with no permissions. Add only what the role/user needs. Review and revoke regularly. Tools: IAM Access Analyzer: analyzes policies and flags overly permissive rules. Shows which permissions are used vs unused over last 90 days. AWS recommends removing unused permissions. IAM Policy Simulator: test what actions a user/role can actually perform before granting access. Condition keys: restrict by time, IP, VPC endpoint, MFA status. Example restrictive S3 policy: Allow s3:GetObject on arn:aws:s3:::mybucket/* with condition aws:SourceVpc=vpc-xxx (only from VPC). Service Control Policies (SCPs): apply to entire AWS accounts. Prevent users from disabling CloudTrail, creating public S3 buckets, disabling GuardDuty. Applies even to account root user. Permission boundaries: max permissions an entity can have. Even if inline policy grants more, boundary limits actual access. For EKS pods: use IRSA (IAM Roles for Service Accounts). Each pod gets its own role with only what it needs. payment-pod can write to payment-bucket only. order-pod cannot access payment-bucket at all. Audit: use AWS Access Advisor (last accessed information) and CloudTrail to see what is actually used.

Active-passive: primary region handles all traffic. Secondary region is warm standby, ready to take over. RTO: 15 minutes. RPO: 5 minutes. Components: Route 53: primary record points to ALB in us-east-1 with health check. If health check fails for 3 consecutive times: automatic failover to ALB in eu-west-1. EKS: active cluster in us-east-1. Passive cluster in eu-west-1, running minimal workloads. ArgoCD: one ArgoCD managing both clusters. Same Git manifests deployed to both. RDS: Multi-AZ in primary region for HA. Cross-region read replica in secondary. RTO for DB failover: promote read replica to primary (5-10 minutes). Scripts this automatically: Lambda triggered by Route 53 health check alarm. S3: cross-region replication enabled on all buckets. Objects sync in seconds. EFS/Azure Files: AWS DataSync for periodic sync. Terraform: identical infrastructure code for both regions with different variable files. Test quarterly: simulate us-east-1 failure by pointing Route 53 health check to a failing endpoint. Verify traffic routes to eu-west-1. Verify database promotion completes. Document every step with expected timing. Refine until RTO target is consistently achieved.

git fetch: downloads changes from remote WITHOUT modifying your working directory or current branch. Updates remote tracking branches (origin/main). Safe operation — never modifies local work. Use to see what changed remotely: git fetch origin, git log HEAD..origin/main shows commits on remote you do not have yet. git pull: git fetch followed by git merge (or git rebase with --rebase flag). Downloads AND integrates remote changes into your current branch. If your branch has local commits, creates a merge commit. With --rebase: replays your local commits on top of remote changes — cleaner linear history but rewrites local commits. git merge: integrates changes from one branch into another. Creates a new merge commit preserving both branches history. Use in Jenkinsfiles and CI: git pull --rebase origin main. Use for long-lived feature branches: git merge main (update feature branch with main changes). Best practice for team workflow: git fetch origin, review what changed (git log HEAD..origin/main), then git pull --rebase origin main. The --rebase keeps history clean without merge commits for routine syncs. Never use git pull on shared branches (main, release) — always prefer explicit fetch + merge with PR review.

Merge conflicts happen when two branches modify the same line of the same file. How to resolve: Step 1: identify conflicts after git merge or git rebase. git status shows "both modified" files. Step 2: open each conflicted file. Conflict markers: <<<<<<< HEAD (your changes), ======= (separator), >>>>>>> feature-branch (incoming changes). Edit the file to keep the correct version — could be yours, theirs, or a combination. Remove the conflict markers entirely. Step 3: after editing each file: git add filename. Step 4: git merge --continue (or git rebase --continue). Tools: git mergetool opens a visual diff tool (vimdiff, VS Code, IntelliJ). VS Code diff editor shows conflicts clearly. Prevention: merge main into your feature branch frequently (git merge main). Small, focused PRs — less code changed means fewer conflicts. Communication: tell team members when you are working in a file that others are also changing. Team rule: never work in the same file at the same time for large changes. If conflicts happen repeatedly in the same file: consider splitting the file into smaller modules or having clearer code ownership.

GitFlow: multiple long-lived branches (main, develop, feature/xxx, release/xxx, hotfix/xxx). PRs go to develop. Release branch created from develop. After testing, merged to main and tagged. Hotfixes branch from main. Complex, lots of merges, good for scheduled release cycles. Problems: long-lived branches diverge significantly. Integration issues discovered late. Trunk-based development (TBD): one main branch (trunk). Everyone commits to main (or short-lived feature branches of 1-2 days). Continuous integration — code goes to production daily. Feature flags hide incomplete features in production. Why TBD is better for DevOps: forces small, focused commits. Catches integration issues immediately (CI runs on every commit). Aligns with continuous deployment — if code is always in a releasable state, you can deploy anytime. At HPE we used TBD: feature branches never lasted more than 2 days. Short-lived branches, small PRs, fast reviews. Paired with ArgoCD: every merge to main triggered deployment to dev. Allowed us to deploy to production 5x per day safely. GitFlow is still valid for: embedded software, regulated releases, large teams with long QA cycles. For cloud-native microservices: TBD is the right choice.

OpenShift Route exposes a service externally through the built-in HAProxy router. Three TLS termination types: Edge termination: TLS terminated at the router. Traffic from router to pod is HTTP (unencrypted). Certificate stored in the Route object. Most common for web apps. Passthrough: TLS passes through to the pod. Pod handles TLS itself. Used when the application must own the TLS handshake (mutual TLS, client certs). Re-encrypt: TLS terminated at router, then re-encrypted to pod with a different certificate. Used when both external and internal encryption are required. Create edge route: oc expose svc/myapp --hostname=myapp.company.com --tls-termination=edge --cert=tls.crt --key=tls.key --ca-cert=ca.crt. Or via YAML: spec.tls.termination: edge, spec.tls.certificate: -----BEGIN CERTIFICATE-----, spec.tls.key: -----BEGIN PRIVATE KEY-----. Automated certificates: OpenShift cert-manager integration or custom cert-manager via OperatorHub. Annotate Route: cert-manager.io/issuer: letsencrypt → cert-manager requests and manages the certificate automatically. Wildcard certs: one certificate for *.apps.cluster.company.com covers all routes.

OpenShift image pull failures: more complex than vanilla K8s because of SCC and built-in registry. Step 1: kubectl describe pod mypod — Events section shows: Failed to pull image "myimage": rpc error: code = Unknown desc = ... Common errors and causes: "unauthorized" — authentication failure. Check if the pull secret is correct and attached to the service account. oc get sa default -n mynamespace -o yaml shows imagePullSecrets. "not found" — image tag wrong or registry unreachable. Check image name exactly: registry/org/repo:tag. "509: certificate signed by unknown authority" — registry uses self-signed cert. Add registry CA to OpenShift cluster image config or use --insecure-registry. For ACR: create pull secret: oc create secret docker-registry acr-pull-secret --docker-server=myacr.azurecr.io --docker-username=$SP_APP_ID --docker-password=$SP_PASSWORD. Attach to service account: oc secrets link default acr-pull-secret --for=pull. For internal OpenShift registry: ensure the BuildConfig pushed the image correctly. oc get is (ImageStream) shows available images. For permission issues: if pod is trying to pull from internal registry, the SA needs system:image-puller role in the namespace where the image lives.

import ssl, socket, datetime def check_cert_expiry(hostname, port=443, warn_days=30): """Check SSL certificate expiry for a hostname""" context = ssl.create_default_context() try: with socket.create_connection((hostname, port), timeout=10) as sock: with context.wrap_socket(sock, server_hostname=hostname) as ssock: cert = ssock.getpeercert() # Parse expiry date expiry_str = cert["notAfter"] # "Nov 15 12:00:00 2024 GMT" expiry = datetime.datetime.strptime(expiry_str, "%b %d %H:%M:%S %Y %Z") days_left = (expiry - datetime.datetime.utcnow()).days status = "CRITICAL" if days_left < 7 else "WARNING" if days_left < warn_days else "OK" return {"hostname": hostname, "expiry": expiry.isoformat(), "days_left": days_left, "status": status} except ssl.SSLError as e: return {"hostname": hostname, "error": str(e), "status": "ERROR"} except Exception as e: return {"hostname": hostname, "error": str(e), "status": "UNREACHABLE"} # Check multiple hosts hosts = ["app.company.com", "api.company.com", "payment.company.com"] for host in hosts: result = check_cert_expiry(host) print(f"{result['status']:10} {host}: {result.get('days_left','N/A')} days remaining") if result["status"] in ["CRITICAL","WARNING"]: # Send to PagerDuty or Slack print(f" ALERT: Certificate expiring soon on {host}")

#!/bin/bash set -euo pipefail DEPLOYMENT="payment-api" NAMESPACE="production" SLACK_WEBHOOK="${SLACK_WEBHOOK_URL}" THRESHOLD=2 # alert if less than 2 pods ready check_deployment() { # Get ready/total replicas READY=$(kubectl get deployment "$DEPLOYMENT" -n "$NAMESPACE" \ -o jsonpath="{.status.readyReplicas}" 2>/dev/null || echo "0") DESIRED=$(kubectl get deployment "$DEPLOYMENT" -n "$NAMESPACE" \ -o jsonpath="{.spec.replicas}" 2>/dev/null || echo "0") echo "Ready: ${READY:-0}/${DESIRED:-0}" if [[ "${READY:-0}" -lt "$THRESHOLD" ]]; then PODS=$(kubectl get pods -n "$NAMESPACE" -l "app=$DEPLOYMENT" \ --no-headers | awk "{print \$1, \$3}") MESSAGE="ALERT: $DEPLOYMENT in $NAMESPACE has only ${READY:-0}/${DESIRED:-0} ready pods\n\nPod status:\n$PODS" curl -s -X POST "$SLACK_WEBHOOK" \ -H "Content-Type: application/json" \ -d "{"text": "$MESSAGE"}" echo "Alert sent to Slack" return 1 fi return 0 } # Run every 60 seconds while true; do check_deployment || true sleep 60 done

Corrupted Terraform state is a critical incident — it can lead to Terraform trying to recreate or delete real infrastructure. Immediate response: set the remote state to locked if possible to prevent anyone else from running terraform apply. Assess damage: terraform state list — if this fails, state is indeed corrupted. Look at the raw state file in Azure Blob/S3. Try to parse as JSON — if it is valid JSON but with wrong content, you may be able to manually fix it. If state file is empty or completely broken: Enable versioning on your Azure Blob/S3 bucket (you did do this, right?). Restore previous version: az storage blob restore --account-name tfstate --container-name state --blob-name prod.tfstate. After restore: run terraform plan — compare what Terraform thinks exists vs actual Azure resources. For any resources Terraform lost track of: terraform import to re-add them to state without recreating. For resources Terraform thinks exist but do not: terraform state rm resource.type.name removes from state without destroying. Testing the recovery: run terraform plan, verify no unexpected creates/destroys, get tech lead review before any apply. Prevention: enable blob versioning and soft delete on state storage. Weekly automated backup of state files. Use Terraform Cloud or remote backend with built-in versioning.

Cluster Autoscaler (CA) automatically adjusts the number of nodes in a node pool. When pods cannot be scheduled (Pending due to insufficient CPU/memory): CA adds nodes. When nodes are underutilised for 10+ minutes: CA removes nodes and reschedules pods. Enable in AKS: az aks nodepool update --name userpool --cluster-name myAKS --resource-group myRG --enable-cluster-autoscaler --min-count 2 --max-count 20. CA checks every 10 seconds for unschedulable pods. Adds nodes in the most suitable pool (based on pod resource requests and node selectors). Scale-down: CA marks nodes for removal if utilisation below 50% for 10 minutes. Drains pods, deletes node from Azure. Caveats: pods with local storage (emptyDir, hostPath) block scale-down. Pods with strict PodDisruptionBudgets block drain. System pods (DaemonSets) are excluded from scale-down. Tune with annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "false" on pods that must not be evicted. For faster scale-up: over-provisioning pattern — keep 2 low-priority pause pods (Deployment with priority class below all real workloads). When real pods need nodes, CA evicts the pause pods immediately creating space. CA pre-provisions nodes for pause pods = nodes ready for real workload spikes.

KEDA (Kubernetes Event-Driven Autoscaling) extends HPA to scale on external events — Kafka lag, Azure Service Bus queue depth, Prometheus metrics, HTTP request rate. Standard HPA scales only on CPU/memory. KEDA fills the gap. Install: helm install keda kedacore/keda -n keda --create-namespace. ScaledObject for Azure Service Bus: apiVersion: keda.sh/v1alpha1, kind: ScaledObject, spec.scaleTargetRef.name: order-processor, spec.minReplicaCount: 0 (scale to zero when queue empty!), spec.maxReplicaCount: 50, spec.triggers: type: azure-servicebus, metadata.queueName: orders, metadata.messageCount: "10" (1 pod per 10 messages). Authentication: TriggerAuthentication using Workload Identity — no credentials stored. Kafka scaler: type: kafka, metadata.bootstrapServers: kafka:9092, metadata.consumerGroup: my-group, metadata.topic: orders, metadata.lagThreshold: "100" (1 pod per 100 messages of lag). Scale to zero: KEDA can scale to 0 replicas when no events — saves compute cost completely. HPA cannot go below 1. HTTP-based scaler: scale on requests per second to an endpoint. Use case: payment processing pods scale from 0 to 50 based on Azure Service Bus queue depth. Zero cost when no payments. Near-instant scale up when orders come in.