Senior Kubernetes Platform - Training Guide

Plan & the whole picture

Target role: Senior Kubernetes Platform Engineer at Avance Services, Santa Clara, California. The JD asks for 7+ years of deep Kubernetes administration, GPU workload support, cluster troubleshooting, and day-2 operations. Your background is strong on AI/ML Kubernetes platform work (Mozn: GKE/EKS + KubeFlow + Databricks) but lighter on day-2 cluster operations at scale. This course fills the gap.

This-week study plan

Six modules, 3-5 hours per day. Mark complete in the checklist below. Stick to the order; each module builds on the last.

Your 30-second interview story

Study checklist

Tick items as you finish. Progress is shown in the header bar.

Where your gaps are

Honest baseline against the JD:

Mind map: the whole course

1. Kubernetes internals

Every troubleshooting story starts here. If you cannot draw the control plane, you cannot debug a cluster. The 7+ years requirement in the JD is asking for fluency with the internals, not just operational use.

1.1 Control plane deep dive

The control plane runs the API and decides what should exist. The nodes run what the API decided. In production you have 3 or 5 control-plane nodes for etcd quorum (Raft needs N/2+1, so 3 survives 1 failure, 5 survives 2).

Components, in order of how often you touch them in a debug:

• kube-apiserver — the only component that talks to etcd. Every other component watches or writes through the API. Authn (who are you), authz (RBAC), admission (validating/mutating webhooks), then write.
• etcd — Raft consensus KV store. Back it up (etcdctl snapshot save) before any control-plane upgrade. Split-brain is fatal.
• kube-scheduler — filter + score. Filter eliminates nodes that cannot run the pod (resources, taints, affinity). Score ranks the survivors (least-allocated, balanced, requested affinity). The binding is then written to the API.
• kube-controller-manager — a pile of reconcile loops (Deployment, ReplicaSet, Node, Job, ServiceAccount, ...). The controllers you touch most are Deployment and Node.
• cloud-controller-manager — cloud-specific glue (load balancers, routes, node lifecycle). On bare-metal you do not run this.

1.2 etcd + Raft

etcd is the source of truth. Everything in the cluster is reconstructable from etcd. The things to know cold:

• Quorum: Raft needs majority. 3 nodes = survive 1 loss. 5 nodes = survive 2. Always odd.
• Backup: etcdctl snapshot save /backup/etcd-snap.db. Restore creates a data dir you point the new etcd at.
• Defrag: etcd stores in B-tree; long-running clusters slow down. etcdctl defrag on each member, one at a time.
• WAL fsync: monitor this metric. Long fsync means slow disk; the apiserver waits on it.
• DB size: keep below 8 GB. Watch etcd_debugging_mvcc_db_total_size_in_bytes.

1.3 API server + auth flow

When a pod is created, the request goes: kubectl apply -> HTTP POST to apiserver -> TLS terminates -> authentication (cert, token, OIDC) -> authorization (RBAC: is this subject allowed to do this verb on this resource?) -> admission (mutating webhooks first, validating webhooks after) -> etcd write -> watch fires to scheduler + controllers + kubelet.

If you can describe that flow out loud, you can debug half the cluster's permission problems. The other half is webhook misconfiguration.

1.4 Scheduler + placement

Two phases: filter then score.

• Filter: resource fit (CPU, memory, extended resources like GPU), NodeName, NodeSelector, taints/tolerations, NodeAffinity, PodAffinity/AntiAffinity, persistent volume locality.
• Score: LeastAllocated (spread out), MostAllocated (pack tight for cost), BalancedResource, NodeAffinity preferred, TaintToleration preferred, image-locality (already on the node).

GPU scheduling: device plugin registers nvidia.com/gpu=N as an extended resource. Pod requests resources.limits.nvidia.com/gpu: 1. Filter checks that the candidate node has at least 1 available. Score then ranks.

1.5 Kubelet + container runtime

The kubelet is the agent on every node. It watches the apiserver for pods bound to itself, then asks the container runtime (CRI) to start them. It reports node + pod status back via heartbeat (default every 10s).

Common kubelet issues:

• Disk pressure (image, logs, emptyDir fill the node) -> kubelet evicts pods.
• PIDs pressure (too many processes) -> new pods rejected.
• Memory pressure -> pods evicted (QoS class matters: Guaranteed > Burstable > BestEffort).
• Kubelet crash -> node NotReady; check journalctl -u kubelet.

1.6 CNI / CSI / CRI in one mental model

These three interfaces are what makes Kubernetes pluggable. Each one is a separate specification with multiple implementations.

• CRI (Container Runtime Interface) — between kubelet and the runtime. Implementations: containerd (most common now), CRI-O.
• CNI (Container Network Interface) — pod-to-pod networking. Implementations: Cilium (eBPF, modern default), Calico (iptables or eBPF), Flannel (legacy).
• CSI (Container Storage Interface) — persistent volume drivers. Implementations: AWS EBS, GCP PD, Rook-Ceph, Longhorn, NFS, many more.

If a problem is "pod cannot start" and the events mention "network not ready" or "plugin not found", it is a CNI problem. If it mentions "volume mount failed" or "PVC stuck Pending", it is CSI.

2. GPU workloads on Kubernetes

This is your interview strength. The JD asks for it explicitly. The work at Mozn (GKE + EKS + KubeFlow + Databricks) maps directly. The deepening here is in failure modes, sharing strategies, and the GPU Operator pattern that production clusters use.

2.1 The GPU device plugin

The NVIDIA device plugin runs as a DaemonSet on every GPU node. On start, it calls nvidia-smi, counts GPUs, and registers them as extended resources on the node object (e.g. nvidia.com/gpu: 8). The scheduler then treats GPU requests like any other resource.

When a pod requests nvidia.com/gpu: 1, the scheduler filters out nodes that cannot satisfy it. On the chosen node, the kubelet passes the GPU device paths into the container via the nvidia-container-runtime, which translates the request into the right --device flags for containerd/CRI-O.

2.2 Sharing: MIG vs time-slicing vs MPS

You almost always want to share GPUs, because they are expensive and not always fully utilised. Three options:

• MIG (Multi-Instance GPU) — hardware partitioning. Only A100 and H100. Each MIG instance is isolated like a separate GPU. Best for production AI serving with strict QoS.
• Time-slicing — software, the same GPU is shared across pods in slices of time. Works on T4, L4, V100, A100. No isolation; pods can see each other's memory if they misbehave. Best for dev/test, batch inference.
• MPS (Multi-Process Service) — deprecated for new work, only mention in interviews for historical context.

2.3 KubeFlow training operators

KubeFlow is more than a logo. For training jobs you use the training-operator, which adds CRDs like TFJob and PyTorchJob. These CRDs implement gang scheduling — all worker pods start together, or none start. Without gang scheduling, a 4-GPU job can hang with 1 pod running because the others cannot bind.

For pipelines you use Kubeflow Pipelines (KFP), which is a workflow engine. For AutoML you use Katib. For serving you use KServe.

2.4 Failure modes you will debug

Things that go wrong on GPU clusters, and how to recognise them:

• Driver mismatch — pod fails to start; dmesg | grep -i xid on the host shows Xid errors. Driver version incompatible with the GPU or the container toolkit.
• Device plugin crash — pods Pending with "0/N nodes are available: insufficient nvidia.com/gpu". The DaemonSet pods in nvidia-device-plugin namespace are CrashLoop or OOMKilled.
• CUDA OOM — pod runs but throws CUDA out of memory. Not a Linux OOM; the GPU has its own memory. You have to reduce batch size or pick a bigger GPU.
• NVLink / fabric failure — multi-GPU jobs fail to initialise collectives (NCCL). Check fabric health, IB subnet manager, NVLink topology.
• GPU thermal throttling — DCGM shows high temp, low clocks. Cooling or airflow problem.

3. Cluster lifecycle

The JD says "cluster lifecycle management" — that means: provision it, bootstrap it, run workloads via GitOps, upgrade it without downtime, and apply policy. This is the difference between "I used Kubernetes" and "I run Kubernetes".

3.1 Provision: kubeadm vs Cluster API

Two ways you will provision. Choose by environment.

• kubeadm — the upstream tool. Manual, explicit, teaches you what every component does. Good for learning, painful for production at scale. kubeadm init on the first control-plane node, kubeadm join on the rest.
• Cluster API (CAPI) — Kubernetes-style declarative API for managing clusters. You write a Cluster resource, a provider (AWS, GCP, Azure, vSphere, bare-metal via Metal3) creates the infrastructure. Same reconciliation pattern as pods. This is what production should use.

Managed (EKS/GKE/AKS) is the third option if the customer is on a cloud. But the JD is talking about AI infrastructure that may be on bare-metal, so kubeadm or CAPI on Metal3 is the likely answer.

3.2 Bootstrap: CNI, ingress, cert-manager

After the cluster is up, before any workload runs, you need the platform layer:

• CNI — Cilium is the modern default. Install via Helm; it brings eBPF-based networking, Hubble for observability, and network policy enforcement.
• Ingress — ingress-nginx is the battle-tested default; Traefik is the modern alternative. Both expose HTTP/HTTPS to the cluster.
• cert-manager — automatic TLS via Let's Encrypt or internal CA. ClusterIssuer + Certificate resources.
• Metrics server — required for HPA and kubectl top. Deploy via Helm.
• Storage classes — the default StorageClass that new PVCs bind to. Cloud provider or Rook-Ceph on bare-metal.

3.3 GitOps with ArgoCD

ArgoCD watches a Git repo and applies whatever is there. The cluster's actual state converges to the desired state in Git. The standard pattern is app-of-apps: a root Application that points to a directory of other Applications.

For cluster bootstrap, the GitOps order is: install ArgoCD via Helm first, then have ArgoCD install everything else. This avoids the chicken-and-egg of "GitOps before ArgoCD exists".

Alternatives: Flux CD (CNCF, lighter weight), Jenkins X. ArgoCD has the best UI; Flux has the best GitOps purity.

3.4 Upgrade procedure (cordon/drain/replace)

The discipline:

1. etcd backup first — non-negotiable.
2. One control-plane node at a time. Verify with kubectl get nodes after each.
3. Never skip a minor version. K8s 1.29 -> 1.31 needs 1.29 -> 1.30 -> 1.31, not 1.29 -> 1.31.
4. For workers: kubectl cordon <node> (no new pods), kubectl drain --ignore-daemonsets --delete-emptydir-data (evict safely), upgrade kubelet + containerd, uncordon.
5. PodDisruptionBudgets on your workloads prevent drain from removing too many at once.

3.5 Policy + admission control

The cluster's rules. Without these, anyone with namespace access can do anything.

• RBAC — Role/RoleBinding (namespace) or ClusterRole/ClusterRoleBinding (cluster). Least-privilege.
• Pod Security Standards — the built-in baseline/privileged/restricted profiles. Apply at namespace level.
• OPA Gatekeeper or Kyverno — policy engines that run as admission webhooks. Kyverno is easier to learn; Gatekeeper is more powerful for complex Rego policies.
• ResourceQuotas + LimitRanges — prevent one team from using all the CPU.

4. Observability

The JD asks for "production monitoring, alerting, logging, and observability frameworks". On Kubernetes the answer is the standard three-pillar stack: metrics + logs + traces, all flowing into Grafana, with SLOs on top.

4.1 Metrics, logs, traces

Each pillar answers a different question:

• Metrics (Prometheus) — "what is the number?" Latency, error rate, CPU, memory, queue depth, GPU utilisation. Numeric, aggregatable, alertable.
• Logs (Loki) — "what did the app say?" Structured or unstructured text per event. Searchable, not aggregatable. Use them after metrics tell you something is wrong.
• Traces (Tempo/Jaeger) — "where did the time go in this request?" Distributed context across services. Essential for microservices, useful for AI inference pipelines.

4.2 Prometheus + kube-state-metrics

Prometheus scrapes /metrics endpoints. In a cluster you get metrics from:

• kubelet (built-in /metrics) — node + pod + container resource usage.
• cAdvisor (built into kubelet) — container-level resource usage.
• kube-state-metrics (extra deployment) — Deployment, ReplicaSet, Pod, Node status as metrics. This is what gives you "pod stuck Pending" alerts.
• node-exporter (DaemonSet) — host-level: CPU, memory, disk, network.
• DCGM exporter (GPU nodes) — per-GPU metrics.

The prometheus-operator CRDs (ServiceMonitor, PodMonitor, PrometheusRule) make this declarative — you write YAML, the operator configures Prometheus.

4.3 Loki + Tempo + OpenTelemetry

Loki is the log store; labels instead of full-text indexing, so it is cheap to run. The Promtail or Alloy agent ships logs from nodes.

Tempo is the trace store; designed to be cheap, integrated with Grafana, and to use the same labels as metrics + logs (one correlation ID across all three pillars).

OpenTelemetry is the unified SDK. Instrument your app once with OTel, configure the collector, and the same trace + metric + log goes to all three backends. This is the modern default.

4.4 SLOs that matter on Kubernetes

Pick the user-facing ones. For a platform serving AI workloads, the canonical four:

1. API server availability — 99.9% of requests succeed over 30 days.
2. Pod scheduling latency — P95 of "time from pod creation to Running" under 5 seconds.
3. Node readiness — 99% of nodes Ready at any moment.
4. DNS resolution — P95 of CoreDNS query latency under 50 ms.

Set SLOs as Prometheus rules; track error budget burn rate; alert on multi-window burn (e.g. 2% budget in 1 hour = page).

5. Troubleshooting

This is the bulk of the JD. The interview will have a scenario question: "a pod is Pending, what do you do?" The answer pattern is: symptom -> events -> layer -> fix. Memorise it.

5.1 The 6 commands

These answer 80% of cluster debugging. Practice them on killercoda until they are muscle memory.

1. kubectl get pods -A — what's the cluster doing right now?
2. kubectl describe pod <name> -n <ns> — the events section tells you why.
3. kubectl logs <pod> --previous — what did the container say before it died?
4. kubectl get events --sort-by='.lastTimestamp' -A — cluster-wide events, newest last.
5. kubectl exec -it <pod> -- sh — get a shell inside a running container.
6. kubectl debug node/<name> --image=busybox -it -- chroot /host — get a shell on the node itself.

5.2 Symptom to layer

Every failure fits one of these patterns:

• Pending — scheduler cannot place. Check events: resource shortage, taint without toleration, PVC not bound, nodeName typo, nodeSelector not matching.
• CrashLoopBackOff — container keeps exiting. logs --previous first; then check image, command, env vars, secrets, probes, resource limits.
• ImagePullBackOff — registry unreachable. Check image name, imagePullSecrets, registry credentials, network to registry.
• NotReady (pod) — readiness probe failing. The pod is Running but not receiving traffic. Check probe config + what it depends on.
• NotReady (node) — kubelet heartbeat lost. Check kubelet journal, disk pressure, network to control plane, certificate expiry.

5.3 DNS failure debugging

DNS is the layer that fails silently. 80% of "service cannot connect" issues are DNS.

1. From inside a pod: nslookup kubernetes.default — if it times out, CoreDNS is the problem.
2. kubectl get pods -n kube-system -l k8s-app=kube-dns — are CoreDNS pods running?
3. Check /etc/resolv.conf inside the pod — does it point to CoreDNS service IP?
4. Check the ndots value — default 5 means short names are tried as FQDN first; this can hide real DNS bugs.
5. For cross-namespace resolution, FQDN is service-name.namespace.svc.cluster.local.

5.4 NetworkPolicy debugging

Once a NetworkPolicy exists in a namespace, the default-deny effect kicks in. Things that used to work stop working.

• kubectl get networkpolicy -A — list everything.
• Cilium Hubble: cilium hubble observe --namespace <ns> — see live flows, marked as allowed or denied.
• From a debug pod: curl -v http://service:port — the verbose output shows where it fails.
• Common gotcha: NetworkPolicy is additive; you need an explicit "allow" for every direction you want, and the podSelector must match the workload's labels.

5.5 Node debugging

When the node itself is the problem:

• kubectl describe node <name> — shows conditions, resource pressure, allocated pods.
• SSH or kubectl debug node/<name> — get on the host.
• journalctl -u kubelet -n 200 — kubelet logs.
• systemctl status containerd — runtime status.
• df -h /var/lib/containerd — image storage fill is a common cause of disk pressure.
• top / free -h — memory pressure.
• dmesg | grep -i -E 'xid|nvidia|memory|oom' — GPU + memory errors.

6. Bare-metal + storage

The "PREFERRED EXPERIENCE" section of the JD asks for bare-metal Kubernetes and high-performance storage integration. This is your biggest gap. This module closes it.

6.1 Provisioning without cloud magic

On bare-metal there is no cloud-init, no metadata service, no auto-scaling group. You provision with PXE/iPXE boot + a provisioning tool.

• Tinkerbell — CNCF-style, declarative bare-metal provisioning built on PXE + iPXE + a workflow engine. Used at Equinix Metal.
• Metal3 — Cluster API provider for bare-metal. The CAPI-native way; integrates with whatever BMC you have (IPMI, Redfish, iLO).
• MAAS — Canonical's metal-as-a-service. Older, well-tested, less Kubernetes-native.

The flow is the same in all three: define machine -> PXE boot -> write image to disk -> hand off to kubeadm or Cluster API.

6.2 Rook + Ceph for stateful workloads

Rook runs Ceph as a Kubernetes operator. You define a CephCluster CR; Rook creates MON, MGR, OSD, MDS, RGW as StatefulSets with PersistentVolumes on the raw disks.

• Block (RBD) — for databases, single-writer workloads.
• File (CephFS via MDS) — for shared-filesystem workloads.
• Object (RGW) — S3-compatible, for blob storage and model artifacts.

For AI workloads the storage class needs high IOPS; tune CRUSH map for the workload, use NVMe SSDs for OSDs, separate pools for hot data (training checkpoints) and cold (archived datasets).

6.3 High-performance filesystems

For AI training that needs many nodes reading the same dataset at once, parallel filesystems are the answer.

• Lustre — the HPC standard. Open source. Hundreds of GB/s throughput.
• IBM Spectrum Scale (GPFS) — enterprise, more polished, costs money.
• WekaFS — modern, all-NVMe, GPU-direct support. Common in AI factories.
• BeeGFS — open source, easier to operate than Lustre.

You mount these via a POSIX CSI driver; pods use them as a regular PVC. The trick is RDMA/RoCE on the network for low-latency access.

6.4 GPU tuning on bare-metal

Three knobs that matter most:

1. Huge pages — CUDA allocates large memory blocks. Configure huge pages at boot, expose via sysctl + pod resources.hugepages-*.
2. CPU isolation — isolcpus= on the kernel command line reserves CPUs for GPU work. Pair with irqaffinity to keep interrupts off the isolated CPUs.
3. NUMA pinning — GPU on PCIe slot 0 is on NUMA node 0; pin the worker thread to NUMA node 0 with numactl or the kubelet topology manager.

Plus the GPU Operator installs the driver, container toolkit, device plugin, and DCGM exporter in one Helm chart. Production should always use it instead of hand-rolled setup.

6.5 Air-gapped registries (Harbor)

On bare-metal in a regulated environment, you cannot pull from Docker Hub. You mirror upstream images into a local registry and use only those.

• Harbor — the standard. CNCF graduated. Built-in vulnerability scanning, image signing, replication from upstream registries.
• Mirror images during a one-time online phase; then operate the cluster offline.
• Sign images with cosign; enforce with Kyverno or Connaisseur admission controllers.

All free resources, one page

Everything below is free or included in your Udemy subscription. The per-module resource strips above have the same content in context; this section is the flat list for planning.

Udemy (you have a subscription)

• Certified Kubernetes Administrator (CKA) with Practice Tests — Mumshad Mannambeth. The single highest-value course. You are 4% in. Aim for 100%.
• Terraform on AWS EKS Kubernetes IaC SRE - 50 Real-World Demos — Kalyan Reddy Daida. You are 3% in. Skip to relevant sections.
• Certified Kubernetes Security Specialist (CKS) — Mumshad Mannambeth. Best troubleshooting practice.
• Certified Kubernetes Application Developer (CKAD) — Mumshad Mannambeth. Workload side (Deployment, Service, Ingress).
• Search "Prometheus monitoring" and "Grafana" on Udemy; pick the most recent 4+ star course with hands-on labs.
• Search "Kubernetes + GPU" topic; filter free + 4+ star; pick one with hands-on labs.

Official documentation (free, authoritative)

• kubernetes.io/docs — concepts, tasks, tutorials, reference.
• Components, scheduling, services + networking.
• etcd for K8s, upgrades, dashboard.
• Debug Running Pods, Debug Cluster.
• NVIDIA GPU Operator — production-grade GPU on K8s.
• KubeFlow — training operators, pipelines, Katib, KServe.
• ArgoCD, Helm, Velero, Cluster API.
• Prometheus, Grafana, OpenTelemetry, prometheus-operator.
• Rook, Ceph, Tinkerbell, Metal3, Harbor.

Free hands-on environments

• Killercoda Kubernetes scenarios — 5-15 min labs, no setup.
• Killercoda CKS scenarios — security + troubleshooting.
• Play with Kubernetes — 4-hour throwaway clusters in browser.
• minikube — local K8s on your Mac (as a VM).
• kind (K8s in Docker) — multi-node clusters in containers, fast.

Free "do it once, learn everything" projects

• Kubernetes the Hard Way — build every component by hand. Painful but teaches you where every piece lives.
• Chaos engineering scenarios — break things on purpose and practice recovery.
• NVIDIA GPU Operator quickstart — bring up a GPU node.
• KubeFlow standalone install — quickest end-to-end is kustomize-based install.
• ArgoCD examples — app-of-apps pattern in 5 minutes.
• Rook quickstart — 10 minutes to a working Ceph cluster in minikube.

Free video series (YouTube)

• TechWorld with Nana — DevOps + K8s explainers, well-paced.
• Just me and Open Source — deep K8s, GitOps, multi-cluster.
• KodeKloud — the CKA instructor's own YouTube channel.
• CNCF — conference talks; search the KubeCon archive.