In-Place Pod Restarts: Boosting Efficiency and Workload Reliability in Kubernetes v1.35 and v1.36

🔄 Update — 20 June 2026: New Cloud Features and Optimization Strategies for Kubernetes Resource Management

The ongoing enhancements in Kubernetes resilience and resource efficiency are reflected in recent cloud platform releases. New lifecycle automation features from Oracle OKE and Qovery, combined with updated best practices for workload tuning, highlight the industry’s focus on maximizing node uptime and optimizing resource utilization.

What’s new?

Oracle OKE Host Groups & Quick Recycle: Oracle Cloud Infrastructure now supports Compute Host Groups for OKE managed nodes, introducing a “Quick Recycle” capability that significantly reduces host unavailability during node maintenance and replacements.
Qovery Provisioning Automation: Qovery’s automated cluster provisioning streamlines infrastructure setups (networking, IAM, autoscaling) and simplifies the lifecycle of ephemeral environment deployments.
Dash0 Guidance on Requests vs. Limits: New observability insights from Dash0 emphasize the importance of right-sizing CPU and memory requests/limits to prevent resource throttling, OOM (Out of Memory) kills, and inflated cluster autoscaling costs.

Why this adds to the article

These updates demonstrate the real-world operational need for efficiency: while in-place Pod restarts speed up container-level recovery, OKE’s Quick Recycle minimizes node-level downtime, and Dash0’s resource right-sizing aligns directly with the goal of maximizing existing hardware utilization without over-provisioning.

Summary

Kubernetes v1.35 introduced “Restart All Containers on Container Exits” (KEP-5532) as an alpha feature, which has now graduated to beta and is enabled by default in Kubernetes v1.36. This capability allows all containers in a Pod to be restarted inline on their existing node upon failure, without deleting and recreating the Pod object. By keeping the Pod sandbox intact, it preserves critical infrastructure including network identity, storage mounts, and hardware device assignments (like GPUs or TPUs), drastically improving recovery speed and operational efficiency.

What happened?

Historically, recovering from software crashes in Kubernetes required deleting and recreating the entire Pod object to achieve a clean environment reset. Under the new RestartAllContainers action (enabled by the RestartAllContainersOnContainerExits feature gate and its dependencies ContainerRestartRules and NodeDeclaredFeatures), developers can define restart rules inside a container’s restartPolicyRules configuration. When a specific container exit code is met (e.g., a watcher sidecar exiting with code 88), the Kubelet halts all containers, preserves the existing sandbox, keeps the IP address and volumes mounted, and then re-runs init containers in the correct order to guarantee a clean setup in a known-good state.

Why it matters

For organizations running scaled platforms—particularly distributed AI/ML model training or complex batch workloads—in-place restarts offer several major advantages:

Reduced Control Plane Churn: Deleting and recreating thousands of Pods during a massive failure puts heavy pressure on the etcd backend and Kubernetes control plane. In-place restarts prevent this “Thundering Herd” effect.
Preserved Hardware Allocations: Accelerators like GPUs and TPUs remain bound to the Pod, eliminating scheduling delays and resource re-allocation overhead.
Warm Local Caches: Because the Pod remains anchored to the same node, restarted containers can instantly access warm local storage caches and model weights.
Resource Race Condition Avoidance: When resources are scarce, recreating a Pod runs the risk of another pending Pod stealing its node placement. In-place recovery keeps the resources locked.

Evidence

The capability is officially integrated into Kubernetes v1.35 and v1.36 and documented under KEP-5532. Detailed implementation findings from the Google Open Source Blog show that the CNCF project JobSet utilized this feature to reduce recovery times from minutes to seconds. Furthermore, Kubernetes introduces a new AllContainersRestarting Pod condition, which is set to True during the restart phase to alert SREs and prevent false-positive autoscaling decisions while container restart metrics increment normally.

Analysis

This feature resolves a long-standing architectural limitation of multi-container Pods. Previously, restarting individual containers could lead to stale or broken states (e.g., a main application attempting to communicate over stale connections to a database sidecar that had restarted). Re-executing init containers within the existing sandbox context ensures that the environment is fully reset. This development reflects Google’s contribution to the CNCF community, bringing internal best practices from massive distributed environments to standard Kubernetes.

Practical Takeaways

SREs and developers should adopt the following best practices:

Ensure Idempotency: Because Kubelet only guarantees “at least once” execution for init containers, all startup scripts, setup files, and initializations must be fully reentrant (idempotent).
Expect Sudden Termination: In-place restarts do not support graceful termination (preStop hooks). SIGKILL is sent almost immediately, so applications must be designed to handle sudden exits gracefully without data corruption.
Update Monitoring & CD Tooling: Observability tools and continuous delivery pipelines should be adjusted to expect re-running init containers within the same sandbox without interpreting them as new deployment rollouts.

Open Questions

How do complex service mesh sidecars behave when the network namespace remains open but application containers are abruptly restarted via SIGKILL?
Are there security implications in high-security workloads where preserving a compromised sandbox despite container crashes might pose a risk?