Kubernetes in hybrid cloud environments is the convergence of two transformative technologies: the most widely adopted container orchestration platform and the infrastructure model that lets enterprises balance control, compliance, and cloud economics. Together, they provide a consistent application platform that spans on-premises data centres and public cloud regions — enabling workload portability, elastic scaling, and unified operations without rewriting a single line of application code.
This guide covers everything you need to architect, deploy, and operate Kubernetes across hybrid environments — from cluster topology to networking, security, storage, and GitOps delivery.
Why Kubernetes for Hybrid Cloud?
Kubernetes abstracts the underlying infrastructure through a consistent API. An application deployed to a Kubernetes cluster running on-premises on bare metal and an application deployed to a managed cluster in Amazon EKS use identical manifests. This API consistency is what makes Kubernetes the natural portability layer for hybrid cloud:
- Move workloads between environments without code changes
- Use the same CI/CD pipeline to deploy anywhere
- Apply the same security policies and RBAC rules across all environments
- Operate one observability stack for all clusters
Hybrid Kubernetes Cluster Architectures
Stretched Cluster (Single Logical Cluster)
A single Kubernetes cluster with nodes spanning on-premises and cloud. The control plane typically runs on-premises or in a dedicated cloud account. Workloads are scheduled across all nodes using node labels and affinity rules. This model provides the simplest operational experience but requires ultra-low latency between sites (under 5 ms) — making it suitable only for co-location scenarios where on-premises and cloud are connected via dark fibre.
Federated Clusters (Multiple Independent Clusters)
Each environment has its own independent Kubernetes cluster. A federation layer — KubeFed, Admiralty, or Liqo — distributes workloads across clusters based on policy, capacity, and cost. This is the recommended architecture for most enterprises: clusters are independent failure domains, network requirements between sites are relaxed, and each cluster can be upgraded independently.
Managed Control Plane + Self-Managed Workers
Use a managed Kubernetes service (EKS, AKS, GKE) for the control plane and run worker nodes on-premises via the provider’s hybrid extensions: AWS Outposts, Azure Arc-enabled Kubernetes, or Google Anthos. This reduces control plane operational burden while keeping data processing on-premises.
Cluster Distributions for On-Premises
| Distribution | Best For | Key Features |
|---|---|---|
| Red Hat OpenShift | Enterprise, regulated industries | Integrated CI/CD, security policies, support contract |
| Rancher (RKE2) | Multi-cluster management | Centralised management UI, lightweight, FIPS-compliant |
| VMware Tanzu | VMware environments | Native vSphere integration, NSX-T networking |
| k3s | Edge, resource-constrained nodes | Single binary, ARM support, minimal footprint |
| Upstream kubeadm | Full control, custom builds | Maximum flexibility, maximum operational responsibility |
Hybrid Networking for Kubernetes
Networking is the most complex aspect of hybrid Kubernetes. Pods in on-premises clusters need to communicate with pods in cloud clusters — and with cloud services — reliably and securely.
CNI Selection
The Container Network Interface (CNI) plugin defines how pods communicate within and across clusters. For hybrid environments, Cilium is the preferred choice: it provides eBPF-based networking, native cluster mesh connectivity between multiple clusters, and consistent network policies that translate to both on-premises firewall rules and cloud security groups.
Cross-Cluster Service Discovery
Services in one cluster must be discoverable by pods in another. Options include:
- Cilium Cluster Mesh: Extends Kubernetes DNS across clusters — pods use standard DNS names to reach services in remote clusters
- Istio multi-cluster: Service mesh that provides cross-cluster load balancing, traffic management, and mTLS
- Submariner: Open-source cross-cluster connectivity tool with service discovery and L3 networking
Workload Placement and Scheduling
Not all workloads belong in the cloud. Define placement policies based on:
- Data gravity: Workloads that process large on-premises datasets should run on-premises to avoid expensive data egress
- Compliance: Workloads handling regulated data must run in environments with appropriate controls
- Latency: Workloads requiring sub-millisecond response to on-premises systems should run on-premises
- Cost: Workloads with variable demand benefit from cloud elasticity; steady-state workloads are cheaper on owned hardware
Implement placement using node labels, node selectors, affinity/anti-affinity rules, and taints and tolerations. For federation, use placement policies in your federation controller to express the same logic at a higher level.
Storage in Hybrid Kubernetes
Storage is often the binding constraint on workload portability. Options for hybrid environments:
- Rook/Ceph: Software-defined storage that provides block, object, and file storage on bare-metal Kubernetes nodes. Can be stretched across sites with synchronous replication
- Portworx: Commercial Kubernetes-native storage with enterprise DR, encryption, and auto-piloting
- NetApp Astra / Trident: Integration with NetApp on-premises arrays and cloud storage services under a single CSI driver
- Longhorn: Lightweight distributed block storage for Kubernetes, excellent for edge and hybrid scenarios
Security for Hybrid Kubernetes
- mTLS everywhere: Use a service mesh (Istio or Linkerd) to enforce mutual TLS between all services, on-premises and cloud
- Pod Security Admission: Enforce baseline or restricted pod security standards across all clusters using consistent PodSecurityAdmission policies
- Image signing and scanning: Sign container images with Cosign. Scan with Trivy in CI. Enforce admission policies that reject unsigned or vulnerable images
- RBAC consistency: Define RBAC roles in a central repository and apply them to all clusters via GitOps. Never grant cluster-admin except to break-glass accounts
- Secrets management: Use Vault Agent Injector or External Secrets Operator to inject secrets at runtime — never store secrets in Kubernetes Secrets objects in Git
GitOps Delivery for Hybrid Kubernetes
ArgoCD is the preferred GitOps tool for multi-cluster hybrid environments. A central ArgoCD instance (or a hub-spoke federation of ArgoCD instances) manages all clusters — on-premises and cloud. Applications are defined in Git as ArgoCD Application or ApplicationSet resources, and ArgoCD continuously reconciles each cluster’s state with the desired state in Git.
Key ArgoCD patterns for hybrid:
- ApplicationSet with ClusterGenerator: Automatically create one ArgoCD Application per cluster based on cluster labels — ideal for deploying the same application across all environments
- Sync waves: Control the order in which resources are applied across clusters (e.g., deploy to on-premises first, then cloud after smoke tests pass)
- Health checks: Define custom health checks for your application’s specific readiness signals
Frequently Asked Questions
Do I need a separate Kubernetes cluster for on-premises and cloud?
For most organisations, yes — separate clusters per environment are recommended. They provide independent failure domains, relaxed network latency requirements between sites, and independent upgrade schedules. A federation layer (KubeFed, Admiralty, or a GitOps tool like ArgoCD ApplicationSets) then provides unified management across clusters without coupling their lifecycles.
What is the difference between OpenShift and upstream Kubernetes?
Red Hat OpenShift is an enterprise Kubernetes distribution that adds an integrated CI/CD platform (Tekton Pipelines), a developer portal (OpenShift Console), enhanced security policies (SecurityContextConstraints), a built-in container registry, and a support contract. Upstream Kubernetes provides maximum flexibility but requires you to assemble and maintain these components yourself. For enterprises that need vendor support and an integrated developer experience, OpenShift is worth the additional cost.
How do you handle persistent storage for stateful applications in hybrid Kubernetes?
Persistent storage for stateful applications in hybrid Kubernetes requires a storage solution that spans environments. Options include Rook/Ceph for software-defined storage on bare-metal, Portworx for enterprise-grade requirements, or cloud storage accessed via cross-environment networking. For workloads with strict data residency requirements, keep both the application and its storage in the same environment rather than splitting them across the hybrid boundary.
What is Cilium Cluster Mesh?
Cilium Cluster Mesh is a feature of the Cilium CNI that extends Kubernetes service discovery and connectivity across multiple clusters. Pods in one cluster can reach services in a remote cluster using standard Kubernetes DNS names, without any application changes. Cluster Mesh also enforces consistent NetworkPolicies across all connected clusters and provides cross-cluster load balancing with health-aware failover.
How do you manage Kubernetes upgrades across a hybrid fleet?
Upgrade clusters one at a time, starting with the lowest-criticality environment. Use a GitOps-driven approach: update the cluster version declaration in Git, let your GitOps operator apply the change, validate application health, then proceed to the next cluster. Managed cloud Kubernetes services (EKS, AKS) handle control plane upgrades for you; on-premises clusters require more manual coordination. Never let cluster version skew exceed two minor versions across your fleet.
What is the cost of running Kubernetes on-premises vs managed cloud Kubernetes?
On-premises Kubernetes has high upfront capital cost (servers, networking, storage) but low per-workload run cost for steady-state utilisation. Managed cloud Kubernetes (EKS, AKS, GKE) has zero control plane management overhead but charges for node instances, load balancers, storage, and data transfer. For workloads with consistent, predictable resource consumption above 60 % utilisation, on-premises Kubernetes is typically more economical. For variable or spiky workloads, cloud-managed Kubernetes wins on total cost.
Conclusion
Kubernetes is the lingua franca of hybrid cloud infrastructure. By standardising on a single orchestration platform across on-premises and cloud environments, organisations gain genuine workload portability, consistent operations, and the ability to optimise placement decisions over time as requirements evolve.
OpsNexus designs and operates hybrid Kubernetes environments for enterprises, from initial architecture through to day-2 operations and platform maturity. Talk to our team to explore how Kubernetes can unify your hybrid cloud infrastructure.
