Skip to main content

Load Balancing Between Multiple Kubernetes Clusters

Let's say you have three services, foo, bar, and baz, deployed in a single Kubernetes cluster, but eventually decide you need to extend to multiple clusters for reasons like:

  • Redundancy and failover
  • Load distribution and scalability
  • Hosting services close to your customers in different regions
  • Deploying some clusters on-premises and others with a cloud provider

With ngrok, you don't need to provision new load balancers or other complex networking tools—Endpoint Pooling lets you share traffic between replicas of your services on any number of clusters by deploying them on the same internal URL. With the addition of a Cloud Endpoint and Traffic Policy, you'll have a single "front door" for all your load-balanced K8s services behind a single hostname, like https://api.example.com.

tip

We have another guide for load balancing to individual K8s services with ngrok custom resources, Ingress objects, or Gateway API resources. This guide uses our custom resources to create endpoints for your services.

1. Install the ngrok Kubernetes Operator​

Check out our installation instructions for details on how to use Helm to deploy the open-source Operator to each cluster you'd like to load-balance between.

2. Create internal Agent Endpoints on your first cluster​

The YAML manifest below creates three internal agent endpoints, each of which will proxy traffic to your foo, bar, and baz pods, respectively. When you use our custom resources to create agent endpoints, pooling is enabled automatically.

Loading…

Save it to your local workstation as ingress.yaml and apply to your cluster with kubectl apply -f ingress.yaml.

These internal agent endpoints won't yet be available to public traffic, because they are only accessible through other ngrok endpoints in your account, but you'll fix that in the next two steps.

Repeat the process on any other clusters you want to load balance traffic between.

4. Create a Cloud Endpoint and add routing with Traffic Policy​

If you don't yet have a reserved domain, go ahead and reserve one in the dashboard.

The easiest way to create a cloud endpoint is in the Endpoint section of your ngrok dashboard.

Click New, then Cloud Endpoint. Leave the binding as Public and fill in the full URL of the domain you reserved, then click Create Cloud Endpoint.

The Traffic Policy rules below:

  1. Uses the forward-internal action to route all requests to the /foo path to the pool you created at https://foo.internal. - The pooled endpoints then proxy requests proxies the request to the foo Service/Deployment in your cluster.
  2. Does the same for the paths /bar and /baz.
  3. Denies all traffic to paths that don't (yet) have a service.
Loading…

Paste the following Traffic Policy rules into the dashboard and click Save.

Now, when you request https://api.example.com/foo or any other route, your traffic will be automatically load-balanced between replicas of your services on multiple Kubernetes clusters.

What's next?​

With load balancing solved for, you can now add new services or entire Kubernetes clusters by adding new endpoints to existing pools or configuring new routes with Traffic Policy.

Speaking of Traffic Policy, here are a few common ways folks filter, manage, and orchestrate requests:

Or, if you're looking for a more step-by-step experience for deploying ngrok as an API gateway, check out our guides: