The external interfaces to Kubernetes ingress

18 May 2021 Tags: Distributed systems

Defining a Kubernetes ingress seems simple enough: Define a YAML file and you’re good. But that only defines the interface from services inside the cluster to whatever external agent is connecting external client requests to cluster services. It says nothing about the interface from that external agent to the Kubernetes control and data planes. If you are building a cloud provider, how do you connect your external load balancer to Kubernetes clusters inside your cloud? If you are configuring an entire on-prem cluster from scratch, how do you set up access to the Internet?

Over time, the Kubernetes architects have proposed several different approaches to connecting an external load balancer to the cluster. I could never find documentation of the external interfaces for these approaches. While writing my post about the three rings of Kubernetes abstractions, I saw how this connection can be made using well-known features of the Kubernetes API. This helped me understand how Kubernetes ingress actually works and also explained why setting up an Ingress resource requires installing three distinct pieces. This post records this understanding in the hope that readers might benefit.

A nomenclature note: All approaches to incoming external Kubernetes connections are denoted by the generic term “ingress”. The specific approach that uses an Ingress resource is denoted by the proper noun (capitalized), “Ingress”.

Ingress via LoadBalancer service

The first approach to defining Kubernetes ingress was to specify a type LoadBalancer Service, where an external load balancer routes traffic to the pods at the Service Endpoints. The details of the interface between the external load balancer and Kubernetes do not appear to be documented. The general outline seems clear, however:

1. The load balancer uses the Kubernetes API to set a watch on Services of type LoadBalancer.
2. When the load balancer receives notice that such a service has been created, it requests a list of all Endpoints for that service.
3. The load balancer routes incoming traffic across the Endpoints.
4. The load balancer continues monitoring Services and Endpoints, modifying its internal tables as entries are added and deleted.

This design provided a vendor-independent way to specify that an external load balancer existed but configuring that load balancer remained vendor-specific.

Ingress via Ingress resource and controller

The second approach was the Ingress resource and associated Ingress controller. This approach, which graduated to general availability with Kubernetes v1.19 in 2020, provides a richer language for specifying the functions of an external load balancer in a vendor-independent way. The cluster administrator defines rules for routing HTTP/HTTPS requests via an Ingress resource. A vendor-specific Ingress Controller reads this resource, converting its requests into whatever format the vendor’s actual ingress agent requires. For example, the Ingress Controller for Nginx writes an Nginx configuration file and restarts the Nginx load balancer. The actual load balancer then routes traffic to the Endpoints.

The Ingress resource allows vendors to define annotations specific to their load balancer, controlling vendor-specific features. For example, to specify the Amazon Resource Name (ARN) for an SSL certificate to be used by an AWS load balancer, add a service.beta.kubernetes.io/aws-load-balancer-ssl-cert annotation to the Ingress resource.

Ingress via service mesh

In clusters with a service mesh, the mesh often include ingress agents whose role is to bring requests into the mesh. These agents typically interface with the cluster using one of the two above approaches. For example, the default Istio configuration profile defines a LoadBalanced Service implemented by one or more Envoy proxies.

In terms of their external interface, service meshes are no different from other ingress agents. Within a mesh however, the routing of external traffic from the agent to the Endpoints is not necessarily the mechanisms described above for clusters without a mesh, but may instead be routed using the mesh protocol.

Summary

Ingress, the external component that connects external clients to a Kubernetes cluster’s services, is complex and evolving. The Kubernetes design team have developed two distinct approaches to configuring such a service and are apparently considering more. Service mesh designers have built on these to provide approaches matched to the purpose of each mesh.

Understanding ingress is useful for developers to understand the full route a client request takes from an external address to a service inside the cluster.