Exploring AKS Web Application Routing Addon
Microsoft recently released the Web Application Routing addon for AKS as a preview. Quote: The Web Application Routing add-on configures an Ingress controller in your Azure Kubernetes Service (AKS) cluster with SSL termination through certificates stored in Azure Key Vault. Optionally, it also integrates with Open Service Mesh (OSM) for end-to-end encryption of inter cluster communication using mutual TLS (mTLS). As applications are deployed, the add-on creates publicly accessible DNS names for endpoints.
There’s a lot there; unfortunately the documentation at this point in time (October 2022) only shows a few examples and does not yet do a good job of exploring more general use cases or configuration options. So I decided to do some exploration on my own and see what I could find.
Discovery #1: It can be a simple Nginx ingress controller wthout any bells and whistles
The documentation does not really make this clear, but you can use the addon even if all you want is a (simple) ingress-nginx ingress controller. Why would you want this versus, for example, simply installing the ingress-nginx controller via helm? One reason is the addon (once it moves from preview to GA) will be fully supported and managed by Microsoft. I’ve worked with some customers for whom this is a requirement. Additionally, because it is an addon, it can be provisioned via infrastructure as code templates (eg, Bicep, ARM, Terraform) all in one shot; no need to separately deploy the ingress controller via helm.
Let’s walk through an example: I’m going to start with an AKS cluster I’ve previously provisioned. First step is to install the addon:
RG="demo"
LOCATION="westus3"
AKS="demoaks"
az aks enable-addons -g $RG -n $AKS --addons web_application_routing
Next, let’s run a pair of very simple webservers that echo their hostname: (Yes, I recognize that this is imperative which is not the “pure” declarative K8S way. But it’s quick and simple for this exercise)
kubectl run server1 --image=k8s.gcr.io/e2e-test-images/agnhost:2.40 --labels="app=server1" --port=80 --command -- /agnhost serve-hostname --http --port "80"
kubectl expose pod server1 --port 80
kubectl run server2 --image=k8s.gcr.io/e2e-test-images/agnhost:2.40 --labels="app=server2" --port=80 --command -- /agnhost serve-hostname --http --port "80"
kubectl expose pod server2 --port 80
Now let’s deploy a simple ingress:
cat <<EOF |kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-sample
spec:
ingressClassName: webapprouting.kubernetes.azure.com
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: server1
port:
number: 80
- path: /server2
pathType: Prefix
backend:
service:
name: server2
port:
number: 80
EOF
You’ll notice there’s no ‘hostname’ defined in the ingress; that’s intentional; we’ll deal with that in the next section. For now, we will need to work with our ingress using an IP address, which I find by looking at the kubernetes service associated with the ingress controller:
$ kubectl get svc -n app-routing-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx LoadBalancer 10.0.248.37 20.118.145.244 80:32224/TCP,443:31564/TCP 43m
I can now test the ingress:
$ curl http://20.118.145.244
server1
$ curl http://20.118.145.244/server2
server2
Discovery #2: It can use Azure DNS labels
The previous example used an ip address, but what if you want to use a hostname? And what if you don’t have a custom domain name to use? Back a while ago, I wrote a blog post on how to use a semi-custom DNS label in conjunction with AKS load balancers. I was curious to see if I could use this technique with the Web Application addon, and was excited to find the answer is yes!
The secret is to apply an annotation to the service. (Yes, this is a little hacky.) Earlier, we saw that the addon creates a service called nginx in the app-routing-system namespace. We can set an annotation on this service to create a dns label:
apphostname="lncblogdemo" # fqdn will be <hostname>.<region>.cloudapp.azure.com per https://learn.microsoft.com/en-us/azure/aks/static-ip#apply-a-dns-label-to-the-service
apphostfullname=${apphostname}.${LOCATION}.cloudapp.azure.com
kubectl annotate service nginx service.beta.kubernetes.io/azure-dns-label-name=${apphostname} -n app-routing-system
And we’ll modify the ingress to require the host header:
cat <<EOF |kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-sample
spec:
ingressClassName: webapprouting.kubernetes.azure.com
rules:
- host: ${apphostfullname}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: server1
port:
number: 80
- path: /server2
pathType: Prefix
backend:
service:
name: server2
port:
number: 80
EOF
Give it a few minutes to propagate, then run:
$ curl http://lncblogdemo.westus3.cloudapp.azure.com
server1
curl http://lncblogdemo.westus3.cloudapp.azure.com/server2
server2
Discovery #3: It can generate a dummy TLS certificate
What if you want to test TLS but don’t have a certificate? (or don’t want to create a self-signed certificate?) In my explorations, I found that the addon will generate its own self-signed, “fake” certificate.
Let’s modify the ingress to use TLS:
cat <<EOF |kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-sample
spec:
ingressClassName: webapprouting.kubernetes.azure.com
tls:
- hosts:
- ${apphostfullname}
rules:
- host: ${apphostfullname}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: server1
port:
number: 80
- path: /server2
pathType: Prefix
backend:
service:
name: server2
port:
number: 80
EOF
Let’s use curl again. Notice that the first try fails due to an invalid tls certificate. I then use the -k option to ignore errors:
$ curl https://lncblogdemo.westus3.cloudapp.azure.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
$ curl -k https://lncblogdemo.westus3.cloudapp.azure.com
server1
Next let’s look at the certificate:
$ curl -k -v https://lncblogdemo.westus3.cloudapp.azure.com
* Trying 20.118.145.244:443...
* TCP_NODELAY set
* Connected to lncblogdemo.westus3.cloudapp.azure.com (20.118.145.244) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Oct 14 14:28:07 2022 GMT
* expire date: Oct 14 14:28:07 2023 GMT
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x560770fe78c0)
> GET / HTTP/2
> Host: lncblogdemo.westus3.cloudapp.azure.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Fri, 14 Oct 2022 15:57:29 GMT
< content-type: text/plain; charset=utf-8
< content-length: 7
< strict-transport-security: max-age=15724800; includeSubDomains
<
* Connection #0 to host lncblogdemo.westus3.cloudapp.azure.com left intact
server1
Notice the CN on the certificate: CN=Kubernetes Ingress Controller Fake Certificate
And there you have it! Three cool things you may not have known about the Web Application addon!
