Open in app

Sign in

Write

Sign in

Kubernetes IngressRoute with Traefik, LetsEncrypt and HAproxy

Allan John
9 min readMay 13, 2021

I was working recently on a single node cluster to test multiple applications. These applications required an Ingress resource to work with hostnames. I used Nginx Ingess in the past and recently I was told by one of my colleague and a mentor to try Traefik with HAproxy. I couldn’t find much docs on the internet, but there was a nice and simple document from traefik, which I modified to support my setup. So here i describe with a simple config to setup with LetsEncrypt for SSL certificates

This document describes to use

  • Traefik to request certificates to LetsEncrypt and store them in a location inside Traefik Pod, which is mounted as a volumeMount from the host machine. Certificates are issued from within Traefik
  • Ingress Route instead of Ingress, as Ingress Route is created as CRD as part of Traefik. The domains are not wildcard domains. They are already defined.
  • HAProxy is used a Loadbalancer, so that single node or multi-node (depends)kubernetes clusters can be deployed with its services exposed as NodePort and those NodePort is exposed to Internet with HAProxy. Here Traefik ports are exposed as Nodeports and to internet with HAProxy

Requirements

A working Kubernetes Cluster.

All code for this document is available here

Procedure

I am testing with a vanilla k8s cluster. Traefik requires a few Custom Resource Definitions (CRDs) to be installed first.

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.usspec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.usspec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.usspec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.usspec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
    singular: ingressrouteudp
  scope: Namespaced---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.usspec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.usspec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore
  scope: Namespaced---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.usspec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: serverstransports.traefik.containo.usspec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: ServersTransport
    plural: serverstransports
    singular: serverstransport
  scope: Namespaced

The Service Account, Cluster role and Cluster role binding

---apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: kube-system
  name: traefik-ingress-controller
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controllerrules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
      - ingressroutes
      - traefikservices
      - ingressroutetcps
      - ingressrouteudps
      - tlsoptions
      - tlsstores
      - serverstransports
    verbs:
      - get
      - list
      - watch---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controllerroleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: kube-system

The service and deployment

---
apiVersion: v1
kind: Service
metadata:
  name: traefik
  namespace: kube-system
spec:
  ports:
    - protocol: TCP
      name: http
      port: 80
    - protocol: TCP
      name: admin
      port: 8080
    - protocol: TCP
      name: https
      port: 443
  type: NodePort
  selector:
    app: traefik
---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: kube-system
  name: traefik
  labels:
    app: traefikspec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      volumes:
        - name: acme
          hostPath:
            path: /opt/traefik/acme
      containers:
        - name: traefik
          image: traefik:v2.4
          args:
            - --api.insecure
            - --accesslog
            - --entrypoints.https.Address=:443
            - --entrypoints.http.Address=:80
            - --providers.kubernetescrd
            - --certificatesresolvers.stageresolver.acme.tlschallenge
            - --certificatesresolvers.stageresolver.acme.email=mytestemail@gmail.com
            - --certificatesresolvers.stageresolver.acme.storage=/acme/staging-acme.json
            - --certificatesresolvers.stageresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
            - --certificatesresolvers.prodresolver.acme.tlschallenge
            - --certificatesresolvers.prodresolver.acme.email=mytestemail@gmail.com
            - --certificatesresolvers.prodresolver.acme.storage=/acme/prod-acme.json
            - --certificatesresolvers.prodresolver.acme.caserver=https://acme-v02.api.letsencrypt.org/directory
          ports:
            - name: web
              containerPort: 80
            - name: websecure
              containerPort: 443
            - name: admin
              containerPort: 8080
          volumeMounts:
            - mountPath: "/acme"
              name: "acme"

This Traefik yaml is modified one from this url. This is a set of Cluster Roles, Role bindings, service account, deployment and service for traefik. The important part to consider is the configuration. Here I am passing as CLI params, a better option would be to create a ConfigMap and load it as a file on the traefik pod

-api.insecure: Enable API for insecure requests. When running in Production, ensure to disable this and use provide access with authentication
--accesslog: Enable accesslog. Can be disabled, I prefer to have it running :)
--entrypoints.https.Address: The HTTPS port, preferably 443. The entrypoint name is https
--entrypoints.http.Address: The HTTP port, preferably 80. The entrypoint name is http
--providers.kubernetescrd: Enable Kubernetes CRD to communicate with the kuberneretes CRD created for Traefik
--certificatesresolvers.stageresolver.acme.tlschallenge: The Challenge to use for verification. Here stageresolveris the certificate resolver to use for staging applications
--certificatesresolvers.stageresolver.acme.email: The email to use for staging resolver. This email is necessary for getting information about certificate expiration.
--certificatesresolvers.stageresolver.acme.storage: The storage location to add the acme configuration, once the certificate is issued
--certificatesresolvers.stageresolver.acme.caserver: The LetsEncrypt Staging URL to issue certificates
--certificatesresolvers.prodresolver.acme.tlschallenge: The Challenge to use for verification. Here prodresolveris the certificate resolver to use for prod applications
--certificatesresolvers.prodresolver.acme.email: The email to use for prod resolver. This email is necessary for getting information about certificate expiration.
--certificatesresolvers.prodresolver.acme.storage: The storage location to add the acme configuration, once the certificate is issued
--certificatesresolvers.prodresolver.acme.caserver: The LetsEncrypt production URL to issue certificates

Now apply the changes:

kubectl apply -f traefik.yaml

Ensure the pods are running

$ kubectl -n kube-system get svc -l app=traefikNAME      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE         traefik   NodePort   10.100.39.110   <none>  80:30080/TCP,8080:30808/TCP,443:30443/TCP   110s$ kubectl -n kube-system get pod -l app=traefikNAME                       READY   STATUS    RESTARTS   AGE
traefik-7c64ff8bc6-4xdqj   1/1     Running   0          113s

Install and configure HAProxy, to expose the host ports 80, 443, 8080 to Kubernetes node ports to access the traefik service

sudo apt install haproxy -y

haproxy.cfg

global
 log /dev/log local0
 log /dev/log local1 notice
 chroot /var/lib/haproxy
 stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
 stats timeout 30s
 user haproxy
 group haproxy
 daemon# Default SSL material locations
 ca-base /etc/ssl/certs
 crt-base /etc/ssl/private# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-ticketsdefaults
 log global
 mode http
 option httplog
 option dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
 errorfile 400 /etc/haproxy/errors/400.http
 errorfile 403 /etc/haproxy/errors/403.http
 errorfile 408 /etc/haproxy/errors/408.http
 errorfile 500 /etc/haproxy/errors/500.http
 errorfile 502 /etc/haproxy/errors/502.http
 errorfile 503 /etc/haproxy/errors/503.http
 errorfile 504 /etc/haproxy/errors/504.httpfrontend traefik-ui
        bind *:8080
        mode tcp
        option tcplog
        default_backend traefik-uifrontend traefik-notls
        bind *:80
        mode tcp
        option tcplog
        default_backend traefik-notlsfrontend traefik-tls
        bind *:443
        mode tcp
        option tcplog
        default_backend traefik-tlsbackend traefik-ui
        mode tcp
        option tcplog
        option tcp-check
        default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
        server traefik 127.0.0.1:30808 checkbackend traefik-notls
        mode tcp
        option tcplog
        option tcp-check
        default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
        server traefik 127.0.0.1:30080 checkbackend traefik-tls
        mode tcp
        option tcplog
        option tcp-check
        default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
        server traefik 127.0.0.1:30443 check

Restart to apply changes to haproxy

sudo systemctl restart haproxy

Check Traefik Dashboard is visible:

http://cluster-ip:8080

Now the testing part. Create two applications. For ex: frontend and backend. I am going to run nginx in frontend and apache on backend. Both of them will be running on Test and Prod namespaces. Starting with Test

test-frontend.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: test
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-test
  namespace: test
spec:
  ports:
    - protocol: TCP
      name: frontend-web
      port: 80
  selector:
    app: frontend-test
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: frontend-svc
  namespace: test
  labels:
    app: frontend-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend-test
  template:
    metadata:
      labels:
        app: frontend-test
    spec:
      containers:
        - name: frontend
          image: nginx
          ports:
            - name: frontend-web
              containerPort: 80

test-backend.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: backend-test
  namespace: test
spec:
  ports:
    - protocol: TCP
      name: backend-web
      port: 80
  selector:
    app: backend-test
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: backend-svc
  namespace: test
  labels:
    app: backend-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend-test
  template:
    metadata:
      labels:
        app: backend-test
    spec:
      containers:
        - name: backend
          image: httpd
          ports:
            - name: backend-web
              containerPort: 80

This is a simple YAML files with a Service and a Deployment in a Namespace

kubectl apply -f test-frontend.yaml
kubectl apply -f test-backend.yaml

Check the output for frontend

$ NAME                                READY   STATUS    RESTARTS   AGE
pod/backend-svc-9598889dc-mdg5k     1/1     Running   0          14s
pod/backend-svc-9598889dc-rq26c     1/1     Running   0          14s
pod/frontend-svc-768888c4ff-gk57g   1/1     Running   0          2m7s
pod/frontend-svc-768888c4ff-qs5db   1/1     Running   0          2m7sNAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/backend-test    ClusterIP   10.104.51.193   <none>        80/TCP    14s
service/frontend-test   ClusterIP   10.105.89.215   <none>        80/TCP    2m7s

Now create an IngressRoute to the applications.

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: frontend-test
  namespace: test
spec:
  entryPoints:
    - http
  routes:
  - match: Host(`test-frontend.traefikingress.com`)
    kind: Rule
    services:
    - name: frontend-test
      port: 80
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: backend-test
  namespace: test
spec:
  entryPoints:
    - http
  routes:
  - match: Host(`test-backend.traefikingress.com`)
    kind: Rule
    services:
    - name: backend-test
      port: 80

Creating the ingress route to check if it works

kubectl apply -f test-ingress.yaml
test-frontend
test-backend

Now as the application is working with ingress route, I will now start to use TLS for the applications. Modify the test-ingress.yaml file and add the tls config, the https endpoint, and a valid domain name, because letsencrypt will need to verify the domain name. The modified ingress yaml looks like this

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: frontend-test
  namespace: test
spec:
  entryPoints:
    - https
  routes:
  - match: Host(`test-frontend.coveredyourface.com`)
    kind: Rule
    services:
    - name: frontend-test
      port: 80
  tls:
    certResolver: stageresolver
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: backend-test
  namespace: test
spec:
  entryPoints:
    - https
  routes:
  - match: Host(`test-backend.coveredyourface.com`)
    kind: Rule
    services:
    - name: backend-test
      port: 80
  tls:
    certResolver: stageresolver

Now applying the yaml file will enable the TLS config for the websites

kubectl apply -f test-ingress.yaml
test-backend
test-frontend

The certificate will show as insecure connection, because the certificates are issued from Staging. Lets deploy Prod with the same deployment, service and TLS with prod resolver, the LetsEncrypt Production URL

prod-frontend.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: prod
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-prod
  namespace: prod
spec:
  ports:
    - protocol: TCP
      name: frontendsvc
      port: 80
  selector:
    app: frontend-prod
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: frontend
  namespace: prod
  labels:
    app: frontend-prod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend-prod
  template:
    metadata:
      labels:
        app: frontend-prod
    spec:
      containers:
        - name: frontend
          image: nginx
          ports:
            - name: frontendsvc
              containerPort: 80

prod-backend.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: backend-prod
  namespace: prod
spec:
  ports:
    - protocol: TCP
      name: backend-web
      port: 80
  selector:
    app: backend-prod
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: backend-svc
  namespace: prod
  labels:
    app: backend-prod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend-prod
  template:
    metadata:
      labels:
        app: backend-prod
    spec:
      containers:
        - name: backend
          image: httpd
          ports:
            - name: backend-web
              containerPort: 80

prod-ingress.yaml

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: frontend-prod
  namespace: prod
spec:
  entryPoints:
    - https
  routes:
  - match: Host(`prod-frontend.coveredyourface.com`)
    kind: Rule
    services:
    - name: frontend-prod
      port: 80
  tls:
    certResolver: prodresolver
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: backend-prod
  namespace: prod
spec:
  entryPoints:
    - https
  routes:
  - match: Host(`prod-backend.coveredyourface.com`)
    kind: Rule
    services:
    - name: backend-prod
      port: 80
  tls:
    certResolver: prodresolver

Applying yamls

kubectl apply -f prod-backend.yaml
kubectl apply -f prod-frontend.yaml
kubectl apply -f prod-ingress.yaml
prod-frontend
prod-backend

Since we are using the Production Url of LetsEncrypt, the certificate will be valid.

An additional configuration to the traefik config, if required, would be to add http to https redirection. If so, in the cli parametes added in the traefik.yaml, add the following 2 lines

- --entrypoints.http.http.redirections.entrypoint.to=https
- --entrypoints.http.http.redirections.entrypoint.scheme=https

The first entry is to tell traefik to redirect http requests to ”https” endpoint name and the second entry is to use the scheme https

Conclusion

Traefik is a really nice tool to work with as an ingress resource. I had been using traefik as a frontend to redirect to services from Consul Backend and it was really nice. Trying out it on Kubernetes was the next step and with TLS provided by LetsEncrypt makes it more interesting when its less configuration

Hope you enjoyed!!

Traefik
Lets Encrypt
Kubernetes
DevOps
Tls

--

--

Allan John

Written by Allan John

117 Followers

Passionate DevOps and Automation Engineer

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams