Kubernetes Ingress with Traefik, CertManager, LetsEncrypt and HAProxy

Introduction

This is a continuation of the part which I had written here. In this document I will describe, how to use

  • Traefik only for loadbalancing the services
  • Cert-Manger for Issuing certificates
  • LetsEncrypt for SSL certificates
  • Ingress instead of IngressRoute
  • Domain used here is WildCard domain

Requirements

A working Kubernetes Cluster

The code for the document can be found here

Procedure

For WildCard domain, I created 2 DNS records on my registrar

  • A Record: @ to the IP of the cluster
  • CNAME Record: * to @

The advantage here, all subdomains created for the domain will be routed to the IP of the cluster. A disadvantage I suspect here is there is no possibility to add new subdomains to another location. To acheive it, add an additional subdomain. For ex:*.k8s. Lets do the fun stuff

Traefik

Deploy Traefik with YAML, Note that compared to the previous documentation, we are creating only a Cluster Role, Cluster role bingind, Service account, Service and Deployment. We avoided all CRDs

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
- ingressroutes
- traefikservices
- ingressroutetcps
- ingressrouteudps
- tlsoptions
- tlsstores
- serverstransports
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: kube-system
labels:
app: traefik
spec:
ports:
- protocol: TCP
name: http
port: 80
nodePort: 30080
- protocol: TCP
name: admin
port: 8080
nodePort: 30808
- protocol: TCP
name: https
port: 443
nodePort: 30443
type: NodePort
selector:
app: traefik
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: kube-system
name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: kube-system
name: traefik
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.4
args:
- --api.insecure
- --accesslog
- --entrypoints.https.Address=:443
- --entrypoints.http.Address=:80
- --providers.kubernetesIngress.ingressClass=traefik
ports:
- name: web
containerPort: 80
- name: websecure
containerPort: 443
- name: admin
containerPort: 8080

Note that the configuration for Traefik is also very short. We have configured the entrypoints and configured the ingressClass name. This is important when using certmanger and creating Issuer and Ingress resource. Applying the changes

kubectl apply -f traefik.yaml

Traefik is running!

$ kubectl -n kube-system get po,svc -l=app=traefik
NAME READY STATUS RESTARTS AGE
pod/traefik-689dcfb858-bkc9t 1/1 Running 0 3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/traefik NodePort 10.103.99.47 <none> 80:30080/TCP,8080:30808/TCP,443:30443/TCP 3s

Since we see the NodePorts are running, Lets configure HAProxy, So we dont need to use the ports all the time.

sudo apt install haproxy

The configuration is similar to the previous setup.

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend traefik-ui
bind *:8080
mode tcp
option tcplog
default_backend traefik-ui
frontend traefik-notls
bind *:80
mode tcp
option tcplog
default_backend traefik-notls
frontend traefik-tls
bind *:443
mode tcp
option tcplog
default_backend traefik-tls
backend traefik-ui
mode tcp
option tcplog
option tcp-check
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server traefik 127.0.0.1:30808 check
backend traefik-notls
mode tcp
option tcplog
option tcp-check
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server traefik 127.0.0.1:30080 check
backend traefik-tls
mode tcp
option tcplog
option tcp-check
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server traefik 127.0.0.1:30443 check

Restart for the effect to take place

sudo systemctl restart haproxy

Cert-Manager

Apply the YAML file to configure cert-manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml

Verify the cert-manager is running

$ kubectl -n cert-manager get po
NAME READY STATUS RESTARTS AGE
cert-manager-7dd5854bb4-nqt7d 1/1 Running 0 3s
cert-manager-cainjector-64c949654c-8lmgz 1/1 Running 0 3s
cert-manager-webhook-6b57b9b886-t4pgc 1/1 Running 0 3s

Now lets configure the Issuer. Here I use ClusterIssuer because I want something a cluster-wide. Issuer can also be used, if we need the Issuer for namespaces and not cluster-wide.

Configure the Cluster Issuer for Staging and Prod

staging-issuer.yaml

---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: myemail@email.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: letsencrypt-staging-acc-key
solvers:
- http01:
ingress:
class: traefik

Its important to add a valid email, if you care about the certificate validity and to get information of certificates or anything else. Note that the ingress class I mentioned here should be equal to the ingress class configured in Traefik. Creating one for Prod as well

prod-issuer.yaml

---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# You must replace this email address with your own.
# Let’s Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: myemail@email.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account’s private key.
name: letsencrypt-prod-acc-key
solvers:
- http01:
ingress:
class: traefik

Applying Issuers

kubectl apply -f staging-issuer.yaml
kubectl apply -f prod-issuer.yaml

Deploy Apps — Staging

Frontend is an Nginx deployment and Backend is an Apache deployment. Just to show the difference. Create a namespace first for test

kubectl create ns test

staging-frontend.yaml

---
apiVersion: v1
kind: Service
metadata:
name: frontend-test
namespace: test
spec:
ports:
- protocol: TCP
name: frontendsvc
port: 80
selector:
app: frontend-test
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: frontend
namespace: test
labels:
app: frontend-test
spec:
replicas: 2
selector:
matchLabels:
app: frontend-test
template:
metadata:
labels:
app: frontend-test
spec:
containers:
- name: frontend
image: nginx
ports:
- name: frontendsvc
containerPort: 80

staging-backend.yaml

---
apiVersion: v1
kind: Service
metadata:
name: backend-test
namespace: test
spec:
ports:
- protocol: TCP
name: backend-web
port: 80
selector:
app: backend-test
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: backend-svc
namespace: test
labels:
app: backend-test
spec:
replicas: 2
selector:
matchLabels:
app: backend-test
template:
metadata:
labels:
app: backend-test
spec:
containers:
- name: backend
image: httpd
ports:
- name: backend-web
containerPort: 80

Apply the deployments

kubectl apply -f staging-frontend.yaml
kubectl apply -f staging-backend.yaml

Check the resources are deployed

$ kubectl -n test get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
backend-svc 2/2 2 2 27s
frontend 2/2 2 2 33s

Now lets configure the Ingress for staging

staging-ingress.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: backend-ingress
namespace: test
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
- hosts:
- "test-backend.coveredyourface.com"
secretName: test-backend-cert
rules:
- host: "test-backend.coveredyourface.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: backend-test
port:
number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frontend-ingress
namespace: test
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
- hosts:
- "test-frontend.coveredyourface.com"
secretName: test-frontend-cert
rules:
- host: "test-frontend.coveredyourface.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-test
port:
number: 80

Note the annotations, which makes the ingress to use Traefik, and the cluster issuer. The last annotation traefik.ingress.kubernetes.io/router.tls: "true" is important. If this is omitted, the TLS will not be applied on the ingress. All requests will be passed to the traefik entrypoint http

Applying ingress

kubectl apply -f staging-ingress.yaml

To check if the certificate is issued, describe the ingress resource

$ kubectl -n test describe ingress backend-ingress 
Name: backend-ingress
Namespace: test
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
test-backend-cert terminates test-backend.coveredyourface.com
Rules:
Host Path Backends
---- ---- --------
test-backend.coveredyourface.com
/ backend-test:80 (10.244.0.40:80,10.244.0.41:80)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateCertificate 18s cert-manager Successfully created Certificate "test-backend-cert"

Checking the status of certificate now

$ kubectl -n test describe certificate test-backend-cert......
Spec:
Dns Names:
test-backend.coveredyourface.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-staging
Secret Name: test-backend-cert
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2021-05-16T08:39:05Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2021-08-14T07:39:04Z
Not Before: 2021-05-16T07:39:04Z
Renewal Time: 2021-07-15T07:39:04Z
Revision: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 10s cert-manager Issuing certificate as Secret does not exist
Normal Generated 10s cert-manager Stored new private key in temporary Secret resource "test-backend-cert-kngfp"
Normal Requested 10s cert-manager Created new CertificateRequest resource "test-backend-cert-rddzg"
Normal Issuing 7s cert-manager The certificate has been successfully issued

Here we can see the secret for the certificate is created and the certificate is succesfully issued. Lets check backend now.

backend-test

Here the certificate will show invalid, as it is issued by the LetsEncrypt staging. This will be similar to frontend as well

frontend-test

Deploy Apps — Prod

The deployment is straightforward. Similar to Test, all resources are same except for the namespaces and the resource names

Create Namespace for prod

kubectl create ns prod

prod-frontend.yaml

---
apiVersion: v1
kind: Service
metadata:
name: frontend-prod
namespace: prod
spec:
ports:
- protocol: TCP
name: frontendsvc
port: 80
selector:
app: frontend-prod
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: frontend
namespace: prod
labels:
app: frontend-prod
spec:
replicas: 2
selector:
matchLabels:
app: frontend-prod
template:
metadata:
labels:
app: frontend-prod
spec:
containers:
- name: frontend
image: nginx
ports:
- name: frontendsvc
containerPort: 80

prod-backend.yaml

---
apiVersion: v1
kind: Service
metadata:
name: backend-prod
namespace: prod
spec:
ports:
- protocol: TCP
name: backend-web
port: 80
selector:
app: backend-prod
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: backend-svc
namespace: prod
labels:
app: backend-prod
spec:
replicas: 2
selector:
matchLabels:
app: backend-prod
template:
metadata:
labels:
app: backend-prod
spec:
containers:
- name: backend
image: httpd
ports:
- name: backend-web
containerPort: 80

Applying deployments

kubectl apply -f prod-frontend.yaml
kubectl apply -f prod-backend.yaml

Configuring ingress. Instead of adding 2 ingresses as I done for staging, Here I am going to add them as a single resource

prod-ingress.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prod-ingress
namespace: prod
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
- hosts:
- "prod-frontend.coveredyourface.com"
- "prod-backend.coveredyourface.com"
secretName: prod-cert
rules:
- host: "prod-frontend.coveredyourface.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-prod
port:
number: 80
- host: "prod-backend.coveredyourface.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: backend-prod
port:
number: 80

Applying the yaml

kubectl apply -f prod-ingress.yaml

Checking ingress

$ kubectl -n prod describe ingress prod-ingress 
Name: prod-ingress
Namespace: prod
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
prod-cert terminates prod-frontend.coveredyourface.com,prod-backend.coveredyourface.com
Rules:
Host Path Backends
---- ---- --------
prod-frontend.coveredyourface.com
/ frontend-prod:80 (10.244.0.42:80,10.244.0.43:80)
prod-backend.coveredyourface.com
/ backend-prod:80 (10.244.0.44:80,10.244.0.45:80)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateCertificate 12s cert-manager Successfully created Certificate "prod-cert"

Here two ingress resources can be seen. Check certificates

$ kubectl -n prod describe certificate prod-cert 
Name: prod-cert
Namespace: prod
....
....

Spec:
Dns Names:
prod-frontend.coveredyourface.com
prod-backend.coveredyourface.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-prod
Secret Name: prod-cert
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2021-05-16T08:57:05Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2021-08-14T07:57:05Z
Not Before: 2021-05-16T07:57:05Z
Renewal Time: 2021-07-15T07:57:05Z
Revision: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 26s cert-manager Issuing certificate as Secret does not exist
Normal Generated 25s cert-manager Stored new private key in temporary Secret resource "prod-cert-x5kwz"
Normal Requested 25s cert-manager Created new CertificateRequest resource "prod-cert-l47g9"
Normal Issuing 2s cert-manager The certificate has been successfully issued

The certificate is Ready as its Issued properly without issues. Lets check the website now

prod-frontend
prod-backend

Here we can see the certificates are valid and is Issued by the Prod CA of Lets Encrypt

Conclusion

Traefik is a really nice Loadbalancing solution which can be used for microservices or services to communicate each other and use Ingress to expose the services the internet with hostnames. SSL certificates issuing from Cert-manager is really nice and splitting from traefik is very good, because with cert-manager in the setup we have the possibility to use other external certificate providers or Vault or Venafi.

Hope you enjoyed!! Thank you

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store