Kubernetes Ingress with Traefik, CertManager, LetsEncrypt and HAProxy
Introduction
This is a continuation of the part which I had written here. In this document I will describe, how to use
- Traefik only for loadbalancing the services
- Cert-Manger for Issuing certificates
- LetsEncrypt for SSL certificates
- Ingress instead of IngressRoute
- Domain used here is WildCard domain
Requirements
A working Kubernetes Cluster
The code for the document can be found here
Procedure
For WildCard domain, I created 2 DNS records on my registrar
- A Record: @ to the IP of the cluster
- CNAME Record: * to @
The advantage here, all subdomains created for the domain will be routed to the IP of the cluster. A disadvantage I suspect here is there is no possibility to add new subdomains to another location. To acheive it, add an additional subdomain. For ex:*.k8s. Lets do the fun stuff
Traefik
Deploy Traefik with YAML, Note that compared to the previous documentation, we are creating only a Cluster Role, Cluster role bingind, Service account, Service and Deployment. We avoided all CRDs
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controllerrules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
resources:
- ingresses
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- traefik.containo.us
resources:
- middlewares
- ingressroutes
- traefikservices
- ingressroutetcps
- ingressrouteudps
- tlsoptions
- tlsstores
- serverstransports
verbs:
- get
- list
- watch---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controllerroleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
name: traefik-ingress-controller
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: kube-system
labels:
app: traefik
spec:
ports:
- protocol: TCP
name: http
port: 80
nodePort: 30080
- protocol: TCP
name: admin
port: 8080
nodePort: 30808
- protocol: TCP
name: https
port: 443
nodePort: 30443
type: NodePort
selector:
app: traefik
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: kube-system
name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: kube-system
name: traefik
labels:
app: traefikspec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
containers:
- name: traefik
image: traefik:v2.4
args:
- --api.insecure
- --accesslog
- --entrypoints.https.Address=:443
- --entrypoints.http.Address=:80
- --providers.kubernetesIngress.ingressClass=traefik
ports:
- name: web
containerPort: 80
- name: websecure
containerPort: 443
- name: admin
containerPort: 8080
Note that the configuration for Traefik is also very short. We have configured the entrypoints and configured the ingressClass name. This is important when using certmanger and creating Issuer and Ingress resource. Applying the changes
kubectl apply -f traefik.yamlTraefik is running!
$ kubectl -n kube-system get po,svc -l=app=traefik
NAME READY STATUS RESTARTS AGE
pod/traefik-689dcfb858-bkc9t 1/1 Running 0 3sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/traefik NodePort 10.103.99.47 <none> 80:30080/TCP,8080:30808/TCP,443:30443/TCP 3s
Since we see the NodePorts are running, Lets configure HAProxy, So we dont need to use the ports all the time.
sudo apt install haproxyThe configuration is similar to the previous setup.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/privatedefaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.httpfrontend traefik-ui
bind *:8080
mode tcp
option tcplog
default_backend traefik-uifrontend traefik-notls
bind *:80
mode tcp
option tcplog
default_backend traefik-notlsfrontend traefik-tls
bind *:443
mode tcp
option tcplog
default_backend traefik-tlsbackend traefik-ui
mode tcp
option tcplog
option tcp-check
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server traefik 127.0.0.1:30808 checkbackend traefik-notls
mode tcp
option tcplog
option tcp-check
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server traefik 127.0.0.1:30080 checkbackend traefik-tls
mode tcp
option tcplog
option tcp-check
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server traefik 127.0.0.1:30443 check
Restart for the effect to take place
sudo systemctl restart haproxyCert-Manager
Apply the YAML file to configure cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yamlVerify the cert-manager is running
$ kubectl -n cert-manager get po
NAME READY STATUS RESTARTS AGE
cert-manager-7dd5854bb4-nqt7d 1/1 Running 0 3s
cert-manager-cainjector-64c949654c-8lmgz 1/1 Running 0 3s
cert-manager-webhook-6b57b9b886-t4pgc 1/1 Running 0 3sNow lets configure the Issuer. Here I use ClusterIssuer because I want something a cluster-wide. Issuer can also be used, if we need the Issuer for namespaces and not cluster-wide.
Configure the Cluster Issuer for Staging and Prod
staging-issuer.yaml
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: myemail@email.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: letsencrypt-staging-acc-key
solvers:
- http01:
ingress:
class: traefikIts important to add a valid email, if you care about the certificate validity and to get information of certificates or anything else. Note that the ingress class I mentioned here should be equal to the ingress class configured in Traefik. Creating one for Prod as well
prod-issuer.yaml
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# You must replace this email address with your own.
# Let’s Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: myemail@email.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account’s private key.
name: letsencrypt-prod-acc-key
solvers:
- http01:
ingress:
class: traefikApplying Issuers
kubectl apply -f staging-issuer.yaml
kubectl apply -f prod-issuer.yamlDeploy Apps — Staging
Frontend is an Nginx deployment and Backend is an Apache deployment. Just to show the difference. Create a namespace first for test
kubectl create ns teststaging-frontend.yaml
---
apiVersion: v1
kind: Service
metadata:
name: frontend-test
namespace: test
spec:
ports:
- protocol: TCP
name: frontendsvc
port: 80
selector:
app: frontend-test
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: frontend
namespace: test
labels:
app: frontend-test
spec:
replicas: 2
selector:
matchLabels:
app: frontend-test
template:
metadata:
labels:
app: frontend-test
spec:
containers:
- name: frontend
image: nginx
ports:
- name: frontendsvc
containerPort: 80staging-backend.yaml
---
apiVersion: v1
kind: Service
metadata:
name: backend-test
namespace: test
spec:
ports:
- protocol: TCP
name: backend-web
port: 80
selector:
app: backend-test
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: backend-svc
namespace: test
labels:
app: backend-test
spec:
replicas: 2
selector:
matchLabels:
app: backend-test
template:
metadata:
labels:
app: backend-test
spec:
containers:
- name: backend
image: httpd
ports:
- name: backend-web
containerPort: 80Apply the deployments
kubectl apply -f staging-frontend.yaml
kubectl apply -f staging-backend.yamlCheck the resources are deployed
$ kubectl -n test get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
backend-svc 2/2 2 2 27s
frontend 2/2 2 2 33sNow lets configure the Ingress for staging
staging-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: backend-ingress
namespace: test
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
- hosts:
- "test-backend.coveredyourface.com"
secretName: test-backend-cert
rules:
- host: "test-backend.coveredyourface.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: backend-test
port:
number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frontend-ingress
namespace: test
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
- hosts:
- "test-frontend.coveredyourface.com"
secretName: test-frontend-cert
rules:
- host: "test-frontend.coveredyourface.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-test
port:
number: 80Note the annotations, which makes the ingress to use Traefik, and the cluster issuer. The last annotation traefik.ingress.kubernetes.io/router.tls: "true" is important. If this is omitted, the TLS will not be applied on the ingress. All requests will be passed to the traefik entrypoint http
Applying ingress
kubectl apply -f staging-ingress.yamlTo check if the certificate is issued, describe the ingress resource
$ kubectl -n test describe ingress backend-ingress
Name: backend-ingress
Namespace: test
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
test-backend-cert terminates test-backend.coveredyourface.com
Rules:
Host Path Backends
---- ---- --------
test-backend.coveredyourface.com
/ backend-test:80 (10.244.0.40:80,10.244.0.41:80)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-staging
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateCertificate 18s cert-manager Successfully created Certificate "test-backend-cert"Checking the status of certificate now
$ kubectl -n test describe certificate test-backend-cert......
Spec:
Dns Names:
test-backend.coveredyourface.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-staging
Secret Name: test-backend-cert
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2021-05-16T08:39:05Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2021-08-14T07:39:04Z
Not Before: 2021-05-16T07:39:04Z
Renewal Time: 2021-07-15T07:39:04Z
Revision: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 10s cert-manager Issuing certificate as Secret does not exist
Normal Generated 10s cert-manager Stored new private key in temporary Secret resource "test-backend-cert-kngfp"
Normal Requested 10s cert-manager Created new CertificateRequest resource "test-backend-cert-rddzg"
Normal Issuing 7s cert-manager The certificate has been successfully issued
Here we can see the secret for the certificate is created and the certificate is succesfully issued. Lets check backend now.
Here the certificate will show invalid, as it is issued by the LetsEncrypt staging. This will be similar to frontend as well
Deploy Apps — Prod
The deployment is straightforward. Similar to Test, all resources are same except for the namespaces and the resource names
Create Namespace for prod
kubectl create ns prodprod-frontend.yaml
---
apiVersion: v1
kind: Service
metadata:
name: frontend-prod
namespace: prod
spec:
ports:
- protocol: TCP
name: frontendsvc
port: 80
selector:
app: frontend-prod
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: frontend
namespace: prod
labels:
app: frontend-prod
spec:
replicas: 2
selector:
matchLabels:
app: frontend-prod
template:
metadata:
labels:
app: frontend-prod
spec:
containers:
- name: frontend
image: nginx
ports:
- name: frontendsvc
containerPort: 80prod-backend.yaml
---
apiVersion: v1
kind: Service
metadata:
name: backend-prod
namespace: prod
spec:
ports:
- protocol: TCP
name: backend-web
port: 80
selector:
app: backend-prod
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: backend-svc
namespace: prod
labels:
app: backend-prod
spec:
replicas: 2
selector:
matchLabels:
app: backend-prod
template:
metadata:
labels:
app: backend-prod
spec:
containers:
- name: backend
image: httpd
ports:
- name: backend-web
containerPort: 80Applying deployments
kubectl apply -f prod-frontend.yaml
kubectl apply -f prod-backend.yamlConfiguring ingress. Instead of adding 2 ingresses as I done for staging, Here I am going to add them as a single resource
prod-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prod-ingress
namespace: prod
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
- hosts:
- "prod-frontend.coveredyourface.com"
- "prod-backend.coveredyourface.com"
secretName: prod-cert
rules:
- host: "prod-frontend.coveredyourface.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-prod
port:
number: 80
- host: "prod-backend.coveredyourface.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: backend-prod
port:
number: 80Applying the yaml
kubectl apply -f prod-ingress.yamlChecking ingress
$ kubectl -n prod describe ingress prod-ingress
Name: prod-ingress
Namespace: prod
Address:
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
prod-cert terminates prod-frontend.coveredyourface.com,prod-backend.coveredyourface.com
Rules:
Host Path Backends
---- ---- --------
prod-frontend.coveredyourface.com
/ frontend-prod:80 (10.244.0.42:80,10.244.0.43:80)
prod-backend.coveredyourface.com
/ backend-prod:80 (10.244.0.44:80,10.244.0.45:80)
Annotations: cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateCertificate 12s cert-manager Successfully created Certificate "prod-cert"Here two ingress resources can be seen. Check certificates
$ kubectl -n prod describe certificate prod-cert
Name: prod-cert
Namespace: prod
........
Spec:
Dns Names:
prod-frontend.coveredyourface.com
prod-backend.coveredyourface.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-prod
Secret Name: prod-cert
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2021-05-16T08:57:05Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2021-08-14T07:57:05Z
Not Before: 2021-05-16T07:57:05Z
Renewal Time: 2021-07-15T07:57:05Z
Revision: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 26s cert-manager Issuing certificate as Secret does not exist
Normal Generated 25s cert-manager Stored new private key in temporary Secret resource "prod-cert-x5kwz"
Normal Requested 25s cert-manager Created new CertificateRequest resource "prod-cert-l47g9"
Normal Issuing 2s cert-manager The certificate has been successfully issued
The certificate is Ready as its Issued properly without issues. Lets check the website now
Here we can see the certificates are valid and is Issued by the Prod CA of Lets Encrypt
Conclusion
Traefik is a really nice Loadbalancing solution which can be used for microservices or services to communicate each other and use Ingress to expose the services the internet with hostnames. SSL certificates issuing from Cert-manager is really nice and splitting from traefik is very good, because with cert-manager in the setup we have the possibility to use other external certificate providers or Vault or Venafi.
Hope you enjoyed!! Thank you
