Is EKS actually better or worse?
Introduction
This post is about my experience with Amazon EKS for deploying a kubernetes cluster as a noob on EKS.
Current situation
I have experience working on local kubernetes cluster like k3d, minikube and cloud provided clusters like AKS. Since the concepts are similar to local, I thought this might be similar and easy task to deploy EKS.
Trying to understand EKS
First I started to deploy EKS with Terraform. So I created the usual stuff to start with:
- A VPC
- 2 x public subnets
- 2 x private subnets
- 1 x NAT gateway
- 1 x Internet gateway
- Security groups
- Route tables with Routes
Then started the EKS deployment. At first I was confused, because in AKS, there is a default nodepool deployed by default. On researching I found there is 3 types of nodegroups
- EKS Managed Nodegroups: These are similar to what Azure provides. AWS provides us with the instances, and they take care of the OS patches on the nodes. This nodes have bootstrap script provided by AWS which is easy to find and join the cluster automatically. This configuration is preferred most of the time. Like deploying a test cluster.
- Self managed Nodegroups: This is used when we need full control over the instances. Like custom bootstrap script, instance role etc. This means we need to fully take care of the instance, OS upgrades and patches. This is really required when working in an organization with certain standards or tools that needs to be installed on the nodes and so on.
- Fargate Profile: This is fully managed EKS with no control over the compute resources. In here everything is managed by AWS. Users can deploy manifests in this cluster. I would not choose this anyway, I prefer some hands on stuff and not some black magic that does everything ;)
Configured and deployed a t2.medium Managed Nodegroup. This worked but the nodes were not able to connect to the cluster. On debugging I found out the security group permissions was not proper. So I opened eveything for the time being, because the nodegroups were deployed in private subnets and EKS on public and private subnets
Finally got the cluster up, and the only thing I can see was coredns, aws-node and kube-proxy pods. Looks like they are the ones provided by AWS by default.
Tried to install an ingress controller, and the loadbalancer cannot connect to the instances. Looks like a security group issue and improper tags
After much frustration and spending like over 4 days on a simple EKS deployment, I got a suggestion from a friend than eksctl was the tool to go with when playing around. And use terraform for production grade setup. So I started to look into it.
To be honest, eksctl looked much simple and easy to deploy. So basically it deploys EKS cluster from cli arguments or from a yaml config file. And I had a running cluster with all config I wanted within an hour! And there was no problem with network issues, cluster connection from nodes, ingress controller deployment, cluster autoscaler or anything at all that I got with deployment with terraform!!!
In a nutshell, eksctl is a tool which will prepare a Cloudformation template and run Cloudformation to deploy an EKS cluster based on the configuration that is provided. So now my idea changed like Terraform generates the config file for EKS and eksctl deploys the cluster with the configuration file.
Verdict?
For me personally, it was hard to understand first the concepts, but after wrapping my head around it, I can say EKS is a pretty good one when compared to other cloud provided clusters.
It allows to build the EKS cluster in any way the user requires. The cluster is totally empty with default configuration. This means the user can bring his own custom tools.
For example: AKS provides default cluster autoscaler. EKS does not provide, so we can choose to deploy any autoscaler tools like cluster-autoscaler or Karpenter in any namespace of your choice, Same goes for CSI drivers.
A disadvantage would be that when in need of a test cluster immediately, one can like AKS better or keep a configuration nearby at all times to deploy EKS with default config. Another one is about the kubernetes version. AKS is far ahead when compared with EKS in terms of kubernetes releases.
Conclusion
EKS is a very good option with kind of feature that allows one to configure the cluster in their own way, which I would say other Cloud providers are not providing it as such. Its like a managed service which can be customized to ones needs. And when getting to learn more on how to handle it, EKS is very impressive and really good with options like spot instances for nodegroups. In the end I am a pretty big fan . Let me know your thoughts if this is the same for you or if there is something you would like to add.
Hope you enjoyed, have a nice day!
