Skip to content

Commit 9f09347

Browse files
lentidaslentidas
committed
feat(examples/eks): add updated and improved EKS deployment example
1 parent 85297b7 commit 9f09347

13 files changed

Lines changed: 655 additions & 330 deletions

File tree

‎docs/modules/ROOT/pages/tutorials/deploy_eks.adoc‎

Lines changed: 192 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,193 @@
1-
= Deployment On Amazon EKS
1+
= Deployment on Amazon EKS
22

3-
_Work In Progress_
3+
An example of a local deployment on Amazon EKS is provided https://github.com/camptocamp/devops-stack/tree/main/examples/eks[here]. Clone this repository and modify the files at your convenience.
4+
In the folder, as in a standard https://developer.hashicorp.com/terraform/tutorials/modules/module#what-is-a-terraform-module[Terraform module], you will find the following files:
5+
6+
* *`terraform.tf`* - declaration of the Terraform providers used in this project;
7+
* *`locals.tf`* - local variables used by the DevOps Stack modules;
8+
* *`main.tf`* - definition of all the deployed modules;
9+
* *`s3_loki.tf`* - creation of the IAM policy, assumable role and bucket used by Loki;
10+
* *`s3_thanos.tf`* - creation of the IAM policy, assumable role and bucket used by Thanos;
11+
* *`csi_drivers.tf`* - creation of the required resources as well as the DevOps Stack modules needed for the CSI drivers of the cluster;
12+
* *`outputs.tf`* - the output variables of the DevOps Stack;
13+
14+
== Requirements
15+
16+
On your local machine, you need to have the following tools installed:
17+
18+
* https://www.terraform.io/[Terraform] to provision the whole stack;
19+
* https://kubernetes.io/docs/reference/kubectl/[`kubectl`] or https://github.com/derailed/k9s[`k9s`]to interact with your cluster;
20+
* https://aws.amazon.com/cli/[AWS CLI] to interact with your AWS account;
21+
* https://dev.to/camptocamp-ops/simple-secret-sharing-with-gopass-and-summon-40jk[`gopass` and `summon`] to easily pass the IAM secrets as environment variables when running `terraform` commands;
22+
23+
Other than that, you will require the following:
24+
25+
* an AWS account;
26+
* an AWS IAM key with at least the ... ... ...
27+
// TODO Check with Christian for other requirements and specificities
28+
* a Route 53 zone;
29+
30+
== Specificities and explanations
31+
32+
=== `secrets.yml`
33+
34+
TIP: Check https://dev.to/camptocamp-ops/simple-secret-sharing-with-gopass-and-summon-40jk[this blog post] for more information on how to configure `gopass` and `summon` to work together.
35+
36+
For simplicity and ease of use, as well as security, the example uses `gopass` and `summon` to pass the IAM credentials to the Terraform commands. The `secrets.yml` file contains the path to the the secret values on the `gopass` password store. On execution, the `summon` command will then read the `secrets.yml` file and pass the credentials as environment variables to the Terraform commands.
37+
38+
The commands presented on this tutorial all use the `summon` command.
39+
40+
IMPORTANT: The environment variable `AWS_DEFAULT_REGION` defines where all the AWS resources created by Terraform will reside, including the EKS cluster.
41+
42+
=== Remote Terraform state
43+
44+
If you do not want to configure the remote Terraform state backend, you can simply remove the `backend` block from the `terraform.tf` file.
45+
46+
NOTE: More information about the remote backends is available on the https://developer.hashicorp.com/terraform/language/settings/backends/configuration[official documentation].
47+
48+
=== OIDC authentication
49+
50+
IMPORTANT: The DevOps Stack modules are developed with OIDC in mind. In production, you should have an identity provider that supports OIDC and use it to authenticate to the DevOps Stack applications.
51+
52+
In this example, we use the https://aws.amazon.com/eks/[Amazon EKS] OIDC provider. We provide a xref:oidc-aws-cognito:ROOT:README.adoc[module] that takes in a Cognito pool ID and its domain to provide you with the required configuration to deploy the DevOps Stack applications.
53+
54+
This assumes that you have created a Cognito pool yourself, however you can use our module to also create the pool and populate it with users, as shown in the example.
55+
56+
NOTE: Check the xref:oidc-aws-cognito:ROOT:README.adoc[AWS Cognito OIDC] usage documentation for more information on how to use it.
57+
58+
The `user_map` variable of that module allows you to create OIDC users used to authenticate to the DevOps Stack applications. You should receive an e-mail from AWS with a temporary password to login for the first time.
59+
60+
=== Let's Encrypt SSL certificates
61+
62+
By default, to avoid rate-limiting your domain by Let's Encrypt, the example uses the `letsencrypt-staging` configuration of the cert-manager module to generate certificates using the Let's Encrypt staging environment which has an invalid CA certificate.
63+
64+
If you feel ready to test with production certificates, you can simply edit the `locals.tf` file and change the `cluster_issuer` variable to `letsencrypt-prod`.
65+
66+
== Deployment
67+
68+
1. Clone the repository and `cd` into the `examples/eks` folder;
69+
70+
2. Adapt the `secrets.yml` file to point to the correct path on your `gopass` password store;
71+
72+
3. Check out the modules you want to deploy in the `main.tf` file, and comment out the others;
73+
+
74+
TIP: You can also add your own Terraform modules in this file or any other file on the root folder. A good place to start to write your own module is to clone the https://github.com/camptocamp/devops-stack-module-template[devops-stack-module-template] repository and adapt it to your needs.
75+
76+
4. On the `oidc` module, adapt the `user_map` variable as you wish (please check the <<oidc-authentication,OIDC section>> for more information).
77+
78+
5. From the source of the example deployment, initialize the Terraform modules and providers:
79+
+
80+
[source,bash]
81+
----
82+
summon terraform init
83+
----
84+
85+
6. Configure the variables in `locals.tf` to your preference:
86+
+
87+
IMPORTANT: The `cluster_name` and `vpc_cidr` must be unique for each DevOps Stack deployment in a single AWS account and the `base_domain` must match a Route 53 zone in that same account.
88+
+
89+
TIP: The xref:eks:ROOT:README.adoc[cluster module documentation] can help you know what to put in the `kubernetes_version`, for example.
90+
+
91+
[source,terraform]
92+
----
93+
include::example$deploy_examples/sks/locals.tf[]
94+
----
95+
96+
7. Finally, run `terraform apply` and accept the proposed changes to create the Kubernetes nodes on Amazon EKS and populate them with our services;
97+
+
98+
[source,bash]
99+
----
100+
summon terraform apply
101+
----
102+
103+
8. After the first deployment (please note the troubleshooting step related with Argo CD), you can go to the `locals.tf` and enable the _ServiceMonitor_ boolean to activate the Prometheus exporters that will send metrics to Prometheus;
104+
+
105+
IMPORTANT: This flag needs to be set as `false` for the first bootstrap of the cluster, otherwise the applications will fail to deploy while the Custom Resource Definitions of the kube-prometheus-stack are not yet created.
106+
+
107+
NOTE: You can either set the flag as `true` in the `locals.tf` file or you can simply delete the line on the modules' declarations, since this variable is set as `true` by default on each module.
108+
+
109+
TIP: Take note of the local called `app_autosync`. If you set the condition of the ternary operator to `false` you will disable the auto-sync for all the DevOps Stack modules. This allows you to choose when to manually sync the module on the Argo CD interface and is useful for troubleshooting purposes.
110+
111+
== Access the cluster and the DevOps Stack applications
112+
113+
To access your cluster you need to use the AWS CLI to recover a Kubeconfig you can use:
114+
115+
[source,bash]
116+
----
117+
summon aws eks update-kubeconfig --name YOUR_CLUSTER_NAME --region YOUR_CLUSTER_ZONE --kubeconfig ~/.kube/NAME_TO_GIVE_YOUR_CONFIG.config
118+
----
119+
120+
Then you can use the `kubectl` or `k9s` command to interact with the cluster:
121+
122+
[source,bash]
123+
----
124+
k9s --kubeconfig ~/.kube/NAME_TO_GIVE_YOUR_CONFIG.config
125+
----
126+
127+
As for the DevOps Stack applications, you can access them through the ingress domain that you can find in the `ingress_domain` output. If you used the code from the example without modifying the outputs, you will see something like this on your terminal after the `terraform apply` has done its job:
128+
129+
[source,shell]
130+
----
131+
Outputs:
132+
133+
devops_admins = <sensitive>
134+
ingress_domain = "your.domain.here"
135+
----
136+
137+
Or you can use `kubectl` to get all the ingresses and their respective URLs:
138+
139+
[source,bash]
140+
----
141+
kubectl get ingress --all-namespaces --kubeconfig ~/.kube/NAME_TO_GIVE_YOUR_CONFIG.config
142+
----
143+
144+
The output `devops_admins` list all users and respective e-mails that were configured using the OIDC module:
145+
146+
[source,bash]
147+
----
148+
summon terraform output devops_admins
149+
----
150+
151+
Those users should have received an e-mail with a temporary password in order to login to the DevOps Stack applications for the first time.
152+
153+
== Stop the cluster
154+
155+
To definitively stop the cluster on a single command (that is the reason we delete some resources from the state file), you can use the following command:
156+
157+
[source,bash]
158+
----
159+
summon terraform state rm $(summon terraform state list | grep "argocd_application\|argocd_project\|kubernetes_\|helm_") && summon terraform destroy
160+
----
161+
162+
== Conclusion
163+
164+
That's it, you now have a fully functional Kubernetes cluster in Amazon EKS with the DevOps Stack applications deployed on it. For more information, keep on reading the https://devops-stack.io/docs/latest/[documentation]. **You can explore the possibilities of each module and get the link to the source code on their respective documentation pages.**
165+
166+
== Troubleshooting
167+
168+
=== `connection_error` during the first deployment
169+
170+
In some cases, you could encounter an error like these the first deployment:
171+
172+
[source,shell]
173+
----
174+
╷
175+
│ Error: error while waiting for application argocd to be created
176+
│
177+
│ with module.argocd.argocd_application.this,
178+
│ on .terraform/modules/argocd/main.tf line 55, in resource "argocd_application" "this":
179+
│ 55: resource "argocd_application" "this" {
180+
│
181+
│ error while waiting for application argocd to be synced and healthy: rpc error: code = Unavailable desc = error reading from server: EOF
182+
╵
183+
----
184+
185+
The error is due to the way we provision Argo CD on the final steps of the deployment. We use the bootstrap Argo CD to deploy the final Argo CD module, which causes a redeployment of Argo CD and consequently a momentary loss of connection between the Argo CD Terraform provider and the Argo CD server.
186+
187+
*You can simply re-run the command `summon terraform apply` to finalize the bootstrap of the cluster every time you encounter this error.*
188+
189+
=== Argo CD interface reload loop when clicking on login
190+
191+
If you encounter a loop when clicking on the login button on the Argo CD interface, you can try to delete the Argo CD server pod and let it be recreated.
192+
193+
TIP: For more informations about the Argo CD module, please refer to the xref:argocd:ROOT:README.adoc[respective documentation page].

‎examples/eks/.terraform-version‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
1.5.2

‎examples/eks/apps.tf‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
module "helloworld_apps" {
2+
source = "git::https://github.com/camptocamp/devops-stack-module-applicationset.git?ref=v2.0.1"
3+
4+
dependency_ids = {
5+
argocd = module.argocd.id
6+
}
7+
8+
name = "helloworld-apps"
9+
argocd_namespace = module.argocd_bootstrap.argocd_namespace
10+
project_dest_namespace = "*"
11+
project_source_repo = "https://github.com/camptocamp/devops-stack-helloworld-templates.git"
12+
13+
app_autosync = local.app_autosync
14+
15+
generators = [
16+
{
17+
git = {
18+
repoURL = "https://github.com/camptocamp/devops-stack-helloworld-templates.git"
19+
revision = "main"
20+
21+
directories = [
22+
{
23+
path = "apps/*"
24+
}
25+
]
26+
}
27+
}
28+
]
29+
template = {
30+
metadata = {
31+
name = "{{path.basename}}"
32+
}
33+
34+
spec = {
35+
project = "helloworld-apps"
36+
37+
source = {
38+
repoURL = "https://github.com/camptocamp/devops-stack-helloworld-templates.git"
39+
targetRevision = "main"
40+
path = "{{path}}"
41+
42+
helm = {
43+
valueFiles = []
44+
# The following value defines this global variables that will be available to all apps in apps/*
45+
# These are needed to generate the ingresses containing the name and base domain of the cluster.
46+
values = <<-EOT
47+
cluster:
48+
name: "${local.cluster_name}"
49+
domain: "${local.base_domain}"
50+
issuer: "${local.cluster_issuer}"
51+
apps:
52+
traefik_dashboard: false
53+
grafana: true
54+
prometheus: true
55+
thanos: true
56+
alertmanager: true
57+
EOT
58+
}
59+
}
60+
61+
destination = {
62+
name = "in-cluster"
63+
namespace = "{{path.basename}}"
64+
}
65+
66+
syncPolicy = {
67+
automated = {
68+
allowEmpty = false
69+
selfHeal = true
70+
prune = true
71+
}
72+
syncOptions = [
73+
"CreateNamespace=true"
74+
]
75+
}
76+
}
77+
}
78+
}

‎examples/eks/csi_drivers.tf‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
resource "aws_efs_file_system" "eks" {
2+
creation_token = module.eks.cluster_name
3+
4+
tags = {
5+
Name = module.eks.cluster_name
6+
}
7+
}
8+
9+
resource "aws_security_group" "efs_eks" {
10+
name = "efs-devops-stack"
11+
description = "Security group for EFS"
12+
vpc_id = module.vpc.vpc_id
13+
14+
egress {
15+
from_port = 0
16+
to_port = 0
17+
protocol = "-1"
18+
cidr_blocks = ["0.0.0.0/0"]
19+
}
20+
21+
ingress {
22+
from_port = 2049
23+
to_port = 2049
24+
protocol = "tcp"
25+
security_groups = [module.eks.node_security_group_id]
26+
}
27+
}
28+
29+
resource "aws_efs_mount_target" "eks" {
30+
count = length(local.private_subnets)
31+
32+
file_system_id = resource.aws_efs_file_system.eks.id
33+
subnet_id = element(module.vpc.private_subnets, count.index)
34+
security_groups = [resource.aws_security_group.efs_eks.id]
35+
}
36+
37+
module "efs" {
38+
source = "git::https://github.com/camptocamp/devops-stack-module-efs-csi-driver.git?ref=v2.1.0"
39+
40+
cluster_name = local.cluster_name
41+
argocd_namespace = module.argocd_bootstrap.argocd_namespace
42+
app_autosync = local.app_autosync
43+
44+
efs_file_system_id = resource.aws_efs_file_system.eks.id
45+
create_role = true
46+
cluster_oidc_issuer_url = module.eks.cluster_oidc_issuer_url
47+
48+
dependency_ids = {
49+
argocd = module.argocd_bootstrap.id
50+
}
51+
}
52+
53+
module "ebs" {
54+
source = "git::https://github.com/camptocamp/devops-stack-module-ebs-csi-driver.git?ref=v2.1.0"
55+
56+
cluster_name = local.cluster_name
57+
argocd_namespace = module.argocd_bootstrap.argocd_namespace
58+
app_autosync = local.app_autosync
59+
60+
create_role = true
61+
cluster_oidc_issuer_url = module.eks.cluster_oidc_issuer_url
62+
63+
dependency_ids = {
64+
argocd = module.argocd_bootstrap.id
65+
}
66+
}

‎examples/eks/efs.tf‎

Lines changed: 0 additions & 46 deletions
This file was deleted.

0 commit comments

Comments
 (0)