Skip to content

Commit 4d72cc8

Browse files
lentidaslentidas
committed
feat: add AKS example and tutorial
1 parent 339544f commit 4d72cc8

14 files changed

Lines changed: 870 additions & 1 deletion

File tree

‎docs/modules/ROOT/pages/tutorials/deploy_aks.adoc‎

Lines changed: 185 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,187 @@
11
= Deployment On Azure AKS
22

3-
_Work In Progress_
3+
An example of a local deployment of a Kubernetes cluster on Azure AKS is provided https://github.com/camptocamp/devops-stack/tree/main/examples/aks[here]. Clone this repository and modify the files at your convenience.
4+
In the repository, as in a standard https://developer.hashicorp.com/terraform/tutorials/modules/module#what-is-a-terraform-module[Terraform module], you will find the following files:
5+
6+
* *`terraform.tf`* - declaration of the Terraform providers used in this project;
7+
* *`locals.tf`* - local variables used by the DevOps Stack modules;
8+
* *`main.tf`* - definition of all the deployed modules;
9+
* *`storage.tf`* - creation of the Storage Account and Storage Container used by Loki and Thanos;
10+
* *`dns.tf`* - creation of the wildcard record for the ingresses of the DevOps Stack components;
11+
* *`oidc.tf`* - addition of the redirect URIs to the Azure AD Enterprise Application in order to use it to authenticate to the DevOps Stack components providing a web interface;
12+
* *`outputs.tf`* - the output variables of the DevOps Stack;
13+
14+
IMPORTANT: The *`requirements`* folder is not part of the Terraform code you execute directly. Its importance is explained on the next section.
15+
16+
== Requirements
17+
18+
On your local machine, you need to have the following tools installed:
19+
20+
* https://learn.microsoft.com/en-us/cli/azure/[Azure CLI] to login to your Azure account and interact with your AKS cluster;
21+
* https://www.terraform.io/[Terraform] to provision the whole stack;
22+
* https://kubernetes.io/docs/reference/kubectl/[`kubectl`] or https://github.com/derailed/k9s[`k9s`]to interact with your cluster;
23+
24+
Other than that, you will require the following:
25+
26+
* An active Azure account with an active subscription;
27+
* An Enterprise Application on Entra ID to use as an identity provider for the DevOps Stack components;
28+
* The Azure subscription needs to have a Key Vault to store the secrets used to pass the credentials of said application to the DevOps Stack components;
29+
* Your Azure account needs to be part of a user group that has been assigned the role `Owner`, `Key Vault Reader` and `Key Vault Secrets User` on the subscription;
30+
* Your Azure account also needs to be an `Owner` of the Enterprise Application in order to add the proper redirect URIs.
31+
32+
[TIP]
33+
====
34+
In this repository, you will find an example of Terraform code that could provision the required resources above. You can find this code https://github.com/camptocamp/devops-stack/tree/main/examples/aks/requirements[here].
35+
36+
*Note that this code needs to be executed by an administrator with the proper rights on the on the subscription but also on Entra ID.*
37+
====
38+
39+
[TIP]
40+
====
41+
An alternative to creating the required resources separately is that your user has an `Application Developer` role assignment on the Entra ID instance the subscription is linked to.
42+
43+
This will allow you to create the Enterprise Application and add the redirect URIs directly with your code, without the need of an administrator.
44+
45+
Check the https://github.com/camptocamp/devops-stack/tree/main/examples/aks/requirements/application.tf[`application.tf`] from the tip above and adapt the Terraform resources in order to create the application yourself.
46+
47+
Or simply create the Enterprise Application and add the redirect URIs manually.
48+
====
49+
50+
== Specificities and explanations
51+
52+
=== Remote Terraform state
53+
54+
If you do not want to configure the remote Terraform state backend, you can simply remove the `backend` block from the `terraform.tf` file.
55+
56+
NOTE: More information about the remote backends is available on the https://developer.hashicorp.com/terraform/language/settings/backends/configuration[official documentation].
57+
58+
=== OIDC authentication
59+
60+
IMPORTANT: The DevOps Stack modules are developed with OIDC in mind. In production, you should have an identity provider that supports OIDC and use it to authenticate to the DevOps Stack applications.
61+
62+
In this example, we use an https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/[Enterprise Applicaion] as OIDC provider.
63+
64+
You can use any other OIDC provider by adapting the `oidc` block in the `locals.tf` file with the proper values.
65+
66+
=== Let's Encrypt SSL certificates
67+
68+
By default, to avoid rate-limiting your domain by Let's Encrypt, the example uses the `letsencrypt-staging` configuration of the cert-manager module to generate certificates. This uses the Let's Encrypt staging environment which has an invalid CA certificate.
69+
70+
If you feel ready to test with production certificates, you can simply edit the `locals.tf` file and change the `cluster_issuer` variable to `letsencrypt-prod`.
71+
72+
== Deployment
73+
74+
1. Clone the repository and `cd` into the `examples/aks` folder;
75+
76+
2. Login to your Azure account with the Azure CLI, set the proper subscription and verify you are connected it:
77+
+
78+
[source,bash]
79+
----
80+
az login
81+
az account set --subscription <subscription_id>
82+
az account show
83+
----
84+
85+
3. Check out the modules you want to deploy in the `main.tf` file, and comment out the others;
86+
+
87+
TIP: You can also add your own Terraform modules in this file or any other file on the root folder. A good place to start to write your own module is to clone the https://github.com/camptocamp/devops-stack-module-template[devops-stack-module-template] repository and adapt it to your needs.
88+
89+
4. From the source of the example deployment, initialize the Terraform modules and providers:
90+
+
91+
[source,bash]
92+
----
93+
terraform init
94+
----
95+
96+
5. Configure the variables in `locals.tf` to your preference:
97+
TIP: The xref:eks:ROOT:README.adoc[cluster module documentation] can help you know what to put in the `kubernetes_version`, for example.
98+
+
99+
[source,terraform]
100+
----
101+
include::example$deploy_examples/aks/locals.tf[]
102+
----
103+
104+
6. Finally, run `terraform apply` and accept the proposed changes to create the Kubernetes nodes on Azure AKS and populate them with our services;
105+
+
106+
[source,bash]
107+
----
108+
terraform apply
109+
----
110+
111+
7. After the first deployment (please note the troubleshooting step related with Argo CD), you can go to the `locals.tf` and enable the _ServiceMonitor_ boolean to activate the Prometheus exporters that will send metrics to Prometheus;
112+
+
113+
IMPORTANT: This flag needs to be set as `false` for the first bootstrap of the cluster, otherwise the applications will fail to deploy while the Custom Resource Definitions of the kube-prometheus-stack are not yet created.
114+
+
115+
NOTE: You can either set the flag as `true` in the `locals.tf` file or you can simply delete the line on the modules' declarations, since this variable is set as `true` by default on each module.
116+
+
117+
TIP: Take note of the local called `app_autosync`. If you set the condition of the ternary operator to `false` you will disable the auto-sync for all the DevOps Stack modules. This allows you to choose when to manually sync the module on the Argo CD interface and is useful for troubleshooting purposes.
118+
119+
== Access the cluster and the DevOps Stack applications
120+
121+
To access your cluster, you need to use the Azure CLI to recover a Kubeconfig you can use:
122+
123+
[source,bash]
124+
----
125+
az aks get-credentials --resource-group YOUR_CLUSTER_RESOURCE_GROUP_NAME --name YOUR_CLUSTER_NAME --file ~/.kube/NAME_TO_GIVE_YOUR_CONFIG.config
126+
----
127+
128+
[NOTE]
129+
====
130+
If you do not add your user's or group's object ID to the `rbac_aad_admin_group_object_ids` variable on the `main.tf`, you will need to use the `--admin` flag on the command above. This will give the privileged Kubeconfig to access the cluster.
131+
====
132+
133+
Then you can use the `kubectl` or `k9s` command to interact with the cluster:
134+
135+
[source,bash]
136+
----
137+
k9s --kubeconfig ~/.kube/NAME_TO_GIVE_YOUR_CONFIG.config
138+
----
139+
140+
As for the DevOps Stack applications, you can access them through the ingress domain that you can find in the `ingress_domain` output. If you used the code from the example without modifying the outputs, you will see something like this on your terminal after the `terraform apply` has done its job:
141+
142+
[source,shell]
143+
----
144+
Outputs:
145+
146+
ingress_domain = "your.domain.here"
147+
----
148+
149+
Or you can use `kubectl` to get all the ingresses and their respective URLs:
150+
151+
[source,bash]
152+
----
153+
kubectl get ingress --all-namespaces --kubeconfig ~/.kube/NAME_TO_GIVE_YOUR_CONFIG.config
154+
----
155+
156+
== Stop the cluster
157+
158+
To definitively stop the cluster on a single command, you can simply use the `terraform destroy` command. This will destroy all the resources created by the Terraform code, including the Kubernetes cluster.
159+
160+
== Troubleshooting
161+
162+
=== `connection_error` during the first deployment
163+
164+
In some cases, you could encounter an error like these the first deployment:
165+
166+
[source,shell]
167+
----
168+
╷
169+
│ Error: error while waiting for application argocd to be created
170+
│
171+
│ with module.argocd.argocd_application.this,
172+
│ on .terraform/modules/argocd/main.tf line 55, in resource "argocd_application" "this":
173+
│ 55: resource "argocd_application" "this" {
174+
│
175+
│ error while waiting for application argocd to be synced and healthy: rpc error: code = Unavailable desc = error reading from server: EOF
176+
╵
177+
----
178+
179+
The error is due to the way we provision Argo CD on the final steps of the deployment. We use the bootstrap Argo CD to deploy the final Argo CD module, which causes a redeployment of Argo CD and consequently a momentary loss of connection between the Argo CD Terraform provider and the Argo CD server.
180+
181+
*You can simply re-run the command `terraform apply` to finalize the bootstrap of the cluster every time you encounter this error.*
182+
183+
=== Argo CD interface reload loop when clicking on login
184+
185+
If you encounter a loop when clicking on the login button on the Argo CD interface, you can try to delete the Argo CD server pod and let it be recreated.
186+
187+
TIP: For more informations about the Argo CD module, please refer to the xref:argocd:ROOT:README.adoc[respective documentation page].

‎examples/aks/.terraform-version‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
1.6.4

‎examples/aks/apps.tf‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
module "helloworld_apps" {
2+
source = "git::https://github.com/camptocamp/devops-stack-module-applicationset.git?ref=v3.0.0"
3+
4+
dependency_ids = {
5+
argocd = module.argocd.id
6+
}
7+
8+
name = "helloworld-apps"
9+
project_dest_namespace = "*"
10+
project_source_repo = "https://github.com/camptocamp/devops-stack-helloworld-templates.git"
11+
12+
app_autosync = local.app_autosync
13+
14+
generators = [
15+
{
16+
git = {
17+
repoURL = "https://github.com/camptocamp/devops-stack-helloworld-templates.git"
18+
revision = "main"
19+
20+
directories = [
21+
{
22+
path = "apps/*"
23+
}
24+
]
25+
}
26+
}
27+
]
28+
template = {
29+
metadata = {
30+
name = "{{path.basename}}"
31+
}
32+
33+
spec = {
34+
project = "helloworld-apps"
35+
36+
source = {
37+
repoURL = "https://github.com/camptocamp/devops-stack-helloworld-templates.git"
38+
targetRevision = "main"
39+
path = "{{path}}"
40+
41+
helm = {
42+
valueFiles = []
43+
# The following value defines this global variables that will be available to all apps in apps/*
44+
# These are needed to generate the ingresses containing the name and base domain of the cluster.
45+
values = <<-EOT
46+
cluster:
47+
name: "${local.cluster_name}"
48+
domain: "${local.base_domain}"
49+
issuer: "${local.cluster_issuer}"
50+
apps:
51+
traefik_dashboard: false
52+
grafana: true
53+
prometheus: true
54+
thanos: true
55+
alertmanager: true
56+
EOT
57+
}
58+
}
59+
60+
destination = {
61+
name = "in-cluster"
62+
namespace = "{{path.basename}}"
63+
}
64+
65+
syncPolicy = {
66+
automated = {
67+
allowEmpty = false
68+
selfHeal = true
69+
prune = true
70+
}
71+
syncOptions = [
72+
"CreateNamespace=true"
73+
]
74+
}
75+
}
76+
}
77+
}

‎examples/aks/dns.tf‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Use these resources if you are creating the DNS zone using this code.
2+
3+
resource "azurerm_dns_zone" "this" {
4+
name = local.base_domain
5+
resource_group_name = resource.azurerm_resource_group.main.name
6+
}
7+
8+
# This resource should be deactivated if there are multiple development clusters on the same account.
9+
resource "azurerm_dns_cname_record" "wildcard" {
10+
count = local.activate_wildcard_record ? 1 : 0
11+
12+
zone_name = resource.azurerm_dns_zone.this.name
13+
name = "*.apps"
14+
resource_group_name = "default"
15+
ttl = 300
16+
record = format("%s-%s.%s.cloudapp.azure.com.", module.aks.cluster_name, replace(resource.azurerm_dns_zone.this.name, ".", "-"), resource.azurerm_resource_group.main.location)
17+
}
18+
19+
# Else use these resources if you are using an existing DNS zone.
20+
21+
# data "azurerm_dns_zone" "this" {
22+
# name = local.base_domain
23+
# resource_group_name = "default"
24+
# }
25+
26+
# # This resource should be deactivated if there are multiple development clusters on the same account.
27+
# resource "azurerm_dns_cname_record" "wildcard" {
28+
# count = local.activate_wildcard_record ? 1 : 0
29+
30+
# zone_name = data.azurerm_dns_zone.this.name
31+
# name = "*.apps"
32+
# resource_group_name = "default"
33+
# ttl = 300
34+
# record = format("%s-%s.%s.cloudapp.azure.com.", module.aks.cluster_name, replace(data.azurerm_dns_zone.this.name, ".", "-"), resource.azurerm_resource_group.main.location)
35+
# }

‎examples/aks/locals.tf‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
locals {
2+
# Parameters for the resources that are created outside this code, but still on the Azure subscription where the DevOps Stack will be deployed.
3+
default_resource_group = "YOUR_DEFAULT_RESOURCE_GROUP" # The default resource group where the Key Vault with the Azure AD application credentials is located.
4+
default_key_vault = "YOUR_KEY_VAULT_NAME" # The name of the Key Vault with the Azure AD application credentials.
5+
oidc_application_name = "YOUR_APPLICATION_NAME" # The name of the Azure AD application that will be used for OIDC authentication.
6+
7+
# Parameters used for this deployment of the DevOps Stack.
8+
common_resource_group = "YOUR_COMMON_RESOURCE_GROUP" # The resource group where the common resources will reside. Must be unique for each DevOps Stack deployment in a single Azure subscription.
9+
location = "YOUR_LOCATION"
10+
kubernetes_version = "1.28"
11+
sku_tier = "Standard"
12+
cluster_name = "YOUR_CLUSTER_NAME" # Must be unique for each DevOps Stack deployment in a single Azure subscription.
13+
base_domain = "your.domain.here" # Must match a DNS zone in the Azure subscription where you are deploying the DevOps Stack.
14+
activate_wildcard_record = true
15+
cluster_issuer = module.cert-manager.cluster_issuers.staging
16+
letsencrypt_issuer_email = "YOUR_EMAIL_ADDRESS"
17+
enable_service_monitor = false # Can be enabled after the first bootstrap.
18+
app_autosync = true ? { allow_empty = false, prune = true, self_heal = true } : {}
19+
20+
# The virtual network CIDR must be unique for each DevOps Stack deployment in a single Azure subscription.
21+
virtual_network_cidr = "10.1.0.0/16"
22+
23+
# Automatic subnets IP range calculation, splitting the virtual_network_cidr above into 6 subnets.
24+
cluster_subnet = cidrsubnet(local.virtual_network_cidr, 8, 0)
25+
26+
# Local containing all the OIDC definitions required by the DevOps Stack modules.
27+
oidc = {
28+
issuer_url = format("https://login.microsoftonline.com/%s/v2.0", data.azuread_client_config.current.tenant_id)
29+
oauth_url = format("https://login.microsoftonline.com/%s/oauth2/authorize", data.azuread_client_config.current.tenant_id)
30+
token_url = format("https://login.microsoftonline.com/%s/oauth2/token", data.azuread_client_config.current.tenant_id)
31+
api_url = format("https://graph.microsoft.com/oidc/userinfo")
32+
client_id = data.azurerm_key_vault_secret.aad_application_client_id.value
33+
client_secret = data.azurerm_key_vault_secret.aad_application_client_secret.value
34+
oauth2_proxy_extra_args = local.cluster_issuer != "letsencrypt-prod" ? [
35+
"--insecure-oidc-skip-issuer-verification=true",
36+
"--ssl-insecure-skip-verify=true",
37+
] : []
38+
}
39+
}

0 commit comments

Comments
 (0)