This repository contains the Terraform scripts responsible for orchestrating the deployment of the MEITREX Platform on a Kubernetes cluster.
We employ Terraform as the central tool for infrastructure-as-code, managing every facet of our Kubernetes resources. At the moment, remote state management is not yet configured, mostly due to the inability to use Terraform Cloud as it's not possible to connect to the Kubernetes cluster from outside the university network. This means that the Terraform state files need to be manually transferred to the individual managing the cluster. This is a point of consideration for future improvements.
Our infrastructure consists of a mixture of "raw" Kubernetes resources and Helm-based deployments. All MEITREX-services are deployed using raw Kubernetes manifests while we use Helm for standard resources like PostgreSQL databases, Keycloak and Minio to simplify management.
Keel serves as our CI tool, automating the update process for our Kubernetes resources. The pull-based approach to CI is particularly advantageous in our setup, as our code and Docker images are hosted externally on GitHub, while the Kubernetes cluster resides within the university network.
All services are exposed through an Nginx ingress that is presumed to already exist in the target cluster. This ingress handles routing and SSL termination, providing a unified access point to various services. Since the ingress of our cluster is not publicly available we route the traffic over the ingress of a different cluster. This second ingress controller handles SSL certificates via Let's Encrypt and is publicly available.
We own the domain meitrex.de, under which the application currently runs. The MINIO service endpoint under minio.meitrex.de and the corresponding dashboard under minio-dashboard.meitrex.de.
- main.tf: Defines providers.
- ingress.tf: Sets up the Nginx ingress for managing external access. Configurations for SSL redirection and proxy buffer sizes are also defined here. All services to be exposed have to be configured here.
- dapr.tf: Deploys the Dapr runtime using Helm charts. Also includes the setup for state and pub-sub components using Redis.
- keel.tf: Manages the deployment of Keel, a tool used for automated Kubernetes deployments, via Helm charts.
- keycloak.tf: Handles the setup for Keycloak, used for identity and access management. It utilizes Helm charts for deployment and includes admin user and password settings.
- prometheus.tf: Deploys the prometheus stack into a separate prometheus namespace to monitor the MEITREX application. The stack contains prometheus itself, the Alert Manager and Grafana.
In frontend.tf, the MEITREX frontend is deployed as a Kubernetes Deployment and exposed through a Kubernetes Service. The deployment specifies environment variables for OAuth and backend URL configurations and includes a liveness probe to monitor the health of the frontend service.
The backend services are generally structured as follows:
- Kubernetes Deployment: Each service is deployed as a Kubernetes Deployment, complete with a Dapr sidecar for http and pub-sub communication.
- Database: A Helm-managed PostgreSQL database is associated with each service, configured with a random password and the default db.
- Optional Resources: Some services include additional resources like Horizontal Pod Autoscalers or Minio deployments for content storage.
While the individual backend deployments might look repetitive, keeping them separate allows us to adapt each service to its specifics and add additional resources like Minio or autoscalers where necessary.
The GraphQL Gateway, configured in gateway.tf, serves as the central entry point for all backend services. It routes incoming HTTP requests to the respective backend services via Dapr. Like other backends, it's deployed as a Kubernetes Deployment and exposed via a Kubernetes Service. An autoscaler and liveness/readiness probes are also configured to ensure scalability and health monitoring.
- Kubernetes Cluster: A running cluster with admin access, a working Nginx ingress controller, the capability to deploy Persistent Volumes and Load Balancers. Place the cluster credentials in a
kubeconfig.yamlfile within the repository. - Terraform CLI: Ensure you have version >= 1.0.11 installed.
- University VPN: If managing the existing cluster, a connection to the university's VPN is required. Additionally if you want to manage the Resources on the GPU PC you need to have a connection to the informatik VPN.
- Terraform State: For managing the existing cluster, obtain and place the current Terraform state within the repository.
variables.tf: Either create a newvariables.tffile or obtain the existing one when managing the existing cluster.
- Clone the Repository: Clone this Terraform repository to your local machine.
- Navigate to the Repo: Open a terminal and navigate to the repository directory.
- Setup: Ensure all prerequisites are met as outlined in the Prerequisites section.
- Initialize Terraform: Run
terraform initto initialize the Terraform workspace. - Apply Configuration: Execute
terraform applyto deploy the resources to your Kubernetes cluster. - Setup keycloak realm: Login into keycloak under
https://meitrex.de/keycloakwith admin credentials. Setup keycloak realm with the configuration in the frontend repository. - Setup webhook for AI Service: Login into the minio instance under
https://minio-dashboard.meitrex.deand add the webhook to all buckets.
For steps 6 & 7 are more detailed guides are available here.
- Disappearing Dapr Sidecars: If Dapr sidecars disappear, causing communication to stop working in the cluster, try restarting the affected deployments.
- Schema Changes in Services: If there are schema changes in individual services without changes in the gateway code, a restart of the gateway deployment is required.
- Setting Environment Variables for a Spring Service: If you want to add a service to this deployment and it is a Spring Boot Service you can set environment variables which are defined in the properties files as follows. Set in the terraform the env example.property as EXAMPLE_PROPERTY. Replace all . with _ and write in capital letters. Spring will automatically set the value.
Hint: For easier management and debugging, it helps to use a Kubernetes management UI like Lens to connect to the cluster, restart deployments or setup port forwarding.