Complete GitLab Deployment on Kubernetes with GitLab Runner

This guide provides a comprehensive walkthrough for deploying GitLab Community Edition and GitLab Runner on a Kubernetes cluster. We’ll use the GitLab Operator for the main deployment and Helm for additional components like cert-manager and GitLab Runner.

Prerequisites and Setup

Before beginning this deployment, ensure you have:

A functional Kubernetes cluster (this guide assumes you’ve followed Install Kubernetes Cluster on Arch Linux with Cilium)
kubectl configured to access your cluster
Helm 3.x installed and configured
Administrative access to your DNS configuration or /etc/hosts file
A domain name for your GitLab instance (we’ll use ak8s.net as an example)

Architecture Overview

This deployment creates a complete GitLab environment with:

GitLab CE: The main GitLab application with web interface, Git repositories, and CI/CD capabilities
cert-manager: Automatic SSL/TLS certificate management using Let’s Encrypt
metrics-server: Kubernetes metrics collection for monitoring and autoscaling
GitLab Runner: CI/CD job execution engine that runs your pipelines
Ingress Controller: Traffic routing and SSL termination

Step 1: Certificate Manager Deployment

Certificate Manager automates the creation and renewal of SSL certificates for your GitLab instance. This ensures secure HTTPS access to your GitLab deployment.

Installing cert-manager

Add the Jetstack Helm repository and update your local repository cache:

helm repo add jetstack https://charts.jetstack.io
helm repo update

Install cert-manager with its Custom Resource Definitions (CRDs):

helm install cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --create-namespace \
    --set installCRDs=true

This command:

Creates a dedicated cert-manager namespace
Installs all required CRDs for certificate management
Enables automatic certificate provisioning for ingress resources

Verify the installation by checking that all cert-manager pods are running:

kubectl get pods -n cert-manager

Step 2: Metrics Server Installation

The metrics server provides resource usage metrics for pods and nodes, enabling features like horizontal pod autoscaling and the kubectl top command.

Checking Current Installation

First, verify if metrics-server is already installed:

kubectl top node

If this command returns node metrics, you can skip this step. If it returns an error, proceed with the installation.

Installing metrics-server

Add the metrics-server Helm repository:

helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/

Install metrics-server with kubelet TLS verification disabled (common for self-hosted clusters):

helm upgrade --install metrics-server metrics-server/metrics-server \
    -n kube-system \
    --set args="{--kubelet-insecure-tls}"

Note: The --kubelet-insecure-tls flag is often necessary in self-hosted Kubernetes environments where kubelet certificates may not be properly configured for the metrics server.

Verify installation:

kubectl top node

You should now see CPU and memory usage for your cluster nodes.

Step 3: GitLab Operator Deployment

The GitLab Operator manages the lifecycle of GitLab instances on Kubernetes, handling upgrades, configuration changes, and maintaining the desired state.

Installing the GitLab Operator

First, check the latest operator version from the GitLab Operator releases page.

Set your desired operator version and install:

GL_OPERATOR_VERSION=0.26.2
PLATFORM=kubernetes
 
# Create the gitlab-system namespace
kubectl create namespace gitlab-system
 
# Deploy the GitLab Operator
kubectl apply -f https://gitlab.com/api/v4/projects/18899486/packages/generic/gitlab-operator/${GL_OPERATOR_VERSION}/gitlab-operator-${PLATFORM}-${GL_OPERATOR_VERSION}.yaml

Verify the operator is running:

kubectl get pods -n gitlab-system

Step 4: GitLab Instance Deployment

Now we’ll create a GitLab Custom Resource Definition (CRD) that describes our desired GitLab installation.

Creating the GitLab Configuration

Create a GitLab CRD manifest. Important: Replace the chart version with an appropriate one from the GitLab Helm Chart releases.

apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
  name: gitlab
  namespace: gitlab-system
spec:
  chart:
    version: "7.6.2"  # Check for latest version
    values:
      gitlab:
        gitaly:
          persistence:
            size: 20Gi
      minio:
        persistence:
          size: 10Gi
      postgresql:
        persistence:
          size: 8Gi
      prometheus:
        persistence:
          size: 10Gi
      redis:
        master:
          persistence:
            size: 5Gi
      global:
        edition: ce
        registry:
          enabled: false
        # pages:
          # enabled: true
        hosts:
          domain: ak8s.net  # Replace with your domain
        ingress:
          configureCertmanager: true
      certmanager-issuer:
        email: tnd@ak8s.net  # Replace with your email

Save this as gitlab-instance.yaml and apply it:

kubectl apply -f gitlab-instance.yaml

Monitoring Deployment Progress

Monitor the deployment progress by watching the deployments:

watch kubectl -n gitlab-system get deploy

The deployment is complete when gitlab-webservice-default shows at least one ready pod. This process typically takes 10-15 minutes depending on your cluster resources and internet connection.

Accessing Your GitLab Instance

Once deployed, get the ingress IP addresses:

kubectl -n gitlab-system get ingress

Add these IP addresses to your /etc/hosts file:

# Example entries for /etc/hosts
192.168.1.100 gitlab.ak8s.net
192.168.1.100 registry.ak8s.net

Retrieving the Initial Root Password

Get the initial root password for GitLab:

kubectl -n gitlab-system get secret gitlab-gitlab-initial-root-password \
    -o jsonpath="{.data.password}" | base64 -d

You can now access GitLab at https://gitlab.ak8s.net using:

Username: root
Password: (the password retrieved above)

Step 5: GitLab Runner Deployment

GitLab Runner executes your CI/CD jobs. We’ll deploy it using Helm with custom configuration.

Configuring CoreDNS for Internal Resolution

Configure your cluster’s CoreDNS to resolve your GitLab domain internally:

kubectl -n kube-system edit configmap coredns

Add a hosts block to the Corefile configuration:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
        hosts ak8s.net {
          192.168.1.100 gitlab.ak8s.net  # Replace with your actual IP
          fallthrough
        }
    }
# ... rest of the ConfigMap remains unchanged

Restart CoreDNS to apply the configuration:

kubectl -n kube-system rollout restart deployment coredns

Extracting GitLab TLS Certificate

Extract the TLS certificate for GitLab Runner to trust:

kubectl -n gitlab-system get secret gitlab-gitlab-tls \
    -o jsonpath="{.data.tls\.crt}" | base64 -d > gitlab.ak8s.net.crt

Create a Kubernetes secret containing the certificate:

kubectl -n gitlab-system create secret generic gitlab-tls \
    --from-file=gitlab.ak8s.net.crt

Configuring GitLab Runner

Obtaining the Runner Token

Access your GitLab instance at https://gitlab.ak8s.net
Navigate to Admin Area → CI/CD → Runners
Click New instance runner
Configure the runner settings as needed
Copy the runner token (starts with glrt-)

Creating Runner Configuration

Instead of including the entire Helm values file, retrieve and customize it:

# Get the default values for GitLab Runner
helm repo add gitlab https://charts.gitlab.io
helm repo update
helm show values gitlab/gitlab-runner > gitlab-runner-values.yaml

Edit gitlab-runner-values.yaml and modify these key sections:

Essential Configuration Changes:

# GitLab connection
gitlabUrl: https://gitlab.ak8s.net/
runnerToken: "glrt-YOUR-RUNNER-TOKEN-HERE"  # Replace with your actual token
 
# Certificate configuration
certsSecretName: gitlab-tls
 
# Concurrent jobs
concurrent: 10
 
# Kubernetes executor configuration
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "ubuntu:22.04"
        [[runners.kubernetes.volumes.empty_dir]]
          name = "cache"
          mount_path = "/cache"
        [[runners.kubernetes.volumes.secret]]
          name = "gitlab-tls"
          mount_path = "/etc/gitlab-runner/certs/"
 
# RBAC configuration
rbac:
  create: true
  rules:
    - resources: ["configmaps", "pods", "pods/attach", "secrets", "services"]
      verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create", "patch", "delete"]
 
# Security context
securityContext:
  allowPrivilegeEscalation: true
  readOnlyRootFilesystem: false
  runAsNonRoot: true
  privileged: true
  capabilities:
    drop: ["ALL"]
 
podSecurityContext:
  runAsUser: 100
  fsGroup: 65533

Installing GitLab Runner

Deploy GitLab Runner using the customized values:

helm install --namespace gitlab-system gitlab-runner \
    -f gitlab-runner-values.yaml \
    gitlab/gitlab-runner

Verify the runner is connected by checking:

kubectl get pods -n gitlab-system | grep runner

And in your GitLab instance, navigate to Admin Area → CI/CD → Runners to see the connected runner.

Configuration Verification and Testing

Verifying the Complete Setup

Check all deployments are ready:

kubectl get deployments -n gitlab-system
kubectl get deployments -n cert-manager

Verify ingress resources:
```
kubectl get ingress -n gitlab-system
```

Check certificate status:

kubectl get certificates -n gitlab-system

Test GitLab Runner connectivity:

kubectl logs -n gitlab-system deployment/gitlab-runner

Testing CI/CD Pipeline

Create a simple .gitlab-ci.yml in a test repository:

stages:
  - test
 
test-job:
  stage: test
  script:
    - echo "Hello from GitLab Runner on Kubernetes!"
    - kubectl version --client
  tags:
    - kubernetes

This pipeline will verify that your runner can execute jobs and has access to kubectl.

Complete Kubernetes Ecosystem Integration

Your GitLab deployment is now part of a comprehensive Kubernetes ecosystem. To maximize its potential, consider integrating these complementary services:

Monitoring and Observability

Deploy comprehensive monitoring for your GitLab instance using our Prometheus and Grafana monitoring stack. This provides:

GitLab Performance Metrics: Monitor GitLab service health, response times, and resource usage
Kubernetes Cluster Metrics: Track overall cluster health and resource consumption
CI/CD Pipeline Monitoring: Observe runner performance and job execution statistics
Storage and Database Monitoring: Monitor PostgreSQL, Redis, and persistent volume usage

Key benefits for GitLab deployments:

Proactive identification of performance bottlenecks
Historical analysis of resource usage patterns
Alerting for service disruptions or resource exhaustion
Capacity planning for scaling decisions

Container Registry Integration

While this guide disables GitLab’s built-in container registry for simplicity, you can deploy a dedicated private Docker registry to complement your GitLab instance:

Secure Image Storage: Self-hosted container images for your CI/CD pipelines
Faster Image Pulls: Registry deployed within your cluster network
Integration with GitLab CI: Use the private registry in your .gitlab-ci.yml pipelines
Cost Control: Avoid external registry costs for private images

Example GitLab CI integration with private registry:

stages:
  - build
  - deploy
 
build-image:
  stage: build
  script:
    - docker build -t registry.ak8s.net/my-app:$CI_COMMIT_SHA .
    - docker push registry.ak8s.net/my-app:$CI_COMMIT_SHA

Network Observability with Cilium

If you’re using our Cilium-based cluster setup, enhance your GitLab monitoring with network observability:

# Enable Hubble for network monitoring
cilium hubble enable --ui
kubectl --namespace kube-system port-forward svc/hubble-ui 12000:80

This provides detailed insights into:

Network traffic between GitLab components
Service-to-service communication patterns
Network policy enforcement
DNS resolution monitoring

Troubleshooting Common Issues

GitLab Pods Not Starting

Issue: GitLab pods remain in Pending or CrashLoopBackOff state.

Solutions:

Check resource availability: kubectl describe nodes
Verify storage classes: kubectl get storageclass
Check pod events: kubectl describe pod <pod-name> -n gitlab-system

Certificate Issues

Issue: SSL certificates not generating or invalid.

Solutions:

Check cert-manager logs: kubectl logs -n cert-manager deployment/cert-manager
Verify DNS configuration points to your cluster
Ensure your email is valid in the cert-manager configuration
For detailed certificate troubleshooting and verification, use OpenSSL diagnostic commands

Runner Connection Issues

Issue: GitLab Runner cannot connect to GitLab instance.

Solutions:

Verify CoreDNS configuration is correct
Check that the runner token is valid and not expired
Ensure the TLS certificate secret is properly mounted
Verify network policies don’t block communication

Performance Optimization

For production deployments, consider:

Resource Limits: Set appropriate CPU and memory limits for all components
Persistence: Use high-performance storage classes for database workloads
Horizontal Scaling: Configure multiple GitLab Runner replicas for high workload
Monitoring: Deploy comprehensive Prometheus and Grafana monitoring for performance insights
Backup Strategy: Implement regular backups of GitLab data and configurations

Monitoring Integration

Implement comprehensive monitoring to ensure optimal GitLab performance:

Deploy Monitoring Stack: Follow our Prometheus and Grafana deployment guide to set up cluster-wide monitoring
GitLab-Specific Metrics: Configure Prometheus to scrape GitLab metrics endpoints for detailed application monitoring
Custom Dashboards: Create Grafana dashboards specific to GitLab performance, including:
- Git repository activity and storage usage
- CI/CD pipeline execution times and success rates
- Database and Redis performance metrics
- Runner job queue lengths and execution statistics
Alerting Rules: Set up alerts for critical GitLab events such as service downtime, resource exhaustion, or failed pipeline thresholds

Security Considerations

Network Policies: Implement Kubernetes network policies to restrict pod-to-pod communication
RBAC: Follow the principle of least privilege for all service accounts
Secrets Management: Use Kubernetes secrets or external secret management systems
Image Security: Regularly scan container images for vulnerabilities
Update Strategy: Establish a regular update schedule for GitLab and its components

References and Resources

Questions Answered in This Document

Q: How do I deploy GitLab on Kubernetes using the GitLab Operator? A: Install cert-manager and metrics-server first, then deploy the GitLab Operator and create a GitLab custom resource with your desired configuration. The operator will handle the complete GitLab deployment automatically.

Q: What are the prerequisites for deploying GitLab on Kubernetes? A: You need a functional Kubernetes cluster, Helm 3.x, kubectl access, administrative DNS access, a domain name, and sufficient cluster resources (at least 8GB RAM and 4 CPU cores recommended).

Q: How do I configure GitLab Runner to work with a self-hosted GitLab instance? A: Extract the GitLab TLS certificate, create a Kubernetes secret, configure CoreDNS for internal resolution, obtain a runner token from GitLab, and deploy the runner with Helm using custom values that include the certificate mount and connection details.

Q: How do I troubleshoot SSL certificate issues with GitLab on Kubernetes? A: Check cert-manager logs, verify DNS points to your cluster, ensure your email is valid in the cert-manager configuration, and confirm that cert-manager can reach Let’s Encrypt servers for certificate validation.

Q: What storage requirements does GitLab need on Kubernetes? A: GitLab requires persistent storage for GitLab data (20Gi), MinIO object storage (10Gi), PostgreSQL database (8Gi), Prometheus metrics (10Gi), and Redis cache (5Gi). Ensure your cluster has a default storage class configured.

Q: How do I monitor GitLab performance and CI/CD metrics on Kubernetes? A: Deploy our Prometheus and Grafana monitoring stack to collect GitLab metrics, create custom dashboards for CI/CD pipeline performance, and set up alerting for critical events like service downtime or resource exhaustion.

Q: Can I use a private Docker registry with GitLab CI/CD on Kubernetes? A: Yes, deploy a private Docker registry on your cluster and configure GitLab CI/CD pipelines to build and push images to it. Update your runner configuration to include registry credentials and modify .gitlab-ci.yml files to reference the private registry.

Q: How do I scale GitLab Runner for high CI/CD workloads? A: Increase the concurrent jobs setting in the runner configuration, deploy multiple runner replicas using Helm values, and consider using horizontal pod autoscaling based on custom metrics for automatic scaling.

Q: How do I troubleshoot GitLab certificate and TLS issues? A: Check cert-manager logs and certificate status, verify DNS configuration, ensure proper TLS secret creation, and use OpenSSL diagnostic commands to validate certificate properties and trust chain configuration.

Q: How do I backup and restore GitLab data in Kubernetes? A: Use GitLab’s built-in backup tools accessed through kubectl exec, backup persistent volumes using your storage provider’s snapshot features, and maintain separate backups of GitLab configuration and secrets.

Q: What’s the complete Kubernetes ecosystem for GitLab development workflows? A: A complete setup includes GitLab for source control and CI/CD, Prometheus and Grafana for monitoring, private Docker registry for image storage, and network observability tools like Cilium Hubble for comprehensive development and operations visibility.

Flouda Vault

Complete GitLab Deployment on Kubernetes with GitLab Runner

Prerequisites and Setup

Architecture Overview

Step 1: Certificate Manager Deployment

Installing cert-manager

Step 2: Metrics Server Installation

Checking Current Installation

Installing metrics-server

Step 3: GitLab Operator Deployment

Installing the GitLab Operator

Step 4: GitLab Instance Deployment

Creating the GitLab Configuration

Monitoring Deployment Progress

Accessing Your GitLab Instance

Retrieving the Initial Root Password

Step 5: GitLab Runner Deployment

Configuring CoreDNS for Internal Resolution

Extracting GitLab TLS Certificate

Configuring GitLab Runner

Obtaining the Runner Token

Creating Runner Configuration

Installing GitLab Runner

Configuration Verification and Testing

Verifying the Complete Setup

Testing CI/CD Pipeline

Complete Kubernetes Ecosystem Integration

Monitoring and Observability

Container Registry Integration

Network Observability with Cilium

Troubleshooting Common Issues

GitLab Pods Not Starting

Certificate Issues

Runner Connection Issues

Performance Optimization

Monitoring Integration

Security Considerations

References and Resources

Questions Answered in This Document

Graph View

Table of Contents

Backlinks