CI/CD — Jenkins

Jenkins remains a widely-used automation server. This guide covers Pipeline-as-Code with Jenkinsfile, shared libraries, and migration strategies to modern platforms based on production experience managing Jenkins fleets for enterprise microservices.

Jenkins Architecture

Jenkins uses a distributed master/agent architecture. The master handles job scheduling, configuration, and UI; agents execute the actual build workloads:

┌─────────────────────────────────────────┐
│           Jenkins Master                │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│  │  Build   │ │  Build   │ │  Build   │ │
│  │  Queue   │ │ Queue    │ │ Queue    │ │
│  └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│       └─────────────┼─────────────┘       │
│                     ▼                     │
│            Scheduler + UI                 │
└─────────────────────┬─────────────────────┘
                      │ Agent connections
┌─────────────────────┼─────────────────────┐
│         ▼           ▼           ▼         │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐  │
│  │ Agent 1  │ │ Agent 2  │ │ Agent 3  │  │
│  │ (Docker) │ │ (EC2)    │ │ (K8s)    │  │
│  │          │ │          │ │ Pod      │  │
│  └──────────┘ └──────────┘ └──────────┘  │
└───────────────────────────────────────────┘

Master Node Responsibilities

• Serve the web UI and REST API
• Schedule builds and assign them to agents
• Store job configurations, build history, and artifacts
• Manage plugin ecosystem
• Authenticate users and authorize actions

Agent Types

Pipeline-as-Code with Jenkinsfile

Jenkins Pipeline enables defining CI/CD pipelines in code, stored in the repository alongside application code. There are two syntax options:

Declarative Pipeline is recommended for 95% of use cases. It provides better validation, a clearer structure, and built-in error handling.

Complete Jenkinsfile Example: Docker-Based Build

This production-grade Jenkinsfile demonstrates a declarative pipeline for a containerized Node.js application with parallel testing, security scanning, and deployment stages:

// Jenkinsfile — Declarative Pipeline for Docker Microservice
pipeline {
    agent {
        kubernetes {
            yaml '''
                apiVersion: v1
                kind: Pod
                spec:
                  containers:
                  - name: node
                    image: node:20-alpine
                    command: ['cat']
                    tty: true
                    resources:
                      requests:
                        memory: "512Mi"
                        cpu: "500m"
                      limits:
                        memory: "2Gi"
                        cpu: "2000m"
                  - name: docker
                    image: docker:24-dind
                    securityContext:
                      privileged: true
                    volumeMounts:
                    - name: docker-graph-storage
                      mountPath: /var/lib/docker
                  - name: trivy
                    image: aquasec/trivy:latest
                    command: ['cat']
                    tty: true
                  volumes:
                  - name: docker-graph-storage
                    emptyDir: {}
            '''
        }
    }

    options {
        buildDiscarder(logRotator(numToKeepStr: '20'))
        timeout(time: 30, unit: 'MINUTES')
        disableConcurrentBuilds()
        timestamps()
        ansiColor('xterm')
    }

    environment {
        APP_NAME = 'payment-service'
        DOCKER_REGISTRY = '123456789012.dkr.ecr.us-east-1.amazonaws.com'
        IMAGE_TAG = "${env.GIT_COMMIT.take(7)}"
        AWS_REGION = 'us-east-1'
        SONARQUBE_URL = 'https://sonarqube.internal.example.com'
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
                script {
                    env.GIT_BRANCH_NAME = env.BRANCH_NAME ?: env.GIT_BRANCH?.replaceAll('origin/', '')
                    echo "Building branch: ${env.GIT_BRANCH_NAME}, commit: ${env.GIT_COMMIT}"
                }
            }
        }

        stage('Install & Lint') {
            steps {
                container('node') {
                    sh '''
                        npm ci --prefer-offline --no-audit
                        npm run lint
                    '''
                }
            }
        }

        stage('Test') {
            parallel {
                stage('Unit Tests') {
                    steps {
                        container('node') {
                            sh '''
                                npm run test:unit -- --coverage --reporters=default --reporters=jest-junit
                            '''
                        }
                    }
                    post {
                        always {
                            junit 'junit.xml'
                            publishHTML([
                                allowMissing: false,
                                alwaysLinkToLastBuild: true,
                                keepAll: true,
                                reportDir: 'coverage/lcov-report',
                                reportFiles: 'index.html',
                                reportName: 'Coverage Report'
                            ])
                        }
                    }
                }

                stage('Integration Tests') {
                    steps {
                        container('node') {
                            sh '''
                                npm run test:integration
                            '''
                        }
                    }
                }

                stage('SonarQube Analysis') {
                    steps {
                        container('node') {
                            withSonarQubeEnv('SonarQube') {
                                sh '''
                                    npx sonarqube-scanner \
                                        -Dsonar.projectKey=${APP_NAME} \
                                        -Dsonar.sources=src \
                                        -Dsonar.tests=tests \
                                        -Dsonar.javascript.lcov.reportPaths=coverage/lcov.info \
                                        -Dsonar.qualitygate.wait=true
                                '''
                            }
                        }
                    }
                }
            }
        }

        stage('Build & Push Image') {
            when {
                anyOf {
                    branch 'main'
                    branch 'release/*'
                    tag 'v*'
                }
            }
            steps {
                container('docker') {
                    script {
                        // Login to ECR
                        sh '''
                            apk add --no-cache aws-cli
                            aws ecr get-login-password --region ${AWS_REGION} | \
                                docker login --username AWS --password-stdin ${DOCKER_REGISTRY}
                        '''

                        // Build image
                        sh """
                            docker build \
                                --build-arg NODE_ENV=production \
                                --build-arg BUILD_DATE=\$(date -u +%Y-%m-%dT%H:%M:%SZ) \
                                --build-arg VCS_REF=${env.GIT_COMMIT} \
                                -t ${DOCKER_REGISTRY}/${APP_NAME}:${IMAGE_TAG} \
                                -t ${DOCKER_REGISTRY}/${APP_NAME}:latest \
                                .
                        """

                        // Push image
                        sh """
                            docker push ${DOCKER_REGISTRY}/${APP_NAME}:${IMAGE_TAG}
                            docker push ${DOCKER_REGISTRY}/${APP_NAME}:latest
                        """
                    }
                }
            }
        }

        stage('Security Scan') {
            when {
                anyOf {
                    branch 'main'
                    branch 'release/*'
                }
            }
            steps {
                container('trivy') {
                    sh """
                        trivy image \
                            --severity HIGH,CRITICAL \
                            --exit-code 1 \
                            --no-progress \
                            ${DOCKER_REGISTRY}/${APP_NAME}:${IMAGE_TAG}
                    """
                }
            }
        }

        stage('Deploy to Staging') {
            when {
                branch 'main'
            }
            steps {
                container('node') {
                    sh '''
                        npm run deploy:staging -- --image-tag=${IMAGE_TAG}
                    '''
                }
            }
            post {
                success {
                    slackSend(
                        color: 'good',
                        message: "${APP_NAME} deployed to staging: ${DOCKER_REGISTRY}/${APP_NAME}:${IMAGE_TAG}"
                    )
                }
                failure {
                    slackSend(
                        color: 'danger',
                        message: "${APP_NAME} staging deployment FAILED"
                    )
                }
            }
        }

        stage('Deploy to Production') {
            when {
                tag 'v*'
            }
            steps {
                timeout(time: 10, unit: 'MINUTES') {
                    input message: 'Deploy to Production?', ok: 'Deploy',
                          submitterParameter: 'APPROVER'
                }
                container('node') {
                    script {
                        def version = env.TAG_NAME ?: env.GIT_BRANCH?.replaceAll('origin/', '')
                        sh """
                            npm run deploy:prod -- --image-tag=${IMAGE_TAG} --version=${version}
                        """
                    }
                }
            }
            post {
                success {
                    slackSend(
                        color: 'good',
                        message: "${APP_NAME} ${env.TAG_NAME} deployed to PRODUCTION by ${env.APPROVER}"
                    )
                }
            }
        }
    }

    post {
        always {
            script {
                // Cleanup
                cleanWs(
                    deleteDirs: true,
                    notFailBuild: true,
                    patterns: [[pattern: '.git', type: 'EXCLUDE']]
                )
            }
        }
        failure {
            slackSend(
                color: 'danger',
                message: "${APP_NAME} build FAILED on branch ${env.GIT_BRANCH_NAME}: ${env.BUILD_URL}"
            )
        }
        fixed {
            slackSend(
                color: 'good',
                message: "${APP_NAME} build FIXED on branch ${env.GIT_BRANCH_NAME}"
            )
        }
    }
}

Shared Libraries

Jenkins Shared Libraries enable centralizing pipeline logic that can be reused across multiple pipelines. This is essential for standardizing CI/CD patterns across an organization.

Directory Structure

jenkins-shared-libraries/
├── vars/
│   ├── standardBuild.groovy       # Main pipeline template
│   ├── dockerBuild.groovy         # Docker helper steps
│   ├── notifySlack.groovy         # Notification helpers
│   └── deployHelm.groovy          # Deployment helpers
├── src/
│   └── org/mycompany/
│       └── Utils.groovy           # Groovy utility classes
├── resources/
│   ├── checkstyle.xml             # Static configuration files
│   └── helm-values-template.yaml
├── test/
│   └── vars/
│       └── StandardBuildTest.groovy
└── pom.xml                         # For testing with JUnit

Standard Pipeline (vars/standardBuild.groovy)

// vars/standardBuild.groovy
def call(Map config = [:]) {
    // Default configuration
    def defaultConfig = [
        appName: 'app',
        nodeVersion: '20',
        runUnitTests: true,
        runIntegrationTests: true,
        runSecurityScan: true,
        dockerRegistry: '123456789012.dkr.ecr.us-east-1.amazonaws.com',
        awsRegion: 'us-east-1',
        deployBranches: ['main'],
        stagingBranch: 'main'
    ]

    // Merge with user config
    config = defaultConfig + config

    pipeline {
        agent {
            kubernetes {
                yaml libraryResource('pod-templates/standard-agent.yaml')
            }
        }

        options {
            buildDiscarder(logRotator(numToKeepStr: '20'))
            timeout(time: 30, unit: 'MINUTES')
            disableConcurrentBuilds()
        }

        environment {
            APP_NAME = config.appName
            IMAGE_TAG = "${env.GIT_COMMIT.take(7)}"
        }

        stages {
            stage('Setup') {
                steps {
                    script {
                        echo "Building ${config.appName} with Node ${config.nodeVersion}"
                        checkout scm
                    }
                }
            }

            stage('Install Dependencies') {
                steps {
                    container('node') {
                        sh 'npm ci --prefer-offline'
                    }
                }
            }

            stage('Lint') {
                steps {
                    container('node') {
                        sh 'npm run lint'
                    }
                }
            }

            stage('Unit Tests') {
                when { expression { config.runUnitTests } }
                steps {
                    container('node') {
                        sh 'npm run test:unit -- --coverage'
                    }
                }
                post {
                    always {
                        junit 'junit.xml'
                    }
                }
            }

            stage('Build & Push') {
                when {
                    anyOf {
                        branch pattern: config.deployBranches.join('|'), comparator: 'REGEXP'
                        tag 'v*'
                    }
                }
                steps {
                    script {
                        dockerBuild(
                            imageName: "${config.dockerRegistry}/${config.appName}",
                            tag: env.IMAGE_TAG,
                            push: true
                        )
                    }
                }
            }

            stage('Deploy') {
                when {
                    branch config.stagingBranch
                }
                steps {
                    script {
                        deployHelm(
                            appName: config.appName,
                            imageTag: env.IMAGE_TAG,
                            environment: 'staging'
                        )
                    }
                }
            }
        }

        post {
            failure {
                notifySlack(status: 'FAILED', appName: config.appName)
            }
            fixed {
                notifySlack(status: 'FIXED', appName: config.appName)
            }
        }
    }
}

Using the Shared Library

// Jenkinsfile in any service repository
@Library('my-shared-library@main') _

standardBuild(
    appName: 'payment-service',
    nodeVersion: '20',
    runIntegrationTests: true,
    deployBranches: ['main', 'develop']
)

Multibranch Pipelines

Multibranch Pipeline projects automatically create pipelines for branches and pull requests in a repository. This is the recommended way to organize Jenkins pipelines for GitHub and Bitbucket repositories:

1. In Jenkins, create a New Item of type Multibranch Pipeline
2. Under Branch Sources, add GitHub or Bitbucket
3. Configure credentials and repository URL
4. Set build configuration mode to by Jenkinsfile
5. Configure branch discovery: discover all branches + PRs
6. Add orphaned item strategy to keep only last 20 builds
7. Save — Jenkins will scan the repository and create pipelines per branch

Jenkinsfile with Branch Logic

pipeline {
    agent any
    stages {
        stage('Build') {
            steps { sh 'make build' }
        }
        stage('Test') {
            steps { sh 'make test' }
        }
        stage('Deploy Staging') {
            when { branch 'main' }
            steps {
                sh 'make deploy-staging'
            }
        }
        stage('Deploy Production') {
            when { buildingTag() }
            steps {
                input message: 'Deploy to Production?'
                sh 'make deploy-production'
            }
        }
    }
}

Blue/Green Deployment with Jenkins

stage('Blue/Green Deploy') {
    steps {
        script {
            def currentColor = sh(
                script: "kubectl get svc ${APP_NAME} -o jsonpath='{.spec.selector.version}'",
                returnStdout: true
            ).trim()

            def newColor = currentColor == 'blue' ? 'green' : 'blue'

            echo "Current: ${currentColor}, Deploying to: ${newColor}"

            // Deploy to inactive environment
            sh """
                sed -e 's/__COLOR__/${newColor}/g' \
                    -e 's/__TAG__/${IMAGE_TAG}/g' \
                    k8s-deployment.yaml | kubectl apply -f -
            """

            // Wait for rollout
            sh "kubectl rollout status deployment/${APP_NAME}-${newColor} --timeout=300s"

            // Health check
            def healthCheck = sh(
                script: "kubectl exec deploy/${APP_NAME}-${newColor} -- wget -qO- http://localhost:8080/health",
                returnStatus: true
            )

            if (healthCheck == 0) {
                // Switch service selector
                sh "kubectl patch svc ${APP_NAME} -p '{\"spec\":{\"selector\":{\"version\":\"${newColor}\"}}}'"

                // Scale down old environment after cooldown
                sh "kubectl scale deployment/${APP_NAME}-${currentColor} --replicas=0"
            } else {
                error("Health check failed for ${newColor}. Keeping ${currentColor} active.")
            }
        }
    }
}

SonarQube Integration

// Install SonarQube Scanner plugin in Jenkins
// Configure SonarQube server in Manage Jenkins > Configure System

stage('SonarQube Analysis') {
    steps {
        withSonarQubeEnv('SonarQube-Production') {
            sh """
                npx sonarqube-scanner \
                    -Dsonar.projectKey=${APP_NAME} \
                    -Dsonar.projectName="${APP_NAME}" \
                    -Dsonar.projectVersion=${BUILD_NUMBER} \
                    -Dsonar.sources=src \
                    -Dsonar.tests=__tests__ \
                    -Dsonar.javascript.lcov.reportPaths=coverage/lcov.info \
                    -Dsonar.coverage.exclusions="**/*.test.js,**/*.spec.js" \
                    -Dsonar.qualitygate.wait=true
            """
        }
    }
}

// For Quality Gate webhook (required for wait to work):
// In SonarQube: Administration > Configuration > Webhooks > Add Jenkins URL
// URL: https://jenkins.example.com/sonarqube-webhook/

Migration Checklist: Jenkins to GitHub Actions

Based on my experience migrating 50+ microservices from Jenkins to GitHub Actions, the following checklist ensures a smooth transition:

Jenkins to GitHub Actions Concept Mapping

┌─────────────────────────────┬──────────────────────────────────────┐
│ Jenkins Concept             │ GitHub Actions Equivalent            │
├─────────────────────────────┼──────────────────────────────────────┤
│ Jenkinsfile                 │ .github/workflows/*.yml              │
│ Declarative Pipeline        │ Workflow YAML structure              │
│ Stages                      │ jobs:                                │
│ Steps                       │ steps:                               │
│ Shared Library (vars/)      │ Reusable workflows (workflow_call)   │
│ Shared Library (src/)       │ Composite actions                    │
│ credentials()               │ secrets.GH_SECRET                    │
│ withCredentials             │ env: or with: secrets:               │
│ input                       │ workflow_dispatch inputs             │
│ build job:                  │ jobs..needs:                    │
│ parallel stages             │ jobs without needs (parallel)        │
│ when conditions             │ if: conditions on jobs/steps         │
│ post actions                │ jobs with if: failure()/success()    │
│ node/agent label            │ runs-on:                             │
│ Kubernetes plugin           │ Container jobs or self-hosted        │
│ Blue Ocean                  │ GitHub Actions built-in UI           │
│ Job parameters              │ workflow_dispatch inputs             │
└─────────────────────────────┴──────────────────────────────────────┘

Related Topics

• CI/CD — GitHub Actions
• CI/CD — CircleCI
• Terraform Basics
• Kubernetes Operations