End-to-End DevSecOps CI/CD Pipeline

From GitHub Commit to Production Kubernetes Deployment with Automated Shift-Left Security Scans & GitOps

🚀 DevSecOps Architecture Series | Cloud DevOps Hub

📌 Overview: Why Modern DevOps Requires DevSecOps

Building a modern software delivery pipeline is not just about moving code from a developer's laptop to production. True DevSecOps embeds automated security gates, code quality analysis, vulnerability scanning, and GitOps synchronization directly into every build step.

Here is the architectural workflow for an End-to-End DevSecOps CI/CD Pipeline:

End to End DevSecOps CI CD Pipeline Architecture Diagram
Figure 1: End-to-End DevSecOps Pipeline Flow from Source Commit to Kubernetes Production
💡 Pipeline Mantra: Automation First • Shift-Left Security • GitOps Deployment • Continuous Delivery.

🔄 Stage-by-Stage Pipeline Execution Flow

Below is an in-depth, step-by-step architectural breakdown of all 13 automated stages in this enterprise DevSecOps pipeline—explaining the tools used, CLI commands executed, quality gate failure criteria, and why each stage matters in production.

Stage 01

👨‍💻 1. Developer Code Commit & Local Hooks

What Happens: A developer writes application feature code and unit test suites on a local feature branch (e.g. feature/auth-service). Before committing, local Git pre-commit hooks (using Husky or Gitleaks) automatically scan the local workspace for accidentally hardcoded credentials, secret keys, or AWS access tokens.

Tools Used: Git, Gitleaks, Husky, IDE linters.

Developer Terminal Commands
# Check workspace status and commit code
git checkout -b feature/auth-service
git add .
git commit -m "feat: implement JWT authentication endpoint with unit tests"
Stage 02

🐙 2. GitHub Push & Webhook Event Trigger

What Happens: The developer pushes the committed code branch to the centralized GitHub repository or opens a Pull Request (PR). GitHub receives the push event and fires an automated HTTP POST Webhook payload containing repository metadata, commit author details, and the unique Git commit SHA (e.g. a1b2c3d) to Jenkins.

Tools Used: GitHub, Git Webhooks.

Push to Remote Repository
git push origin feature/auth-service
Stage 03

🤵 3. Jenkins Pipeline Job Initialization

What Happens: The Jenkins Automation Server catches the incoming GitHub webhook payload. Jenkins provisions an isolated, clean build environment (such as an ephemeral Kubernetes pod agent or Docker worker node), checks out the exact Git commit SHA, and parses the project's Jenkinsfile pipeline definition.

Tools Used: Jenkins, Jenkins Kubernetes Plugin, Declarative Pipeline.

Jenkinsfile Checkout Stage
stage('Checkout Source Code') {
    steps {
        checkout scm
        echo "Building Commit SHA: ${GIT_COMMIT}"
    }
}
Stage 04

⚙️ 4. Application Build & Unit Testing

What Happens: Jenkins compiles the application source code into executable binaries and runs the automated unit test suite. If any unit test fails or compilation errors occur, the pipeline halts immediately, marking the build status as FAILED and notifying the team via Slack/Email before any further security stages run.

Tools Used: Maven, Gradle, npm, Go build, JUnit, Jest.

Jenkins Build & Test Execution
# Example Maven build & unit test command
mvn clean test -DskipTests=false
Stage 05

📊 5. SonarQube Code Quality & Security Analysis (SAST)

What Happens: Static Application Security Testing (SAST). SonarQube scans raw source code line-by-line to detect code smells, potential logic bugs, security hotspots (e.g. unhandled null pointers, SQL injection risks), and unit test coverage percentage.

Quality Gate Criteria: The pipeline blocks progress if code coverage drops below 80% or if any CRITICAL or BLOCKER security vulnerability is identified.

SonarQube Scanner Execution & Quality Gate Check
stage('SonarQube SAST Scan') {
    steps {
        withSonarQubeEnv('SonarQubeServer') {
            sh 'mvn sonar:sonar -Dsonar.projectKey=my-devsecops-app'
        }
        // Fail pipeline if Quality Gate fails
        timeout(time: 5, unit: 'MINUTES') {
            script {
                def qg = waitForQualityGate()
                if (qg.status != 'OK') {
                    error "Pipeline aborted due to SonarQube Quality Gate failure: ${qg.status}"
                }
            }
        }
    }
}
Stage 06

🛡️ 6. OWASP Dependency-Check (SCA)

What Happens: Software Composition Analysis (SCA). OWASP Dependency-Check scans all third-party open-source libraries and packages (e.g. pom.xml, package.json, requirements.txt) against the National Vulnerability Database (NVD) to identify known Common Vulnerabilities and Exposures (CVEs).

Failure Threshold: If any imported open-source dependency has a CVSS score >= 7.0 (High/Critical vulnerability), OWASP Dependency-Check aborts the build.

OWASP Dependency Scan Command
dependency-check.sh --project "DevSecOps-App" \
                     --scan ./ \
                     --format "ALL" \
                     --failOnCVSS 7
Stage 07

🐳 7. Docker Image Build & Immutable Tagging

What Happens: Once source code quality and third-party libraries pass inspection, Docker Engine packages the verified application binary and runtime environment into an immutable container image. The image is tagged with the exact Git commit SHA (e.g. my-repo/my-app:v1.0.0-a1b2c3d) to guarantee complete traceability.

Tools Used: Docker Engine, Multi-stage Dockerfiles.

Docker Container Build
docker build -t 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:${GIT_COMMIT} .
Stage 08

🔍 8. Aqua Trivy Container Vulnerability Scan

What Happens: Container Image Vulnerability Scanning. Aqua Trivy scans the built Docker image layer-by-layer, checking the base operating system packages (e.g. Alpine/Debian/Ubuntu OS packages) and runtime binaries for unpatched CVEs and container misconfigurations.

Failure Threshold: Configured with --exit-code 1 --severity HIGH,CRITICAL so that if any unpatched Critical or High OS vulnerability is detected, Trivy stops the image from ever reaching the registry.

Trivy Image Scan Command
trivy image --exit-code 1 \
            --severity HIGH,CRITICAL \
            123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:${GIT_COMMIT}
Stage 09

📦 9. Push Container Image to Amazon ECR

What Happens: Once Trivy verifies 0 critical vulnerabilities, Jenkins authenticates securely to AWS using short-lived IAM credentials (via AWS STS) and pushes the security-vetted Docker container image to Amazon Elastic Container Registry (ECR).

Tools Used: AWS CLI, Amazon ECR, Docker Push.

AWS ECR Login & Image Push
# Authenticate Docker against Amazon ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com

# Push immutable container image
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:${GIT_COMMIT}
Stage 10

📝 10. Update Helm Chart Manifests (values.yaml)

What Happens: In GitOps, application configuration and deployment versioning are managed via Git repositories. Jenkins uses YAML processors (like yq) to update the container image tag inside the values.yaml file of the dedicated Helm configuration repository, pointing it to the new Git commit SHA.

Tools Used: Helm 3, yq, GitOps repository.

Automated Version Bumping in Helm values.yaml
# Update image tag using yq processor
yq eval '.image.tag = "'${GIT_COMMIT}'"' -i helm-chart/values.yaml
Stage 11

🐙 11. GitOps Configuration Repository Push

What Happens: Jenkins commits the updated values.yaml manifest back to the remote GitOps Infrastructure repository on GitHub. This commit acts as the explicit, auditable trigger for deployment.

Tools Used: Git, GitHub GitOps Repo.

GitOps Commit & Push
git config user.name "Jenkins CI Bot"
git config user.email "jenkins-bot@mycompany.com"
git commit -am "chore(deploy): update image tag to ${GIT_COMMIT}"
git push origin main
Stage 12

🐙 12. Argo CD Cluster Synchronization (GitOps Engine)

What Happens: Argo CD runs natively inside the Kubernetes cluster as a GitOps continuous delivery controller. It continuously polls the GitOps repository on GitHub. Upon detecting the new commit in values.yaml, Argo CD pulls the updated Helm chart and automatically synchronizes the cluster state to match Git.

Why it's better than `kubectl apply`: Argo CD detects and auto-heals cluster configuration drift, eliminates storing cluster admin credentials in Jenkins, and provides automated 1-click rollbacks!

Stage 13

🚀 13. Zero-Downtime Deployment on Production Kubernetes

What Happens: Kubernetes executes a RollingUpdate Deployment Strategy on Amazon EKS. New application pods spin up with the new container image, pass readiness and liveness health checks, traffic is gracefully shifted via Kubernetes Services, and old pods are safely terminated.

Result: Your security-vetted application update is live to global users with zero downtime!

💡 Key Lessons Learned & Best Practices

🧠 What I Learned

Building this pipeline proves that DevOps is not just about deployment—it's about:

  • Automation: Removing manual human intervention and manual SSH commands.
  • Security: Shifting security left so bugs are caught during build, not in production.
  • Observability: Tracking build status and container vulnerability metrics.
  • Reliability: Standardized pipelines ensure builds pass strict Quality Gates.
  • Repeatability: Every commit builds and deploys identically.

✅ Best Practices Implemented

  • Automate every deployment: No manual file uploads or manual edits.
  • Fail early: Stop the pipeline immediately if SonarQube or Trivy security checks fail.
  • Git as Single Source of Truth: Store all infrastructure and Helm deployment configs in Git (GitOps).
  • Proactive Monitoring: Monitor cluster health and set proactive alarms.
⚙️

Automation First

Zero manual steps from code commit to cluster deployment.

🔒

Shift-Left Security

Security checks (SonarQube, OWASP, Trivy) run in CI.

☁️

GitOps Deployment

Argo CD syncs cluster state continuously from Git.

♾️

Continuous Delivery

Zero-downtime rolling updates on Kubernetes.

❓ Frequently Asked Questions (FAQ) & Interview Prep

Q1: What does "Shift-Left Security" mean in a DevSecOps pipeline?

Answer: "Shift-Left Security" means integrating automated security testing early in the software development lifecycle (during code commit and build phases) rather than waiting until right before or after production deployment. Tools like SonarQube, OWASP Dependency-Check, and Trivy catch vulnerabilities when they are cheapest and easiest to fix.

Q2: Why use Argo CD (GitOps) instead of running `kubectl apply` inside Jenkins?

Answer: Using Argo CD keeps your Kubernetes credentials out of Jenkins, improving security. Argo CD runs inside Kubernetes, continuously monitoring your Git repository for changes and auto-correcting any manual drift in the cluster to ensure Git remains the true single source of truth.

Q3: What is the difference between SonarQube (SAST) and Aqua Trivy (Container Scanning)?

Answer: SonarQube performs Static Application Security Testing (SAST) on application source code (checking for code smells, logic bugs, SQL injection). Aqua Trivy scans the compiled Docker image and underlying Linux OS packages for known CVE vulnerabilities.

Q4: How does updating `values.yaml` trigger a deployment in a GitOps workflow?

Answer: The CI pipeline updates the container image tag in `values.yaml` and commits it to Git. Argo CD monitors the Git repo, detects the modified `values.yaml`, and automatically applies a rolling update to the Kubernetes cluster using the new image from Amazon ECR.

🚀
Cloud DevOps Hub Guide DevSecOps & GitOps Architecture Series