Senior DevOps Engineer Runbook

Jenkins Production
Install Guide

A complete, opinionated guide to deploying Jenkins LTS on Ubuntu 22.04 with Nginx reverse proxy, Let's Encrypt TLS, hardened systemd service, and production security controls.

โšก7 Phases ยท 35+ Steps
๐Ÿ”’Security-first approach
๐Ÿ–ฅUbuntu 22.04 LTS
โ˜•OpenJDK 17
PHASE 01
Prerequisites & Server Prep
Verify your server meets production requirements and prepare the OS before touching Jenkins.
MINIMUM SPECS
4 vCPUs ยท 8 GB RAM ยท 50 GB SSD
RECOMMENDED SPECS
8 vCPUs ยท 16 GB RAM ยท 200 GB SSD
OPERATING SYSTEM
Ubuntu 22.04 LTS (Jammy Jellyfish)
OPEN PORTS (INBOUND)
22 (SSH) ยท 80 (HTTP) ยท 443 (HTTPS)
setup steps
01
Update system packages and set hostname
Begin with a fully patched OS. Set a meaningful FQDN hostname โ€” it appears in Jenkins logs, agent connections, and TLS certificate generation.
BASH
# Refresh package lists and apply all security patches
sudo apt update && sudo apt upgrade -y

# Set production hostname (replace with your FQDN)
sudo hostnamectl set-hostname jenkins.yourdomain.com

# Verify hostname is set correctly
hostnamectl
02
Configure swap space RECOMMENDED
Prevents OOM kills during heavy parallel builds. Add 4 GB swap as a safety net. Reduce swappiness to favour RAM on SSD-backed servers.
BASH
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Persist swap across reboots
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# Reduce swappiness for SSD โ€” prefer RAM over swap
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Verify swap is active
free -h
03
Install essential utilities
Install dependencies used later in the guide โ€” curl for downloads, gnupg for key management, and ca-certificates for TLS validation.
BASH
sudo apt install -y \
  curl \
  wget \
  gnupg \
  ca-certificates \
  apt-transport-https \
  software-properties-common \
  ufw \
  unzip
PHASE 02
Java & Jenkins Install
Jenkins requires Java 17 or 21. Always install from the official Debian package repository โ€” never the Ubuntu default.
01
Install OpenJDK 17 REQUIRED
Jenkins LTS is certified on Java 17. Set JAVA_HOME permanently so it persists across reboots and is available to all services.
BASH
sudo apt install -y fontconfig openjdk-17-jre

# Verify Java version
java -version
# Expected: openjdk version "17.x.x" ...

# Set JAVA_HOME for all users and services
echo 'JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64' | sudo tee -a /etc/environment
source /etc/environment
echo $JAVA_HOME
02
Add the official Jenkins APT repository CRITICAL
Never install Jenkins from Ubuntu's default repo โ€” it is outdated and unsupported. Always use the official Jenkins Debian stable repository.
BASH
# Download and trust Jenkins GPG key
sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key

# Add the Jenkins stable (LTS) repository
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian-stable binary/" | \
  sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null

# Refresh apt with the new repo
sudo apt update
03
Install Jenkins LTS STABLE
LTS releases are updated every 12 weeks with security patches only. The weekly release is for development/testing โ€” never use it in production.
BASH
sudo apt install -y jenkins

# Verify the installed version
dpkg -l jenkins
jenkins --version
Production Rule: Pin the Jenkins version after install to prevent apt upgrade from silently breaking your instance. Create /etc/apt/preferences.d/jenkins with:
Package: jenkins
Pin: version 2.xxx.x
Pin-Priority: 1001
04
Tune JVM heap and GC settings CRITICAL
Under-provisioned JVM heap is the #1 cause of Jenkins instability. Set this before first start. Use G1GC for predictable pause times on heaps above 2 GB.
BASH
# Create a systemd drop-in for Jenkins JVM args
sudo mkdir -p /etc/systemd/system/jenkins.service.d/

sudo tee /etc/systemd/system/jenkins.service.d/override.conf <<EOF
[Service]
Environment="JAVA_OPTS=-Xmx4g -Xms2g \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:+HeapDumpOnOutOfMemoryError \
  -XX:HeapDumpPath=/var/lib/jenkins/heapdump.hprof \
  -Djava.awt.headless=true"
Environment="JENKINS_PORT=8080"
Environment="JENKINS_LISTEN_ADDRESS=127.0.0.1"
EOF

sudo systemctl daemon-reload
Sizing Guide: Set -Xmx to ~50% of total RAM. 8 GB server โ†’ -Xmx4g. 16 GB server โ†’ -Xmx8g. Always leave headroom for the OS and build agents.
PHASE 03
Systemd Service & Firewall
Enable Jenkins as a managed system service, configure auto-restart on failure, and lock down network access via UFW.
01
Enable and start the Jenkins systemd service
Systemd manages process lifecycle, auto-restart on crash, and log collection via journald. Enable at boot before starting.
BASH
# Enable Jenkins to start on boot
sudo systemctl enable jenkins

# Start Jenkins service now
sudo systemctl start jenkins

# Confirm it is running (look for "active (running)")
sudo systemctl status jenkins

# Tail live logs to watch startup progress
sudo journalctl -u jenkins -f
02
Add systemd resource limits and restart policy
Prevent runaway builds from consuming all system resources. Set file descriptor and process limits above Jenkins defaults.
BASH
# Append resource limits to the override created in Phase 2
sudo tee -a /etc/systemd/system/jenkins.service.d/override.conf <<EOF

# Auto-restart on crash (not on clean stop)
Restart=on-failure
RestartSec=5s

# High file descriptor limit (needed for many concurrent builds)
LimitNOFILE=65536
LimitNPROC=32768

# Basic sandboxing
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/jenkins /var/log/jenkins /tmp
EOF

sudo systemctl daemon-reload
sudo systemctl restart jenkins
03
Configure UFW firewall CRITICAL
Never expose port 8080 to the internet. All web traffic must go through Nginx on 443. Allow only SSH, HTTP, and HTTPS. Do this in the correct order to avoid lockout.
BASH
# STEP 1: Allow SSH FIRST โ€” do this before enabling UFW
# or you will lock yourself out of the server
sudo ufw allow OpenSSH

# Allow HTTP and HTTPS (for Nginx reverse proxy)
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Explicitly block direct Jenkins port access
sudo ufw deny 8080/tcp

# If you use JNLP agents on port 50000,
# allow ONLY from your agent subnet (never 0.0.0.0)
# sudo ufw allow from 10.0.1.0/24 to any port 50000

# Enable UFW with default deny-incoming policy
sudo ufw --force enable

# Confirm rules are applied correctly
sudo ufw status verbose
Do not skip the SSH rule! Always add ufw allow OpenSSH before ufw enable. Enabling UFW with default deny-incoming without an SSH rule will immediately lock you out of the server.
PHASE 04
Nginx Reverse Proxy & TLS
Terminate TLS at Nginx and proxy all traffic to Jenkins on localhost:8080. Nginx also adds security headers and enforces HTTPS redirects.
01
Install Nginx and Certbot
BASH
sudo apt install -y nginx certbot python3-certbot-nginx

# Enable Nginx at boot and start it
sudo systemctl enable nginx
sudo systemctl start nginx
02
Create the Nginx server block for Jenkins
The X-Forwarded-* headers are required โ€” Jenkins uses them to generate correct redirect URLs and webhook callback URIs. Without them, Jenkins redirects to http://localhost:8080.
NGINX
sudo tee /etc/nginx/sites-available/jenkins <<'EOF'
upstream jenkins {
  keepalive 32;
  server 127.0.0.1:8080 fail_timeout=0;
}

# Redirect all HTTP to HTTPS
server {
  listen 80;
  server_name jenkins.yourdomain.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;
  server_name jenkins.yourdomain.com;

  # TLS โ€” Certbot will populate these automatically
  ssl_certificate     /etc/letsencrypt/live/jenkins.yourdomain.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/jenkins.yourdomain.com/privkey.pem;
  ssl_protocols       TLSv1.2 TLSv1.3;
  ssl_ciphers         HIGH:!aNULL:!MD5;
  ssl_prefer_server_ciphers on;
  ssl_session_cache   shared:SSL:10m;
  ssl_session_timeout 10m;

  # Security headers
  add_header X-Content-Type-Options    nosniff;
  add_header X-Frame-Options           SAMEORIGIN;
  add_header X-XSS-Protection         "1; mode=block";
  add_header Referrer-Policy           "no-referrer-when-downgrade";
  add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

  # Proxy to Jenkins
  location / {
    proxy_pass         http://jenkins;
    proxy_http_version 1.1;
    proxy_set_header   Connection        "";
    proxy_set_header   Host              $host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_set_header   X-Forwarded-Host  $host;
    proxy_read_timeout 90s;
    proxy_connect_timeout 5s;
    client_max_body_size 50m;
  }
}
EOF

# Enable the site and validate config
sudo ln -s /etc/nginx/sites-available/jenkins \
           /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
03
Obtain TLS certificate via Certbot CRITICAL
Ensure your DNS A record resolves to this server's public IP before running Certbot, or the ACME challenge will fail.
BASH
# Verify DNS resolves before running Certbot
dig +short jenkins.yourdomain.com
# Should return your server's public IP

# Obtain and auto-configure the TLS certificate
sudo certbot --nginx \
  -d jenkins.yourdomain.com \
  --non-interactive \
  --agree-tos \
  --email admin@yourdomain.com \
  --redirect

# Verify the auto-renewal timer is active
sudo systemctl status certbot.timer

# Test renewal without making changes
sudo certbot renew --dry-run
04
Restrict Jenkins to listen on localhost only
After placing Nginx in front, bind Jenkins to 127.0.0.1 only. This ensures no traffic can bypass the proxy and reach Jenkins directly.
BASH
# Verify the override.conf already has this line
# (it was set in Phase 2, Step 4)
grep JENKINS_LISTEN_ADDRESS \
  /etc/systemd/system/jenkins.service.d/override.conf
# Expected: Environment="JENKINS_LISTEN_ADDRESS=127.0.0.1"

# If missing, add it now
echo 'Environment="JENKINS_LISTEN_ADDRESS=127.0.0.1"' | \
  sudo tee -a /etc/systemd/system/jenkins.service.d/override.conf

sudo systemctl daemon-reload
sudo systemctl restart jenkins

# Verify โ€” this should FAIL (connection refused from external IP)
# curl http://<YOUR_PUBLIC_IP>:8080  # should timeout/refuse
# curl https://jenkins.yourdomain.com  # should succeed
PHASE 05
Initial Setup & Plugin Configuration
Complete the first-time configuration wizard, configure the Jenkins URL, and install a curated set of production plugins.
01
Retrieve the initial admin password
Jenkins writes a one-time password to disk during first boot. You need this to unlock the web UI at https://jenkins.yourdomain.com.
BASH
# Print the initial admin unlock password
sudo cat /var/lib/jenkins/secrets/initialAdminPassword

# Also watch startup logs if Jenkins is still initialising
sudo journalctl -u jenkins -f | grep -i "password"
02
Set Jenkins URL in System Configuration CRITICAL
Navigate to: Manage Jenkins โ†’ System โ†’ Jenkins URL. Set this to your HTTPS URL. Jenkins uses this to generate webhook callback URLs, email links, and agent JNLPs. An incorrect URL breaks all of these.
Set Jenkins URL to: https://jenkins.yourdomain.com โ€” include the trailing slash if prompted. This setting persists in /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml.
03
Disable executors on the controller node CRITICAL
In production, the controller should only orchestrate โ€” never execute builds. Builds on the controller can access Jenkins internals and expose credentials. Go to: Manage Jenkins โ†’ Nodes โ†’ Built-In Node โ†’ Configure โ†’ Number of executors: 0
Security Boundary: Running builds on the controller grants pipeline code access to the Jenkins process, secret store, and credential files. Set executors to 0 and use dedicated agent nodes for all build workloads.
04
Install recommended production plugins
Install via Manage Jenkins โ†’ Plugins โ†’ Available, or use the Jenkins CLI. Only install what you need โ€” plugin bloat slows startup and increases attack surface.
BASH โ€” Jenkins CLI
# Download Jenkins CLI jar
wget -O jenkins-cli.jar \
  https://jenkins.yourdomain.com/jnlpJars/jenkins-cli.jar

# Install plugins via CLI (run from admin credentials)
java -jar jenkins-cli.jar \
  -s https://jenkins.yourdomain.com \
  -auth admin:<YOUR_TOKEN> \
  install-plugin \
    pipeline-stage-view \
    workflow-aggregator \
    git \
    github-branch-source \
    credentials-binding \
    matrix-auth \
    audit-trail \
    role-strategy \
    configuration-as-code \
    prometheus \
    build-timeout \
    timestamper \
    slack \
    email-ext \
    -restart
05
Bootstrap Jenkins Configuration as Code (JCasC)
After installing the configuration-as-code plugin, manage all Jenkins config via YAML in source control. This makes your instance reproducible and auditable.
BASH
# Create JCasC config directory
sudo mkdir -p /var/lib/jenkins/casc_configs
sudo chown jenkins:jenkins /var/lib/jenkins/casc_configs

# Tell Jenkins where to find the JCasC config
sudo tee -a /etc/systemd/system/jenkins.service.d/override.conf <<EOF
Environment="CASC_JENKINS_CONFIG=/var/lib/jenkins/casc_configs"
EOF

sudo systemctl daemon-reload
sudo systemctl restart jenkins
PHASE 06
Security Hardening
Production Jenkins is a high-value target โ€” it holds SCM credentials, deployment keys, and cloud tokens. Lock it down before connecting to any pipeline.
01
Switch to Matrix-based security CRITICAL
Jenkins ships with "Anyone can do anything" by default. Switch to Matrix-based authorization immediately. Create your admin account first, then disable anonymous access.
YAML โ€” JCasC
# /var/lib/jenkins/casc_configs/security.yaml
jenkins:
  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: "admin"
          password: "${JENKINS_ADMIN_PASSWORD}"
  authorizationStrategy:
    projectMatrix:
      permissions:
        - "Overall/Administer:admin"
        - "Overall/Read:authenticated"
        - "Job/Read:authenticated"
        - "Job/Build:developers"
        - "Job/Cancel:developers"
02
Disable legacy CLI over remoting and weak agent protocols
The CLI-over-remoting channel has had multiple critical CVEs. Disable it. Restrict agent protocols to JNLP4 (TLS-secured) only.
BASH
# Disable CLI over remoting via JVM flag
# Add to JAVA_OPTS in override.conf:
-Djenkins.CLI.disabled=true

# Via Manage Jenkins โ†’ Security:
# [ ] Enable CLI over Remoting  (UNCHECK THIS)
# Agent protocols: check only "JNLP4-connect"
# [ ] JNLP-connect  (UNCHECK โ€” no TLS)
# [ ] JNLP2-connect (UNCHECK โ€” deprecated)
# [x] JNLP4-connect (KEEP โ€” TLS-secured)

# Verify CLI over Remoting is disabled
curl -s -o /dev/null -w "%{http_code}" \
  https://jenkins.yourdomain.com/cli/
# Expected: 404 or 403 (not 200)
03
Harden filesystem permissions on JENKINS_HOME
Restrict access to credentials, secrets, and config files. Only the jenkins system user should be able to read these files.
BASH
# Lock down JENKINS_HOME directory
sudo chmod 750 /var/lib/jenkins
sudo chown -R jenkins:jenkins /var/lib/jenkins

# Protect the credentials store (contains all secrets)
sudo chmod 600 /var/lib/jenkins/credentials.xml

# Protect the master encryption key
sudo chmod 600 /var/lib/jenkins/secrets/master.key
sudo chmod 700 /var/lib/jenkins/secrets/

# Lock down log directory
sudo chmod 750 /var/log/jenkins

# Verify โ€” no other user should be able to read these
ls -la /var/lib/jenkins/secrets/
ls -la /var/lib/jenkins/credentials.xml
04
Automate encrypted backups DON'T SKIP
Back up at minimum: jobs/, credentials.xml, config.xml, secrets/. Run nightly to an off-box destination like S3.
BASH
sudo tee /usr/local/bin/jenkins-backup.sh <<'EOF' #!/bin/bash set -euo pipefail BACKUP_DIR="/tmp/jenkins-backup-$(date +%Y%m%d-%H%M)" JENKINS_HOME="/var/lib/jenkins" S3_BUCKET="s3://your-backup-bucket/jenkins" mkdir -p "$BACKUP_DIR" # Copy critical files only cp -r \ "$JENKINS_HOME/jobs" \ "$JENKINS_HOME/config.xml" \ "$JENKINS_HOME/credentials.xml" \ "$JENKINS_HOME/secrets" \ "$JENKINS_HOME/users" \ "$JENKINS_HOME/casc_configs" \ "$BACKUP_DIR/" # Compress and encrypt with GPG tar -czf "${BACKUP_DIR}.tar.gz" "$BACKUP_DIR" gpg --symmetric --cipher-algo AES256 "${BACKUP_DIR}.tar.gz" # Upload to S3 aws s3 cp "${BACKUP_DIR}.tar.gz.gpg" "$S3_BUCKET/" # Clean up local temp files rm -rf "$BACKUP_DIR" "${BACKUP_DIR}.tar.gz" "${BACKUP_DIR}.tar.gz.gpg" EOF sudo chmod +x /usr/local/bin/jenkins-backup.sh # Schedule nightly at 02:00 echo "0 2 * * * jenkins /usr/local/bin/jenkins-backup.sh \ >> /var/log/jenkins/backup.log 2>&1" | \ sudo tee /etc/cron.d/jenkins-backup
PHASE 07
Go-Live Checklist
Tick every item before handling real workloads or CI/CD pipelines. This checklist is your production sign-off.
0
of 15 complete
0%
tls & network
โœ“
Jenkins accessible at https://jenkins.yourdomain.com with a valid TLS certificate (no browser warnings)
TLS / NETWORK
โœ“
HTTP requests redirect to HTTPS (no plain-text access allowed)
TLS / NETWORK
โœ“
Port 8080 is NOT reachable from the internet (verified with nmap or external port scan)
TLS / NETWORK
โœ“
Certbot auto-renewal verified with certbot renew --dry-run
TLS / NETWORK
security & access
โœ“
Anonymous access disabled; dedicated admin account created with a strong password
SECURITY
โœ“
Controller node executors set to 0 โ€” no builds run on the controller
SECURITY
โœ“
CLI over Remoting disabled; agent protocols restricted to JNLP4 only
SECURITY
โœ“
Audit Trail plugin installed and writing logs to /var/log/jenkins/audit.log
SECURITY
reliability & operations
โœ“
Jenkins service configured for auto-restart on failure (Restart=on-failure in systemd)
RELIABILITY
โœ“
JVM heap properly sized (-Xmx set to ~50% of total RAM โ€” not the default 256m)
RELIABILITY
โœ“
Backup job configured, tested with a successful restore drill
RELIABILITY
โœ“
Jenkins version pinned to prevent accidental upgrades via apt upgrade
RELIABILITY
configuration & monitoring
โœ“
Jenkins URL set correctly in System Configuration (https://jenkins.yourdomain.com)
CONFIGURATION
โœ“
Prometheus metrics endpoint accessible (/prometheus) for monitoring and alerting
MONITORING
โœ“
Disk usage alert configured โ€” alert threshold set at 80% capacity on /var/lib/jenkins
MONITORING