Senior DevOps Engineer Runbook
Jenkins Production
Install Guide
A complete, opinionated guide to deploying Jenkins LTS on Ubuntu 22.04 with Nginx reverse proxy, Let's Encrypt TLS, hardened systemd service, and production security controls.
PHASE 01
Prerequisites & Server Prep
Verify your server meets production requirements and prepare the OS before touching Jenkins.
MINIMUM SPECS
4 vCPUs ยท 8 GB RAM ยท 50 GB SSD
RECOMMENDED SPECS
8 vCPUs ยท 16 GB RAM ยท 200 GB SSD
OPERATING SYSTEM
Ubuntu 22.04 LTS (Jammy Jellyfish)
OPEN PORTS (INBOUND)
22 (SSH) ยท 80 (HTTP) ยท 443 (HTTPS)
setup steps
01
Update system packages and set hostname
Begin with a fully patched OS. Set a meaningful FQDN hostname โ it appears in Jenkins logs, agent connections, and TLS certificate generation.
BASH
# Refresh package lists and apply all security patches sudo apt update && sudo apt upgrade -y # Set production hostname (replace with your FQDN) sudo hostnamectl set-hostname jenkins.yourdomain.com # Verify hostname is set correctly hostnamectl
02
Configure swap space RECOMMENDED
Prevents OOM kills during heavy parallel builds. Add 4 GB swap as a safety net. Reduce swappiness to favour RAM on SSD-backed servers.
BASH
sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile # Persist swap across reboots echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab # Reduce swappiness for SSD โ prefer RAM over swap echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf sudo sysctl -p # Verify swap is active free -h
03
Install essential utilities
Install dependencies used later in the guide โ curl for downloads, gnupg for key management, and ca-certificates for TLS validation.
BASH
sudo apt install -y \ curl \ wget \ gnupg \ ca-certificates \ apt-transport-https \ software-properties-common \ ufw \ unzip
PHASE 02
Java & Jenkins Install
Jenkins requires Java 17 or 21. Always install from the official Debian package repository โ never the Ubuntu default.
01
Install OpenJDK 17 REQUIRED
Jenkins LTS is certified on Java 17. Set JAVA_HOME permanently so it persists across reboots and is available to all services.
BASH
sudo apt install -y fontconfig openjdk-17-jre # Verify Java version java -version # Expected: openjdk version "17.x.x" ... # Set JAVA_HOME for all users and services echo 'JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64' | sudo tee -a /etc/environment source /etc/environment echo $JAVA_HOME
02
Add the official Jenkins APT repository CRITICAL
Never install Jenkins from Ubuntu's default repo โ it is outdated and unsupported. Always use the official Jenkins Debian stable repository.
BASH
# Download and trust Jenkins GPG key sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \ https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key # Add the Jenkins stable (LTS) repository echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \ https://pkg.jenkins.io/debian-stable binary/" | \ sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null # Refresh apt with the new repo sudo apt update
03
Install Jenkins LTS STABLE
LTS releases are updated every 12 weeks with security patches only. The weekly release is for development/testing โ never use it in production.
BASH
sudo apt install -y jenkins
# Verify the installed version
dpkg -l jenkins
jenkins --version
Production Rule: Pin the Jenkins version after install to prevent
apt upgrade from silently breaking your instance. Create /etc/apt/preferences.d/jenkins with: Package: jenkins
Pin: version 2.xxx.x
Pin-Priority: 1001
04
Tune JVM heap and GC settings CRITICAL
Under-provisioned JVM heap is the #1 cause of Jenkins instability. Set this before first start. Use G1GC for predictable pause times on heaps above 2 GB.
BASH
# Create a systemd drop-in for Jenkins JVM args
sudo mkdir -p /etc/systemd/system/jenkins.service.d/
sudo tee /etc/systemd/system/jenkins.service.d/override.conf <<EOF
[Service]
Environment="JAVA_OPTS=-Xmx4g -Xms2g \
-XX:+UseG1GC \
-XX:MaxGCPauseMillis=200 \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/var/lib/jenkins/heapdump.hprof \
-Djava.awt.headless=true"
Environment="JENKINS_PORT=8080"
Environment="JENKINS_LISTEN_ADDRESS=127.0.0.1"
EOF
sudo systemctl daemon-reload
Sizing Guide: Set
-Xmx to ~50% of total RAM. 8 GB server โ -Xmx4g. 16 GB server โ -Xmx8g. Always leave headroom for the OS and build agents.
PHASE 03
Systemd Service & Firewall
Enable Jenkins as a managed system service, configure auto-restart on failure, and lock down network access via UFW.
01
Enable and start the Jenkins systemd service
Systemd manages process lifecycle, auto-restart on crash, and log collection via journald. Enable at boot before starting.
BASH
# Enable Jenkins to start on boot sudo systemctl enable jenkins # Start Jenkins service now sudo systemctl start jenkins # Confirm it is running (look for "active (running)") sudo systemctl status jenkins # Tail live logs to watch startup progress sudo journalctl -u jenkins -f
02
Add systemd resource limits and restart policy
Prevent runaway builds from consuming all system resources. Set file descriptor and process limits above Jenkins defaults.
BASH
# Append resource limits to the override created in Phase 2 sudo tee -a /etc/systemd/system/jenkins.service.d/override.conf <<EOF # Auto-restart on crash (not on clean stop) Restart=on-failure RestartSec=5s # High file descriptor limit (needed for many concurrent builds) LimitNOFILE=65536 LimitNPROC=32768 # Basic sandboxing NoNewPrivileges=true ProtectSystem=strict ReadWritePaths=/var/lib/jenkins /var/log/jenkins /tmp EOF sudo systemctl daemon-reload sudo systemctl restart jenkins
03
Configure UFW firewall CRITICAL
Never expose port 8080 to the internet. All web traffic must go through Nginx on 443. Allow only SSH, HTTP, and HTTPS. Do this in the correct order to avoid lockout.
BASH
# STEP 1: Allow SSH FIRST โ do this before enabling UFW # or you will lock yourself out of the server sudo ufw allow OpenSSH # Allow HTTP and HTTPS (for Nginx reverse proxy) sudo ufw allow 80/tcp sudo ufw allow 443/tcp # Explicitly block direct Jenkins port access sudo ufw deny 8080/tcp # If you use JNLP agents on port 50000, # allow ONLY from your agent subnet (never 0.0.0.0) # sudo ufw allow from 10.0.1.0/24 to any port 50000 # Enable UFW with default deny-incoming policy sudo ufw --force enable # Confirm rules are applied correctly sudo ufw status verbose
Do not skip the SSH rule! Always add
ufw allow OpenSSH before ufw enable. Enabling UFW with default deny-incoming without an SSH rule will immediately lock you out of the server.
PHASE 04
Nginx Reverse Proxy & TLS
Terminate TLS at Nginx and proxy all traffic to Jenkins on localhost:8080. Nginx also adds security headers and enforces HTTPS redirects.
01
Install Nginx and Certbot
BASH
sudo apt install -y nginx certbot python3-certbot-nginx
# Enable Nginx at boot and start it
sudo systemctl enable nginx
sudo systemctl start nginx
02
Create the Nginx server block for Jenkins
The
X-Forwarded-* headers are required โ Jenkins uses them to generate correct redirect URLs and webhook callback URIs. Without them, Jenkins redirects to http://localhost:8080.NGINX
sudo tee /etc/nginx/sites-available/jenkins <<'EOF'
upstream jenkins {
keepalive 32;
server 127.0.0.1:8080 fail_timeout=0;
}
# Redirect all HTTP to HTTPS
server {
listen 80;
server_name jenkins.yourdomain.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
server_name jenkins.yourdomain.com;
# TLS โ Certbot will populate these automatically
ssl_certificate /etc/letsencrypt/live/jenkins.yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/jenkins.yourdomain.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# Security headers
add_header X-Content-Type-Options nosniff;
add_header X-Frame-Options SAMEORIGIN;
add_header X-XSS-Protection "1; mode=block";
add_header Referrer-Policy "no-referrer-when-downgrade";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
# Proxy to Jenkins
location / {
proxy_pass http://jenkins;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_read_timeout 90s;
proxy_connect_timeout 5s;
client_max_body_size 50m;
}
}
EOF
# Enable the site and validate config
sudo ln -s /etc/nginx/sites-available/jenkins \
/etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
03
Obtain TLS certificate via Certbot CRITICAL
Ensure your DNS A record resolves to this server's public IP before running Certbot, or the ACME challenge will fail.
BASH
# Verify DNS resolves before running Certbot dig +short jenkins.yourdomain.com # Should return your server's public IP # Obtain and auto-configure the TLS certificate sudo certbot --nginx \ -d jenkins.yourdomain.com \ --non-interactive \ --agree-tos \ --email admin@yourdomain.com \ --redirect # Verify the auto-renewal timer is active sudo systemctl status certbot.timer # Test renewal without making changes sudo certbot renew --dry-run
04
Restrict Jenkins to listen on localhost only
After placing Nginx in front, bind Jenkins to
127.0.0.1 only. This ensures no traffic can bypass the proxy and reach Jenkins directly.BASH
# Verify the override.conf already has this line # (it was set in Phase 2, Step 4) grep JENKINS_LISTEN_ADDRESS \ /etc/systemd/system/jenkins.service.d/override.conf # Expected: Environment="JENKINS_LISTEN_ADDRESS=127.0.0.1" # If missing, add it now echo 'Environment="JENKINS_LISTEN_ADDRESS=127.0.0.1"' | \ sudo tee -a /etc/systemd/system/jenkins.service.d/override.conf sudo systemctl daemon-reload sudo systemctl restart jenkins # Verify โ this should FAIL (connection refused from external IP) # curl http://<YOUR_PUBLIC_IP>:8080 # should timeout/refuse # curl https://jenkins.yourdomain.com # should succeed
PHASE 05
Initial Setup & Plugin Configuration
Complete the first-time configuration wizard, configure the Jenkins URL, and install a curated set of production plugins.
01
Retrieve the initial admin password
Jenkins writes a one-time password to disk during first boot. You need this to unlock the web UI at
https://jenkins.yourdomain.com.BASH
# Print the initial admin unlock password sudo cat /var/lib/jenkins/secrets/initialAdminPassword # Also watch startup logs if Jenkins is still initialising sudo journalctl -u jenkins -f | grep -i "password"
02
Set Jenkins URL in System Configuration CRITICAL
Navigate to: Manage Jenkins โ System โ Jenkins URL. Set this to your HTTPS URL. Jenkins uses this to generate webhook callback URLs, email links, and agent JNLPs. An incorrect URL breaks all of these.
Set Jenkins URL to:
https://jenkins.yourdomain.com โ include the trailing slash if prompted. This setting persists in /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml.
03
Disable executors on the controller node CRITICAL
In production, the controller should only orchestrate โ never execute builds. Builds on the controller can access Jenkins internals and expose credentials. Go to: Manage Jenkins โ Nodes โ Built-In Node โ Configure โ Number of executors: 0
Security Boundary: Running builds on the controller grants pipeline code access to the Jenkins process, secret store, and credential files. Set executors to 0 and use dedicated agent nodes for all build workloads.
04
Install recommended production plugins
Install via Manage Jenkins โ Plugins โ Available, or use the Jenkins CLI. Only install what you need โ plugin bloat slows startup and increases attack surface.
BASH โ Jenkins CLI
# Download Jenkins CLI jar wget -O jenkins-cli.jar \ https://jenkins.yourdomain.com/jnlpJars/jenkins-cli.jar # Install plugins via CLI (run from admin credentials) java -jar jenkins-cli.jar \ -s https://jenkins.yourdomain.com \ -auth admin:<YOUR_TOKEN> \ install-plugin \ pipeline-stage-view \ workflow-aggregator \ git \ github-branch-source \ credentials-binding \ matrix-auth \ audit-trail \ role-strategy \ configuration-as-code \ prometheus \ build-timeout \ timestamper \ slack \ email-ext \ -restart
05
Bootstrap Jenkins Configuration as Code (JCasC)
After installing the
configuration-as-code plugin, manage all Jenkins config via YAML in source control. This makes your instance reproducible and auditable.BASH
# Create JCasC config directory sudo mkdir -p /var/lib/jenkins/casc_configs sudo chown jenkins:jenkins /var/lib/jenkins/casc_configs # Tell Jenkins where to find the JCasC config sudo tee -a /etc/systemd/system/jenkins.service.d/override.conf <<EOF Environment="CASC_JENKINS_CONFIG=/var/lib/jenkins/casc_configs" EOF sudo systemctl daemon-reload sudo systemctl restart jenkins
PHASE 06
Security Hardening
Production Jenkins is a high-value target โ it holds SCM credentials, deployment keys, and cloud tokens. Lock it down before connecting to any pipeline.
01
Switch to Matrix-based security CRITICAL
Jenkins ships with "Anyone can do anything" by default. Switch to Matrix-based authorization immediately. Create your admin account first, then disable anonymous access.
YAML โ JCasC
# /var/lib/jenkins/casc_configs/security.yaml
jenkins:
securityRealm:
local:
allowsSignup: false
users:
- id: "admin"
password: "${JENKINS_ADMIN_PASSWORD}"
authorizationStrategy:
projectMatrix:
permissions:
- "Overall/Administer:admin"
- "Overall/Read:authenticated"
- "Job/Read:authenticated"
- "Job/Build:developers"
- "Job/Cancel:developers"
02
Disable legacy CLI over remoting and weak agent protocols
The CLI-over-remoting channel has had multiple critical CVEs. Disable it. Restrict agent protocols to JNLP4 (TLS-secured) only.
BASH
# Disable CLI over remoting via JVM flag # Add to JAVA_OPTS in override.conf: -Djenkins.CLI.disabled=true # Via Manage Jenkins โ Security: # [ ] Enable CLI over Remoting (UNCHECK THIS) # Agent protocols: check only "JNLP4-connect" # [ ] JNLP-connect (UNCHECK โ no TLS) # [ ] JNLP2-connect (UNCHECK โ deprecated) # [x] JNLP4-connect (KEEP โ TLS-secured) # Verify CLI over Remoting is disabled curl -s -o /dev/null -w "%{http_code}" \ https://jenkins.yourdomain.com/cli/ # Expected: 404 or 403 (not 200)
03
Harden filesystem permissions on JENKINS_HOME
Restrict access to credentials, secrets, and config files. Only the jenkins system user should be able to read these files.
BASH
# Lock down JENKINS_HOME directory sudo chmod 750 /var/lib/jenkins sudo chown -R jenkins:jenkins /var/lib/jenkins # Protect the credentials store (contains all secrets) sudo chmod 600 /var/lib/jenkins/credentials.xml # Protect the master encryption key sudo chmod 600 /var/lib/jenkins/secrets/master.key sudo chmod 700 /var/lib/jenkins/secrets/ # Lock down log directory sudo chmod 750 /var/log/jenkins # Verify โ no other user should be able to read these ls -la /var/lib/jenkins/secrets/ ls -la /var/lib/jenkins/credentials.xml
04
Automate encrypted backups DON'T SKIP
Back up at minimum:
jobs/, credentials.xml, config.xml, secrets/. Run nightly to an off-box destination like S3.BASH
sudo tee /usr/local/bin/jenkins-backup.sh <<'EOF' #!/bin/bash set -euo pipefail BACKUP_DIR="/tmp/jenkins-backup-$(date +%Y%m%d-%H%M)" JENKINS_HOME="/var/lib/jenkins" S3_BUCKET="s3://your-backup-bucket/jenkins" mkdir -p "$BACKUP_DIR" # Copy critical files only cp -r \ "$JENKINS_HOME/jobs" \ "$JENKINS_HOME/config.xml" \ "$JENKINS_HOME/credentials.xml" \ "$JENKINS_HOME/secrets" \ "$JENKINS_HOME/users" \ "$JENKINS_HOME/casc_configs" \ "$BACKUP_DIR/" # Compress and encrypt with GPG tar -czf "${BACKUP_DIR}.tar.gz" "$BACKUP_DIR" gpg --symmetric --cipher-algo AES256 "${BACKUP_DIR}.tar.gz" # Upload to S3 aws s3 cp "${BACKUP_DIR}.tar.gz.gpg" "$S3_BUCKET/" # Clean up local temp files rm -rf "$BACKUP_DIR" "${BACKUP_DIR}.tar.gz" "${BACKUP_DIR}.tar.gz.gpg" EOF sudo chmod +x /usr/local/bin/jenkins-backup.sh # Schedule nightly at 02:00 echo "0 2 * * * jenkins /usr/local/bin/jenkins-backup.sh \ >> /var/log/jenkins/backup.log 2>&1" | \ sudo tee /etc/cron.d/jenkins-backup
PHASE 07
Go-Live Checklist
Tick every item before handling real workloads or CI/CD pipelines. This checklist is your production sign-off.
0
of 15 complete
0%
tls & network
โ
Jenkins accessible at
https://jenkins.yourdomain.com with a valid TLS certificate (no browser warnings)TLS / NETWORK
โ
HTTP requests redirect to HTTPS (no plain-text access allowed)
TLS / NETWORK
โ
Port 8080 is NOT reachable from the internet (verified with nmap or external port scan)
TLS / NETWORK
โ
Certbot auto-renewal verified with
certbot renew --dry-runTLS / NETWORK
security & access
โ
Anonymous access disabled; dedicated admin account created with a strong password
SECURITY
โ
Controller node executors set to 0 โ no builds run on the controller
SECURITY
โ
CLI over Remoting disabled; agent protocols restricted to JNLP4 only
SECURITY
โ
Audit Trail plugin installed and writing logs to
/var/log/jenkins/audit.logSECURITY
reliability & operations
โ
Jenkins service configured for auto-restart on failure (
Restart=on-failure in systemd)RELIABILITY
โ
JVM heap properly sized (
-Xmx set to ~50% of total RAM โ not the default 256m)RELIABILITY
โ
Backup job configured, tested with a successful restore drill
RELIABILITY
โ
Jenkins version pinned to prevent accidental upgrades via
apt upgradeRELIABILITY
configuration & monitoring
โ
Jenkins URL set correctly in System Configuration (
https://jenkins.yourdomain.com)CONFIGURATION
โ
Prometheus metrics endpoint accessible (
/prometheus) for monitoring and alertingMONITORING
โ
Disk usage alert configured โ alert threshold set at 80% capacity on
/var/lib/jenkinsMONITORING