Linux Shell Scripting: Automate Your Dev Workflow

The One Line That Matters

set -euo pipefail — put this at the top of every script. If you remember nothing else from this article, remember that.

It catches unset variables. It stops execution on errors. It fails pipelines properly. Without it, your script keeps running after something breaks, silently corrupting whatever comes next.

Everything below is organized by what you are trying to do. Making scripts that will not silently break. Processing files without losing data. Automating deploys. And knowing when to stop using Bash entirely.

Scripts That Will Not Silently Break

Before quoting rules, before variables, before anything clever: the shebang line. #!/usr/bin/env bash. It tells the kernel which interpreter to run.

Quoting is where scripts go wrong first. Here is the full picture before you see any code. Double quotes expand variables but prevent word splitting — use them around every variable reference, always. Single quotes treat everything as a literal string. $(...) does command substitution. Backticks do the same thing but are harder to read and do not nest. Forget they exist.

basics.sh

#!/usr/bin/env bash
# A simple script demonstrating variables and quoting

# Variable assignment (no spaces around the equals sign)
APP_NAME="my-web-app"
VERSION="2.4.1"
DEPLOY_DIR="/opt/apps/${APP_NAME}"

# Double quotes preserve spaces and expand variables
echo "Deploying ${APP_NAME} version ${VERSION}..."
echo "Target directory: ${DEPLOY_DIR}"

# Single quotes treat everything literally
echo 'This will print ${APP_NAME} literally, no expansion.'

# Command substitution with $()
CURRENT_DATE=$(date +%Y-%m-%d)
GIT_HASH=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown")
echo "Build date: ${CURRENT_DATE}"
echo "Git commit: ${GIT_HASH}"

# Default values with parameter expansion
PORT="${1:-8080}"        # Use first arg, default to 8080
ENV="${2:-development}"  # Use second arg, default to development
echo "Starting on port ${PORT} in ${ENV} mode"

${VAR} vs $VAR? Same thing when the variable stands alone. But $APP_NAME_backup.tar.gz looks for a variable called APP_NAME_backup. Braces make boundaries explicit. Just always use them.

${VAR:-default} gives you a fallback when the variable is unset. ${VAR:=default} does the same but also assigns it. Memorize both. They show up in every script that takes optional arguments, and they replace five lines of if/else checking.

Conditionals and Test Expressions

[[ ... ]] for new scripts. Always. The old [ ... ] (literally the test command) has quoting landmines that [[ ]] eliminates. Only use single brackets if you need strict POSIX portability on some ancient system.

Pre-deployment check script showing several patterns at once:

conditionals.sh

#!/usr/bin/env bash

CONFIG_FILE="./config/deploy.yml"
MIN_DISK_MB=500

# File existence checks
if [[ ! -f "${CONFIG_FILE}" ]]; then
  echo "Error: Config file not found at ${CONFIG_FILE}" >&2
  exit 1
fi

# String comparison
BRANCH=$(git branch --show-current)
if [[ "${BRANCH}" != "main" && "${BRANCH}" != "master" ]]; then
  echo "Warning: You are deploying from branch '${BRANCH}', not main."
  read -p "Continue? (y/n): " confirm
  if [[ "${confirm}" != "y" ]]; then
    echo "Deployment cancelled."
    exit 0
  fi
fi

# Numeric comparison
AVAIL_DISK=$(df -m / | awk 'NR==2 {print $4}')
if (( AVAIL_DISK < MIN_DISK_MB )); then
  echo "Error: Only ${AVAIL_DISK}MB free. Need at least ${MIN_DISK_MB}MB." >&2
  exit 1
fi

# Pattern matching with [[ ]]
FILENAME="release-2.4.1.tar.gz"
if [[ "${FILENAME}" == *.tar.gz ]]; then
  echo "Extracting gzipped tarball..."
  tar -xzf "${FILENAME}"
elif [[ "${FILENAME}" == *.zip ]]; then
  echo "Extracting zip archive..."
  unzip "${FILENAME}"
else
  echo "Unknown archive format." >&2
  exit 1
fi

echo "All pre-flight checks passed. Proceeding with deployment."

Error messages go to stderr via >&2. Non-negotiable. Callers need clean stdout.

(( ... )) for arithmetic. Readable. The alternative — [[ $AVAIL_DISK -lt $MIN_DISK_MB ]] — works but looks like it was designed to annoy you. And glob patterns like == *.tar.gz inside double brackets do not need quoting on the right side. That pattern matching is the entire reason [[ ]] exists.

Processing Files Without Losing Data

for when you have a list. while when you are waiting.

That covers 90% of it. until exists too but while ! ... does the same thing and reads better. The script below hits a list of servers, processes log files by glob, and waits for a database to start. Three real patterns.

loops.sh

#!/usr/bin/env bash

# For loop: iterate over a list of servers
SERVERS=("web01.example.com" "web02.example.com" "web03.example.com")

for server in "${SERVERS[@]}"; do
  echo "Checking health of ${server}..."
  if curl -sf "http://${server}/health" > /dev/null 2>&1; then
    echo " [OK] ${server} is healthy"
  else
    echo " [FAIL] ${server} is not responding"
  fi
done

# For loop: iterate over files matching a pattern
echo ""
echo "Processing log files..."
for logfile in /var/log/app/*.log; do
  if [[ ! -f "${logfile}" ]]; then
    continue  # skip if glob did not match anything
  fi
  LINE_COUNT=$(wc -l < "${logfile}")
  echo " ${logfile}: ${LINE_COUNT} lines"
done

# While loop: wait for a service to become available
echo ""
echo "Waiting for database to accept connections..."
MAX_RETRIES=30
RETRY_COUNT=0
while ! pg_isready -h localhost -p 5432 > /dev/null 2>&1; do
  (( ++RETRY_COUNT ))  # pre-increment: (( x++ )) from 0 returns 1 and trips set -e
  if (( RETRY_COUNT >= MAX_RETRIES )); then
    echo "Database did not start within ${MAX_RETRIES} seconds." >&2
    exit 1
  fi
  sleep 1
done
echo "Database is ready after ${RETRY_COUNT} seconds."

# C-style for loop: useful for numeric ranges
echo ""
echo "Creating worker processes..."
for (( i=1; i<=4; i++ )); do
  echo " Starting worker-${i}"
done

File globs in for loops are a trap. No matches? Bash hands you the literal glob string. The script checks [[ ! -f "${logfile}" ]] before processing because of this. Or put shopt -s nullglob at the top so unmatched globs expand to nothing.

while IFS= read -r line; do ... done < file.txt for reading files line by line. The IFS= preserves leading whitespace, -r prevents backslash mangling. Skip either and you get corrupted output on lines with special characters. That matters for files where indentation is meaningful, such as YAML.

Functions and Return Values

Past fifty lines without functions? Good luck maintaining that.

return sets exit status (0-255). echo sends text to stdout. They do completely different things and people mix them up constantly. Need a computed value back from a function? Echo it and capture with $(...). Need pass/fail? Use return 0 or return 1.

functions.sh

#!/usr/bin/env bash

# A logging function with severity levels
log() {
  local level="${1}"
  local message="${2}"
  local timestamp
  timestamp=$(date +"%Y-%m-%d %H:%M:%S")
  echo "[${timestamp}] [${level}] ${message}"
}

# A function that returns a value via echo
get_app_version() {
  local package_file="${1:-package.json}"
  if [[ ! -f "${package_file}" ]]; then
    echo "0.0.0"
    return 1
  fi
  grep -o '"version": "[^"]*"' "${package_file}" | head -1 | cut -d'"' -f4
}

# A validation function that uses return for pass/fail
validate_environment() {
  local errors=0

  if ! command -v docker > /dev/null 2>&1; then
    log "ERROR" "Docker is not installed"
    (( ++errors ))
  fi

  if ! command -v node > /dev/null 2>&1; then
    log "ERROR" "Node.js is not installed"
    (( ++errors ))
  fi

  if [[ ! -f ".env" ]]; then
    log "WARN" ".env file not found, using defaults"
  fi

  return "${errors}"
}

# Using the functions
log "INFO" "Starting build pipeline..."

if validate_environment; then
  log "INFO" "Environment validation passed"
else
  log "ERROR" "Environment validation failed"
  exit 1
fi

VERSION=$(get_app_version) || true  # falls back to 0.0.0; without || true, set -e would stop here
log "INFO" "Building version ${VERSION}"

local on every variable inside functions. Without it, variables leak into global scope. Two functions sharing a variable called result will stomp on each other. Maddening to debug.

command -v to check whether a program exists. Not which. which behaves differently on different systems and sometimes prints noise to stdout. command -v is a shell built-in. Works everywhere.

Text Processing: grep, sed, awk

grep finds lines. sed replaces text in streams. awk pulls columns out of structured text. Three tools, and they cover most of what you will need.

If you can't read the pipeline below, break it into separate commands. No shame in that.

text-processing.sh

#!/usr/bin/env bash

# grep: find all error lines in a log file
echo "=== Recent Errors ==="
grep -i "error" /var/log/app/application.log | tail -20

# grep: count how many 500 errors occurred today
TODAY=$(date +%Y-%m-%d)
ERROR_COUNT=$(grep -c "${TODAY}.*HTTP 500" /var/log/nginx/access.log)
echo "500 errors today: ${ERROR_COUNT}"

# sed: replace a configuration value in-place
sed -i 's/^DB_HOST=.*/DB_HOST=db.production.internal/' .env
sed -i 's/^DEBUG=true/DEBUG=false/' .env

# sed: remove all comment lines and blank lines from a config
sed '/^#/d; /^$/d' config.ini > config.clean.ini

# awk: parse disk usage and flag partitions over 80%
echo ""
echo "=== Disk Usage Warnings ==="
df -h | awk 'NR>1 {
  usage = int($5)
  if (usage > 80) {
    printf " WARNING: %s is at %s (%s used)\n", $6, $5, $3
  }
}'

# awk: generate a summary of response times from access logs
echo ""
echo "=== Response Time Summary ==="
awk '{
  sum += $NF; count++
  if ($NF > max) max = $NF
  if (min == 0 || $NF < min) min = $NF
}
END {
  if (count > 0)
    printf " Avg: %.2fms | Min: %.2fms | Max: %.2fms | Samples: %d\n",
      sum/count, min, max, count
}' /var/log/app/response_times.log

Do not go deep on awk. It is a full programming language and some people write entire programs in it. Bad idea. If your awk gets hard to read, pipe into Python instead. Or use jq for JSON. I have seen awk one-liners nobody on the team could parse, including the person who wrote them a week later. Column extraction and simple field math. That is awk's sweet spot. Everything beyond that belongs in a language with actual error messages.

Cross-platform trap: sed -i on macOS needs sed -i '' 's/old/new/' file. GNU sed on Linux does not require the empty string argument. If your scripts run on both, detect the OS or install GNU sed via Homebrew. Or just stop writing scripts that need to run on both. Pick one.

Error Handling and Debugging

Your pg_dump fails because somebody rotated the password. Script moves on. tar packages up an empty directory. You find out six weeks later when you need to restore from backup and every archive is empty. This happens more often than people admit.

set -euo pipefail again. -e exits on any command failure. -u treats unset variables as errors. -o pipefail means a pipeline fails if any command in the chain fails, not just the last one. Three flags. Most silent failures gone.

One trap with set -e: arithmetic that evaluates to 0 returns a non-zero status. (( count++ )) when count is 0 evaluates to 0, so the script exits on the spot. Use (( ++count )) or count=$((count + 1)) instead.

For cleanup, trap with the EXIT signal:

error-handling.sh

#!/usr/bin/env bash
set -euo pipefail

# Create a temporary directory and ensure it gets cleaned up
WORK_DIR=$(mktemp -d)
echo "Working in temporary directory: ${WORK_DIR}"

# trap ensures cleanup runs even if the script fails
cleanup() {
  local exit_code=$?
  echo "Cleaning up temporary files..."
  rm -rf "${WORK_DIR}"
  if (( exit_code != 0 )); then
    echo "Script failed with exit code ${exit_code}" >&2
  fi
}
trap cleanup EXIT
trap 'exit 130' INT   # Ctrl+C: exit, which runs the EXIT trap once
trap 'exit 143' TERM

# A retry function with exponential backoff
retry() {
  local max_attempts="${1}"
  local delay=1
  shift

  for (( attempt=1; attempt<=max_attempts; attempt++ )); do
    if "$@"; then
      return 0
    fi
    echo " Attempt ${attempt}/${max_attempts} failed. Retrying in ${delay}s..." >&2
    sleep "${delay}"
    (( delay *= 2 ))
  done

  echo " All ${max_attempts} attempts failed." >&2
  return 1
}

# Download an artifact with retry logic
echo "Downloading build artifact..."
retry 3 curl -fSL -o "${WORK_DIR}/app.tar.gz" \
  "https://releases.example.com/app-latest.tar.gz"

# Extract and verify
tar -xzf "${WORK_DIR}/app.tar.gz" -C "${WORK_DIR}"
echo "Build artifact extracted successfully."

# Debugging tip: use set -x to trace execution
# set -x  # Uncomment to see every command before it runs
# set +x  # Turn tracing back off

The EXIT trap runs whenever the shell exits: normal exit, an error under set -e, or a signal you turn into an exit, as the INT and TERM traps above do. Put the cleanup on EXIT only, so it runs once. Nothing can catch SIGKILL (kill -9), so cleanup cannot be guaranteed against it.

set -x for debugging. Prints every command before it runs, with variables expanded. Put it before the suspicious section, set +x after. Targeted trace.

And quote your variables. Always. Unquoted variable with a space? Word splitting. Containing *? Filesystem glob expansion. Both can be destructive. Dead simple rule: if it is a variable, double-quote it.

Automating Deploys and Backups

Starting points to adapt. Run each one against a test directory or a throwaway database first.

Automated Backup Script

Timestamped backup of a directory and a PostgreSQL database, compressed, with automatic cleanup of old archives.

backup.sh

#!/usr/bin/env bash
set -euo pipefail

# Configuration
APP_DIR="/opt/apps/my-web-app"
BACKUP_DIR="/backups"
DB_NAME="app_production"
RETENTION_DAYS=7
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_NAME="backup_${TIMESTAMP}"

log() { echo "[$(date +%H:%M:%S)] $1"; }

# Create backup directory
mkdir -p "${BACKUP_DIR}/${BACKUP_NAME}"

# Backup application files
log "Backing up application files..."
tar -czf "${BACKUP_DIR}/${BACKUP_NAME}/app_files.tar.gz" \
  -C "$(dirname "${APP_DIR}")" \
  "$(basename "${APP_DIR}")"

# Backup database
log "Dumping database..."
pg_dump "${DB_NAME}" | gzip > "${BACKUP_DIR}/${BACKUP_NAME}/db_dump.sql.gz"

# Create a final archive and remove the directory
log "Compressing final archive..."
tar -czf "${BACKUP_DIR}/${BACKUP_NAME}.tar.gz" \
  -C "${BACKUP_DIR}" "${BACKUP_NAME}"
rm -rf "${BACKUP_DIR}/${BACKUP_NAME}"

# Clean up old backups
log "Removing backups older than ${RETENTION_DAYS} days..."
find "${BACKUP_DIR}" -name "backup_*.tar.gz" -mtime +"${RETENTION_DAYS}" -delete

# Report final size
SIZE=$(du -sh "${BACKUP_DIR}/${BACKUP_NAME}.tar.gz" | cut -f1)
log "Backup complete: ${BACKUP_NAME}.tar.gz (${SIZE})"

Deployment Script

deploy.sh

#!/usr/bin/env bash
set -euo pipefail

APP_NAME="my-web-app"
DEPLOY_ROOT="/opt/apps/${APP_NAME}"
RELEASES_DIR="${DEPLOY_ROOT}/releases"
CURRENT_LINK="${DEPLOY_ROOT}/current"
RELEASE_ID=$(date +%Y%m%d%H%M%S)
RELEASE_DIR="${RELEASES_DIR}/${RELEASE_ID}"
KEEP_RELEASES=5

log() { echo "[deploy] $1"; }

# Create release directory and extract new code
mkdir -p "${RELEASE_DIR}"
log "Extracting release ${RELEASE_ID}..."
tar -xzf "./build/${APP_NAME}.tar.gz" -C "${RELEASE_DIR}"

# Install dependencies and build
log "Installing dependencies..."
cd "${RELEASE_DIR}" && npm ci --omit=dev

# Swap the symlink atomically
log "Switching to new release..."
ln -sfn "${RELEASE_DIR}" "${CURRENT_LINK}"

# Restart the application
log "Restarting service..."
systemctl restart "${APP_NAME}"

# Health check
sleep 3
if curl -sf http://localhost:3000/health > /dev/null; then
  log "Health check passed. Deployment successful!"
else
  log "Health check FAILED. Rolling back..."
  PREVIOUS=$(ls -1t "${RELEASES_DIR}" | sed -n '2p')
  ln -sfn "${RELEASES_DIR}/${PREVIOUS}" "${CURRENT_LINK}"
  systemctl restart "${APP_NAME}"
  exit 1
fi

# Prune old releases, keeping only the latest N
log "Pruning old releases (keeping ${KEEP_RELEASES})..."
ls -1t "${RELEASES_DIR}" | tail -n +$(( KEEP_RELEASES + 1 )) | \
  while IFS= read -r old_release; do
    rm -rf "${RELEASES_DIR}/${old_release}"
    log " Removed release: ${old_release}"
  done

log "Deployment complete."

System Monitoring Script

Lightweight check of CPU, memory, disk, and key services. For small deployments where Prometheus would be overkill. It reads /proc, so it is Linux-only, and it exits non-zero when anything is wrong, which makes it easy to wire into cron mail or a chat webhook.

monitor.sh

#!/usr/bin/env bash
set -euo pipefail

DISK_LIMIT="${DISK_LIMIT:-85}"   # percent used
MEM_LIMIT="${MEM_LIMIT:-90}"
SERVICES=(nginx postgresql)
problems=0

report() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"
  (( ++problems ))
}

# Load: 1-minute average compared with the number of CPUs
read -r load _ < /proc/loadavg
cpus=$(nproc)
if awk -v l="${load}" -v c="${cpus}" 'BEGIN { exit !(l > c) }'; then
  report "Load ${load} is above ${cpus} CPUs"
fi

# Memory: percent of total that is not available
mem_used=$(awk '/MemTotal/ {t=$2} /MemAvailable/ {a=$2} END { printf "%d", (t - a) * 100 / t }' /proc/meminfo)
if (( mem_used >= MEM_LIMIT )); then
  report "Memory ${mem_used}% used"
fi

# Disk: every real filesystem
while read -r used mount; do
  used=${used%\%}
  if (( used >= DISK_LIMIT )); then
    report "Disk ${mount} is ${used}% full"
  fi
done < <(df -P -x tmpfs -x devtmpfs --output=pcent,target | tail -n +2)

# Services managed by systemd
for svc in "${SERVICES[@]}"; do
  if ! systemctl is-active --quiet "${svc}"; then
    report "Service ${svc} is not running"
  fi
done

if (( problems == 0 )); then
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] OK"
fi
exit $(( problems > 0 ))

Schedule with crontab: 0 * * * * /opt/scripts/monitor.sh >> /var/log/monitor.log 2>&1 for hourly checks.

Other things worth scripting:

Compress logs older than a day, archive weekly, purge after 90 days
Git hooks that run linters and tests before each commit
A bootstrap script that installs dependencies, clones repos, and configures dotfiles on a fresh machine
SQL migration runner that applies files in order and tracks what has already run
SSL certificate expiration checker that triggers Let's Encrypt renewal

When to Stop Using Bash

Past 100 lines, switch to Python. Past the point where you need data structures more complex than an array, switch to Python. Past the point where you are parsing JSON without jq, definitely switch to Python.

Bash is glue. Good glue. But still glue.

One more thing. Install ShellCheck and run it on every script. It catches quoting bugs, portability issues, and subtle mistakes that would take hours to track down manually. Free. Fast. It will teach you more about Bash than reading documentation ever will.

Anurag Sinha

Founder & Lead Developer

Anurag is a full stack developer and self-taught programmer from India with over five years of experience building web applications, APIs, and cloud-native systems. He researches, writes and reviews every tutorial on Codertronix, and updates them when the tools change.

Linux Shell Scripting: Automate Your Daily Development Workflow