Docker Compose: Orchestrating Multi-Container Applications

Introduction

Run docker compose up and your entire stack starts. Run docker compose down and it stops. That is the pitch.

The interesting parts are everything that goes wrong between those two commands. Networking gotchas where containers cannot find each other. Volume permissions that silently corrupt your database. Health checks that look right but do not actually prevent startup races. Most Compose tutorials spend too long on what services are and too little on the things that will actually bite you at 2am.

Docker Compose Fundamentals

Everything lives in a YAML file — usually compose.yaml (the older docker-compose.yml name still works). Services, networks, volumes.

services:
  web:
    image: nginx:alpine
    ports:
      - "8080:80"

  api:
    build: ./api
    ports:
      - "3000:3000"
    depends_on:
      - db

  db:
    image: postgres:16
    environment:
      POSTGRES_DB: myapp
      POSTGRES_PASSWORD: secret
    volumes:
      - db_data:/var/lib/postgresql/data

volumes:
  db_data:

Each service name doubles as a hostname. Your API connects to the database at db:5432. No IP addresses to look up. And this is the part most people miss at first — the name you pick is not just a label, it is DNS. Name your service database instead of db and every connection string in your app changes. So pick names deliberately.

The top-level volumes key declares the named volume db_data, and the db service mounts it at Postgres's data directory, so the database survives docker compose down. No networks key is needed here; Compose creates a default network for the project.

docker compose up creates networks, pulls or builds images, starts containers. docker compose down tears it down. Add -v to nuke volumes too.

Services and How They Talk to Each Other

Each service is one container. But a service is useless in isolation — what matters is how services find each other on the network. By default, docker compose up creates one network for your entire project. Every service joins it. Every service can reach every other service by name.

For most projects, that default network is all you need. The problems start when it is not enough.

Using Pre-Built Images vs. Building from Source

services:
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
      args:
        NODE_ENV: development
    ports:
      - "5173:5173"
    volumes:
      - ./frontend/src:/app/src
    command: npm run dev

  backend:
    build:
      context: ./backend
      target: development
    ports:
      - "8000:8000"
    environment:
      DATABASE_URL: postgres://user:pass@db:5432/myapp
      REDIS_URL: redis://cache:6379
    depends_on:
      - db
      - cache

  cache:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    restart: unless-stopped

  db:
    image: postgres:16-alpine
    ports:
      - "5432:5432"
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: myapp

volumes:
  db_data:

Standard stuff.

The backend service is more interesting. It targets a specific stage (development) in a multi-stage Dockerfile — one Dockerfile across dev, test, and production, just select the stage you want. And look at the connection URLs: db:5432 and cache:6379. Service names are hostnames. But here is the gotcha: if you rename a service, every other service referencing it by hostname breaks silently. No error at compose time. The API just fails to connect at runtime.

restart: unless-stopped on the cache is the right default for most services. Docker restarts it after crashes but leaves it alone if you explicitly stop it.

Custom Networks

Your frontend should not be able to reach the database directly. Custom networks control which services can talk to each other. The idea: put the frontend and API on one network, put the API and database on another. The API sits on both, acting as the only bridge. The frontend tries to connect to db directly and gets nothing — no route exists. That isolation matters more as your stack grows, because with dozens of microservices you do not want every container able to talk to every other container. Limits the blast radius when something breaks.

services:
  frontend:
    image: my-frontend:latest
    networks:
      - frontend_net
    ports:
      - "3000:3000"

  api:
    image: my-api:latest
    networks:
      - frontend_net
      - backend_net
    ports:
      - "8000:8000"

  db:
    image: postgres:16-alpine
    networks:
      - backend_net
    volumes:
      - db_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    networks:
      - backend_net

networks:
  frontend_net:
    driver: bridge
  backend_net:
    driver: bridge

volumes:
  db_data:

Nothing surprising here.

One thing worth noting: the driver: bridge is the default and you can omit it. But being explicit helps when someone new reads the file. Or when you need to switch to overlay for Swarm later and want to see the diff clearly.

Volumes and Data Persistence

Containers are ephemeral. Your database data needs to survive restarts. Two types of mounts: named volumes (managed by Docker, persist across restarts) and bind mounts (map a host directory into the container, good for live code reloading).

services:
  app:
    build: .
    volumes:
      # Bind mount for live reloading in development
      - ./src:/app/src
      # Named volume for node_modules (avoids host conflicts)
      - node_modules:/app/node_modules
      # Named volume for uploaded files
      - uploads:/app/uploads

  db:
    image: postgres:16-alpine
    volumes:
      # Named volume for persistent database storage
      - pgdata:/var/lib/postgresql/data
      # Bind mount for init scripts (runs once on first start)
      - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql

  grafana:
    image: grafana/grafana:latest
    volumes:
      # Named volume with custom driver options
      - grafana_storage:/var/lib/grafana
      # Read-only bind mount for provisioning
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro

volumes:
  pgdata:
    driver: local
  node_modules:
  uploads:
  grafana_storage:

Watch the node_modules trick. If you bind-mount your entire project directory, the host's node_modules overwrites the container's copy. Native modules compiled for macOS end up inside a Linux container. Bad. A named volume for node_modules keeps the container's copy separate.

The :ro flag on the Grafana mount makes it read-only inside the container. Good habit for config files.

And here is the volume permissions thing that nobody warns you about clearly enough: on Linux, the user inside the container and the user on the host are different UIDs. The official Postgres image runs as UID 999 on its Debian variants and UID 70 on the Alpine variants used in this guide. Your bind-mounted directory is owned by UID 1000. Postgres cannot write to it. The fix is either matching UIDs in your Dockerfile or using named volumes (which Docker manages the permissions for). On macOS and Windows this is invisible because Docker Desktop handles the translation. So you develop happily on a Mac, deploy to Linux, and everything breaks. Every time.

docker compose down preserves named volumes by default. You need docker compose down -v to actually delete them. Intentional — stops you from nuking your database during routine restarts.

Health Checks and Dependencies

depends_on without health check conditions is almost useless. It waits for the container to start, not for the service inside to be ready. Your API starts, tries to connect to Postgres, crashes because Postgres is still initializing. The container is running. The database is not.

services:
  api:
    build: ./api
    ports:
      - "8000:8000"
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_healthy
      migrations:
        condition: service_completed_successfully
    healthcheck:
      # Slim/Alpine images often lack curl; use the runtime the image already has
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 10s

  db:
    image: postgres:16-alpine
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 15s
    volumes:
      - pgdata:/var/lib/postgresql/data

  cache:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 3

  migrations:
    build: ./api
    command: python manage.py migrate
    depends_on:
      db:
        condition: service_healthy

volumes:
  pgdata:

The API health check calls Python's standard library instead of curl, because slim and Alpine base images often ship without curl and a missing binary makes the check fail forever. urlopen raises on connection errors and non-2xx responses, which exits non-zero and marks the container unhealthy. For a Node image, the same idea works with node -e and fetch.

The migrations service is a one-shot container — runs migrations, exits, and the API depends on it with service_completed_successfully. Your API always starts against a current schema instead of racing the migration.

One opinionated take: start_period is the most important timing parameter and the one people skip. Without it, a database that takes 10 seconds to initialize fails health checks during boot, gets marked unhealthy, and your dependent services never start. Set it generously. 15 seconds for Postgres, 30 for Elasticsearch. You lose nothing by being conservative here.

Environment Variables and Secrets

Someone on your team will hardcode production credentials into the Compose file. When, not if.

services:
  api:
    build: ./api
    environment:
      # Inline variable with value
      NODE_ENV: production
      # Variable interpolated from shell or .env file
      DATABASE_URL: postgres://${DB_USER}:${DB_PASS}@db:5432/${DB_NAME}
      JWT_SECRET: ${JWT_SECRET}
      REDIS_URL: redis://cache:6379
    env_file:
      - .env
      # Optional per-developer overrides; Compose errors on a missing file unless required: false
      - path: .env.local
        required: false

  worker:
    build: ./worker
    env_file:
      - .env
      - worker/.env
    environment:
      WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-4}

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASS}
      POSTGRES_DB: ${DB_NAME}

Every env_file entry is required by default, so a missing .env.local stops docker compose up with an error. The long form with required: false makes it optional. ${VARIABLE:-default} provides a fallback. WORKER_CONCURRENCY defaults to 4 if not set anywhere.

# Database configuration
DB_USER=appuser
DB_PASS=supersecretpassword
DB_NAME=myapp_production

# Authentication
JWT_SECRET=your-256-bit-secret-key-here

# Worker settings
WORKER_CONCURRENCY=8

# Logging
LOG_LEVEL=info

Three rules. Never commit .env files with real credentials. Commit an .env.example with placeholder values so new developers know what to set. Use env_file for shared variables, inline environment for service-specific overrides.

Plain Compose also supports secrets without Swarm. Declare them under a top-level secrets key (from a file or an environment variable), grant them per service, and Compose bind-mounts each one read-only at /run/secrets/<name>. Unlike environment variables, they don't show up in docker inspect or get inherited by child processes.

services:
  db:
    image: postgres:16-alpine
    environment:
      # The official image reads the password from this file
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

Many official images, Postgres included, accept *_FILE variants of their environment variables for exactly this. For production, teams commonly inject secrets from CI/CD at deploy time or from a dedicated secrets manager such as HashiCorp Vault.

Profiles and Override Files

Not every service needs to run all the time. Profiles group optional services so they only start when you ask.

services:
  # Core services (always start)
  api:
    build: ./api
    ports:
      - "8000:8000"

  db:
    image: postgres:16-alpine
    volumes:
      - pgdata:/var/lib/postgresql/data

  # Monitoring (only with --profile monitoring)
  prometheus:
    image: prom/prometheus:latest
    profiles:
      - monitoring
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

  grafana:
    image: grafana/grafana:latest
    profiles:
      - monitoring
    ports:
      - "3001:3000"

  # Testing (only with --profile testing)
  selenium:
    image: selenium/standalone-chrome:latest
    profiles:
      - testing
    ports:
      - "4444:4444"

  mailpit:
    image: axllent/mailpit:latest
    profiles:
      - testing
      - development
    ports:
      - "8025:8025"

volumes:
  pgdata:

Services without a profiles key always start. Services with profiles only start when you request them:

docker compose --profile monitoring up starts core services plus monitoring. docker compose --profile monitoring --profile testing up starts everything. No profile flags? Only api and db come up.

Override files work differently. Compose reads compose.yaml first, then automatically merges compose.override.yaml if it exists. Base file stays production-like. Override adds bind mounts, debug ports, development commands. Skip the override in production with docker compose -f compose.yaml up.

You can layer multiple files: docker compose -f compose.yaml -f compose.staging.yaml up. Later files override earlier ones.

Things You Can Safely Ignore Until You Need Them

• extends — service inheritance. Sounds elegant, makes debugging harder. Copy-paste is fine until you have 10+ services sharing config.
• configs — like secrets, but for non-sensitive config files. Bind mounts do the same job in local development.
• deploy — docker compose does apply deploy.resources.limits/reservations and deploy.replicas, which is handy for reproducing memory limits locally. Placement constraints and update/rollback policies only matter on Swarm.
• Custom network drivers — bridge handles everything in single-host development. overlay only matters for multi-host.

Add health checks with condition: service_healthy on every dependency. Not just the database. That single change eliminates most intermittent startup failures. And set up compose.override.yaml from day one — base file stays production-like, dev conveniences go in the override.