Running and Monitoring Workers

At 03:12 a job throws an uncaught Error, PHP-FPM-style worker process exits, and there is nobody there to restart it. At 08:00 the queue has 12 000 pending jobs and every email is late. The worker was never broken — it just had no supervisor, and a queue worker is a process that must outlive any single job.

The process you are supervising

Start from the right artisan command:

php artisan queue:work redis \
  --queue=emails,default \
  --sleep=3 --tries=3 --max-time=3600 --memory=128

The worker checks memory between jobs and exits cleanly when it crosses the limit — supervisor then starts a fresh process. Without that line, a worker that slowly accumulates memory dies as a fatal Allowed memory size exhausted at the worst moment, or worse, drags the box down with it.

Supervisor: the restart policy

; /etc/supervisor/conf.d/worker-emails.conf
[program:worker-emails]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work redis --queue=emails,default --sleep=3 --tries=3 --max-time=3600
numprocs=4
autostart=true
autorestart=true                 # restart on any exit, including success-exits
startsecs=5                      # healthy only after 5s of survival
stopwaitsecs=650                 # MUST exceed your longest job (job timeout + margin)
stopsignal=TERM                  # queue:work finishes the current job, then exits
redirect_stderr=true
stdout_logfile=/var/www/app/storage/logs/worker.log
user=www-data
stopasgroup=true
killasgroup=true                 # no orphaned children when the group is stopped

supervisorctl reread && supervisorctl update
supervisorctl status                 # RUNNING / BACKOFF / FATAL — check after every deploy
supervisorctl restart worker-emails:*

stopwaitsecs deserves emphasis: supervisor sends SIGTERM, waits that many seconds, then SIGKILL. Set it longer than your slowest job, or a deploy will sever a running export mid-write and burn an attempt. Laravel's worker handles SIGTERM gracefully — it stops accepting new jobs and exits after the current one — which is exactly the behavior you want on php artisan queue:restart too: a graceful, deploy-safe bounce of every worker.

autorestart=true covers crashes; startsecs plus a health check covers the half-dead case (a worker running but unable to reach Redis). BACKOFF in supervisorctl status means it is crash-looping — read tail -f storage/logs/worker.log before restarting blindly, because the third restart usually fails the same way. On systemd-only boxes the same contract is three lines:

[Service]
ExecStart=/usr/bin/php /var/www/app/artisan queue:work redis --queue=default
Restart=always
RestartSec=5
MemoryMax=512M

Deploys without dropped jobs

The sequence that works: ship code → php artisan queue:restart (or php artisan horizon:terminate for Horizon) → wait for workers to report stopped in logs → new code is picked up by supervisor's restarted processes. Workers started before a code change keep serving old classes until they exit; --max-time and queue:restart are the two mechanisms that bound that staleness.

This is process-level reliability. The job-level half — what happens when an individual job throws three times — is Dispatching Queued Jobs, and the numbers you watch once this is running are in the third chapter below. Cron-driven maintenance of the same fleet lives in Scheduling Recurring Tasks.

A job is broken: every attempt throws, and the queue drains at zero while your workers burn CPU on the same poison message. Retry limits and backoff are the two knobs that stop that, and the first interview question is always about how the two configuration layers combine.

Where the retry budget comes from

Three sources can set maxTries, and they resolve in a fixed order:

class SyncInventory implements ShouldQueue
{
    public int $tries = 5;                       // 1. the job itself (wins when set)
    public int $maxExceptions = 3;               // cap on *exceptions*, across attempts
    public int|array $backoff = [10, 60, 300];   // 2. escalating delays per attempt
    public ?int $timeout = 90;                   // kill switch (requires pcntl)
}

php artisan queue:work --tries=3     # 3. worker default, only when the job is silent

The job's $tries beats the worker's --tries; the worker flag is only the fallback for jobs that declare nothing. With neither, attempts are unlimited — the config value queue.php → tries (or -1) makes that explicit rather than accidental. $maxExceptions is the stricter sibling: it counts exceptions even when you release() the job yourself, so a job that fails once every attempt but never exhausts $tries still gets cut off.

Backoff is what makes retries polite

public int|array $backoff = [10, 60, 300];   // delay before attempt 2, 3, 4

public function retryUntil(): \DateTime
{
    return now()->addHours(2);               // hard deadline: after this, fail immediately
}

Immediate retries are a denial-of-service tool pointed at your own dependency: if the inventory API is down, three instant attempts fail in 200 ms and you have proved nothing. Escalating backoff gives the upstream time to recover and spreads the herd. retryUntil() caps the whole lifecycle — after the deadline the job goes straight to failed_jobs instead of waiting out the next backoff slot.

The pieces around the job itself:

php artisan queue:failed                 # id, queue, exception, failed_at — run this first
php artisan queue:retry all              # push them back
php artisan queue:retry 9b2f1c
php artisan queue:forget 9b2f1c          # accept the loss
php artisan queue:prune --hours=48       # keep the table honest

For SQS, the same contract lives in the queue's redrive policy: after maxReceiveCount receives, SQS moves the message to a dead-letter queue and your failed_jobs table never sees it — know which side of that boundary your driver sits on. Framework-side, failed() on the job is where you release locks, mark rows, and increment the alerting counter.

A loop that cannot end

The invariant to state in an interview: every job must have a finite terminal state — success, failed_jobs, or DLQ — reachable without human action. Concretely:

• retryUntil() or $tries set on every job class (code review enforces it).
• Backoff array, never bare release() in a loop.
• An alert on failed_jobs growth rate, not just on its size.
• Poison messages removed via queue:forget after root-causing, so the retry storm stops.

Jobs that legitimately retry forever (heartbeats, pollers) are the exception, and they belong on a short --max-time worker pool, isolated from the main queue — a Horizon tries/backoff/timeout triple per queue in config/horizon.php is how you express that isolation declaratively.

The other half of reliability is knowing when the queue is slow rather than broken: wait time and throughput are covered next, and if you skipped Dispatching Queued Jobs, revisit it for where attempts are recorded in the first place.

The queue is not down — it is slow, and you will not find out from a 500 page. When workers fall behind, everything still succeeds; jobs just take 40 minutes. Queue wait time is the early-warning metric, and it is the number producers-outpacing-workers shows up in first.

What to measure

Wait time is the one to alert on: depth without wait time is misleading (a queue with 5 000 jobs and 64 workers may drain in seconds), but a wait time climbing monotonically means arrival rate exceeds service rate and the backlog will grow without bound. The math is one line: if jobs arrive at λ and each worker serves μ, you need ceil(λ / μ) workers with headroom — and Horizon's throughput chart hands you both numbers.

Instrumentation you can type today

php artisan queue:monitor redis:emails,redis:default --max=10
# exits non-zero when any listed queue exceeds --max waiting jobs → wire into cron/healthcheck

redis-cli llen queues:default            # raw depth, no PHP boot
redis-cli zrange horizon:metrics:wait -- -10 -1 WITHSCORES   # recent wait-time samples
supervisorctl status | grep -c RUNNING   # capacity check

queue:monitor is the piece most people miss: it is designed for a monitoring hook (Nagios, a load balancer health check, a cron that pages when emails exceeds 100 waiting jobs). Horizon adds the dashboard: wait time, recent jobs, throughput, and process counts, refreshed by horizon:snapshot — schedule it:

Schedule::command('horizon:snapshot')->everyFiveMinutes();

Scaling: more workers vs bigger workers

numprocs=8                     ; supervisor: fixed pool

// config/horizon.php — auto-scaling balances per queue
'balances' => ['redis'],
'scanning' => ['enabled' => true, 'maxProcesses' => 16, 'balanceMaxShift' => 1, 'balanceCooldown' => 3],
'metrics'  => ['trim_snapshots' => ['jobs' => 60, 'jobs:pending' => 60]],

Before adding processes, check the bottleneck is CPU-bound worker capacity and not a serialized lock: 16 workers hammering one inventory API rate limit will not help. Then decide which queue gets more capacity — priority queues (--queue=emails,default, Horizon's per-queue maxProcesses) let interactive email work skip behind nightly exports, at the cost of starving default if the high queue never empties.

Deploys and restarts that do not drop work

php artisan queue:restart        # every worker exits after its current job (supervisor restarts them)
php artisan horizon:terminate    # Horizon: master re-execs, workers finish gracefully
php artisan horizon:status       # paused / running — check after deploy
supervisorctl status             # confirm RUNNING, not BACKOFF

A graceful restart is the contract: stop taking new jobs, finish the current one, exit, come back on new code. Anything shorter (kill -9, systemctl stop without stopwaitsecs headroom) converts a deploy into duplicate side effects — the job's attempt counter goes up and your idempotency code from Dispatching Queued Jobs has to absorb it.

Interview framing: "How do you know your queue is healthy?" Answer with three numbers — wait time, throughput, and depth — plus the liveness check, then say which alarm fires first when you double traffic: wait time, because depth only grows after workers saturate. The heartbeat that keeps these workers fed with scheduled dispatches is Scheduling Recurring Tasks; the next phase moves to making the request path itself fast, starting with Enabling OPcache.