Background tasks and schedulers

Background tasks and schedulers: work that should not block a web request, jobs that run on a timer, and the trade-offs between in-process threads, separate worker processes, and cron. The pattern shows up everywhere once you ship something real – sending email, generating reports, importing files overnight.

Why not do everything in the request

A user clicks “Send report” and expects a quick response. Generating a PDF, hitting three APIs, and emailing forty people can take minutes. Do the minimum in the HTTP handler – validate input, enqueue a job, return 202 Accepted with a job id. Let a worker do the heavy lifting.

Simple queue with threading (dev only)

For local development or tiny tools, a background thread and a queue get you started. Do not rely on this in production across multiple server processes – each process has its own memory.

import queue
import threading
import uuid

job_queue: queue.Queue = queue.Queue()
results: dict[str, str] = {}

def worker() -> None:
    while True:
        job_id, func, args = job_queue.get()
        try:
            results[job_id] = func(*args)
        except Exception as exc:
            results[job_id] = f"error: {exc}"
        finally:
            job_queue.task_done()

threading.Thread(target=worker, daemon=True).start()

def enqueue(func, *args) -> str:
    job_id = str(uuid.uuid4())
    job_queue.put((job_id, func, args))
    return job_id

Redis and RQ

Production setups usually use a broker – Redis is common – and a worker process that pulls jobs. RQ (Redis Queue) is minimal and Python-native.

from redis import Redis
from rq import Queue

redis_conn = Redis(host="localhost", port=6379)
task_queue = Queue(connection=redis_conn)

def build_report(report_id: str) -> str:
    # long work here
    return f"report-{report_id}.pdf"

# In your Flask route:
# job = task_queue.enqueue(build_report, report_id)
# return {"job_id": job.id}, 202

# Separate terminal: rq worker

Workers restart independently of web workers. Scale them horizontally when the queue backs up. Monitor failed jobs – RQ keeps a failed registry you can inspect and retry.

Celery for heavier workflows

Celery adds routing, schedules, and chains. The configuration cost is higher. Reach for it when you need periodic tasks, retries with backoff, or multiple named queues – not for a single “email this file” button.

Schedulers and cron

System cron still works: call your script on a timetable. APScheduler embeds scheduling inside Python if you want one process to own timers.

from apscheduler.schedulers.blocking import BlockingScheduler

def nightly_cleanup() -> None:
    print("Removing temp files...")

scheduler = BlockingScheduler()
scheduler.add_job(nightly_cleanup, "cron", hour=2, minute=0)
# scheduler.start()  # blocks - run in a dedicated process

Run schedulers as a single instance unless you enjoy duplicate runs. A file lock, Redis lock, or orchestrator rule (“only one replica runs cron”) prevents double execution.

Idempotent jobs

Jobs retry. Networks fail. Design tasks so running them twice does not double-charge a customer or send duplicate emails. Use stable idempotency keys where the outside world allows it.

Observability

Log job start, finish, duration, and failure reason. Expose job status to callers – polling an endpoint beats leaving users staring at a spinner with no feedback.