Building a Distributed HPC Job Scheduler

Scaling High-Performance Computing (HPC) workflows requires a robust control plane. In this deep dive, I explore the architecture of the LSSD Pipeline—a distributed compute scheduler capable of processing 10,000+ jobs across a fleet of worker nodes using a Redis-backed queueing system.

The Problem with Monolithic Execution

When dealing with large-scale simulations or batch processing tasks, running everything on a single node quickly becomes the bottleneck. A monolithic script executing thousands of jobs sequentially or even via local multiprocessing hits memory, CPU, and thermal limits hard.

We needed a system that could dynamically distribute tasks across multiple disparate machines (nodes) while ensuring fault tolerance, observability, and flexible scheduling strategies.

Core Architecture

The architecture is split into three main components:

1. Control Plane (FastAPI): The brain of the operation. Exposes REST endpoints for users to submit job payloads, check statuses, and configure cluster nodes.
2. Task Queue (Redis): Acts as the ultra-fast broker between the control plane and the workers. Using Redis Lists and Pub/Sub mechanics allows for near-zero latency dispatch.
3. Worker Nodes (Python/Celery/Custom): Daemons running on the HPC nodes. They poll the Redis queue, execute the binary/script, and report stdout/stderr and exit codes back to the control plane.

Scheduling Algorithms

A naive First-Come-First-Serve (FCFS) queue is rarely enough for a shared cluster. I implemented four distinct scheduling algorithms within the control plane to optimize resource usage based on the scenario:

Implementation: The Dispatcher

Here is a simplified look at how the FastAPI control plane handles an incoming job request and routes it using Redis.

from fastapi import FastAPI, BackgroundTasks
import redis
import json

app = FastAPI()
redis_client = redis.Redis(host='localhost', port=6379, db=0)

@app.post("/api/v1/jobs/submit")
async def submit_job(payload: dict):
    job_id = generate_uuid()
    
    # 1. Save initial state
    redis_client.hset(f"job:{job_id}", mapping={
        "status": "QUEUED",
        "payload": json.dumps(payload)
    })
    
    # 2. Push to the appropriate queue based on priority
    queue_name = f"queue:{payload.get('priority', 'normal')}"
    redis_client.lpush(queue_name, job_id)
    
    return {"job_id": job_id, "status": "QUEUED"}

Fault Tolerance and Recovery

In an HPC environment, nodes will go down. Power fluctuations, out-of-memory (OOM) kills, or network partitions are inevitable. The LSSD Pipeline handles this gracefully through a heartbeat and lease mechanism.

When a worker picks up a job, it removes it from the main queue and places it in a processing list, simultaneously starting a heartbeat. If the control plane detects a worker's heartbeat has died, it assumes the node failed, takes all jobs from its processing list, and pushes them back onto the main queue for retry.

# Worker side heartbeat logic
def process_job(job_id):
    start_heartbeat_thread(job_id)
    try:
        execute_payload(job_id)
        redis_client.hset(f"job:{job_id}", "status", "COMPLETED")
    except Exception as e:
        redis_client.hset(f"job:{job_id}", "status", "FAILED")
    finally:
        stop_heartbeat_thread()
        redis_client.lrem("processing_queue", 0, job_id)

Conclusion

Building the LSSD Pipeline was an incredible exercise in distributed systems design. By leveraging the blistering speed of Redis and the asynchronous nature of FastAPI, we transformed a fragile, monolithic processing script into a resilient, scalable compute scheduler capable of handling enterprise-grade workloads.