Chapter 11: Scheduling

A planetary scale computer consists of thousands or millions of machines, each capable of running many processes. Scheduling is the art and science of deciding where and when to run work on these machines. Our scheduling service orchestrates the entire fleet — spawning service processes, assigning ports, monitoring health, and reconciling desired state with actual state.

The Scheduler's Data Model

scheduling/src/main.rs The scheduler maintains two core structures. A ServiceSpec describes the desired state: the service name, the Cargo manifest path, an optional binary name, and the desired replica count. An Instance describes reality: a running process with an ID, port, OS process ID, and health status.

struct ServiceSpec {
    name: String,
    manifest_path: String,
    bin_name: String,
    desired_replicas: i32,
}

struct Instance {
    id: String,
    service_name: String,
    port: u16,
    pid: u32,
    status: String, // "starting", "healthy", "unhealthy", "stopped"
}

The gap between desired and actual state drives all scheduling decisions. When a service spec says three replicas but only two instances are running, the scheduler spawns one more. When an instance's health check fails, the scheduler marks it unhealthy and may replace it.

Process Spawning

echo/src/bin/server_v1.rs — this pattern appears in every service The scheduler spawns each service as an OS process via std::process::Command, passing the assigned port through the PORT environment variable. Every service in our system checks for this variable at startup:

let addr = std::env::var("PORT")
    .map(|p| format!("127.0.0.1:{}", p))
    .unwrap_or_else(|_| SYSTEM_ADDRESS.to_string());

This three-line pattern appears in every service's main.rs. It means services can run standalone with their well-known port (for development) or accept a dynamically assigned port (when managed by the scheduler). The spawned process self-registers with discovery, making it immediately discoverable by other services.

Fleet Bootstrap

scheduling/src/main.rs On startup, the scheduler bootstraps the entire fleet from a hardcoded configuration table. Each entry specifies the service name, Cargo manifest path, optional binary name, replica count, and base port. Single-replica services get their well-known ports. Multi-replica services get sequential ports from a base.

const DEFAULT_FLEET: &[FleetEntry] = &[
    FleetEntry { name: "security",      manifest_path: "security/Cargo.toml",
                 bin_name: "", replicas: 1, base_port: 11100 },
    FleetEntry { name: "configuration", manifest_path: "configuration/Cargo.toml",
                 bin_name: "", replicas: 1, base_port: 10500 },
    FleetEntry { name: "echo",          manifest_path: "echo/Cargo.toml",
                 bin_name: "server_v1", replicas: 3, base_port: 10100 },
    FleetEntry { name: "frontend",      manifest_path: "frontend/Cargo.toml",
                 bin_name: "", replicas: 2, base_port: 8081 },
    // ... storage, caching, routing, monitoring, release
];

scheduling/src/main.rs — spawn_instance For each entry, the scheduler calls spawn_instance, which builds a cargo run command with the manifest path and passes the assigned port through the PORT environment variable. The child process ID is captured so the scheduler can later kill the instance if needed.

fn spawn_instance(&mut self, spec: &ServiceSpec, port: u16)
    -> Option<Instance>
{
    let mut cmd = Command::new("cargo");
    cmd.arg("run")
       .arg("--manifest-path").arg(&manifest);
    if !spec.bin_name.is_empty() {
        cmd.arg("--bin").arg(&spec.bin_name);
    }
    cmd.env("PORT", port.to_string());
    match cmd.spawn() {
        Ok(child) => Some(Instance {
            id, service_name: spec.name.clone(),
            port, pid: child.id(),
            status: "starting".to_string(),
        }),
        Err(e) => None,
    }
}

Health Monitoring

scheduling/src/main.rs — health_check_loop A background loop runs every five seconds, probing each instance with a TCP connection attempt. If the connection succeeds, the instance is marked healthy. If it fails, the instance is marked unhealthy. This is the simplest possible health check — a production scheduler would also check application-level health endpoints, resource usage, and response latency.

async fn health_check_loop(shared_state: Arc<Mutex<SchedulerState>>) {
    loop {
        sleep(Duration::from_secs(5)).await;
        let mut state = shared_state.lock().await;
        for instance in state.instances.iter_mut() {
            if instance.status == "stopped" { continue; }
            let addr = format!("127.0.0.1:{}", instance.port);
            match tokio::net::TcpStream::connect(&addr).await {
                Ok(_)  => { instance.status = "healthy".to_string(); }
                Err(_) => { instance.status = "unhealthy".to_string(); }
            }
        }
    }
}

RPC Interface

scheduling/src/lib.rs The scheduling service exposes five procedures through our RPC framework. SCHEDULE_SERVICE registers a new service spec and reconciles it. LIST_INSTANCES returns all running instances. SCALE_SERVICE updates the replica count. STOP_INSTANCE kills a specific instance by process ID. GET_SERVICE returns the spec and instances for one service.

pub const SCHEDULE_SERVICE_PROCEDURE: ProcedureId = 401;
pub const LIST_INSTANCES_PROCEDURE: ProcedureId = 402;
pub const SCALE_SERVICE_PROCEDURE: ProcedureId = 403;
pub const STOP_INSTANCE_PROCEDURE: ProcedureId = 404;
pub const GET_SERVICE_PROCEDURE: ProcedureId = 405;

#[derive(Debug, Serializable, Deserializable)]
pub struct ScheduleServiceArgs {
    pub name: String,
    pub manifest_path: String,
    pub bin_name: String,
    pub replicas: i32,
}

The scheduling dashboard uses these procedures to display the fleet and allow operators to scale services or stop individual instances.