Chapter 12: Release

Software that never changes is software that is never improved. The release service manages rolling deployments across the fleet, coordinating with the scheduler to replace instances one batch at a time with zero downtime.

Interface

release/src/lib.rs The release service exposes five procedures. CREATE_RELEASE starts a new deployment. GET_RELEASE and LIST_RELEASES inspect state. ADVANCE_RELEASE pushes the deployment forward one batch. ROLLBACK reverts a deployment in progress.

pub const CREATE_RELEASE_PROCEDURE: ProcedureId = 501;
pub const GET_RELEASE_PROCEDURE: ProcedureId = 502;
pub const LIST_RELEASES_PROCEDURE: ProcedureId = 503;
pub const ADVANCE_RELEASE_PROCEDURE: ProcedureId = 504;
pub const ROLLBACK_PROCEDURE: ProcedureId = 505;

#[derive(Debug, Serializable, Deserializable)]
pub struct CreateReleaseArgs {
    pub service: String,
    pub version: String,
    pub description: String,
}

Release Lifecycle

release/src/main.rs A release progresses through a simple state machine: created → deploying → deployed (or rolled_back). Each state transition is explicit — an operator advances the release through the release dashboard, giving them control over the pace of the rollout.

struct Release {
    id: String,
    service: String,
    version: String,
    description: String,
    status: String,         // "created", "deploying", "deployed", "rolled_back"
    old_instances: Vec<String>,
    new_instances: Vec<String>,
    batch_progress: i32,
}

Rolling Updates

release/src/main.rs — create_release When a release is created, the service queries the scheduler for the current instances of the target service and snapshots them as the “old” instances. It calculates the batch size as max(1, total / 10) — roughly 10% of the fleet per batch, with a minimum of one instance.

let svc_result = scheduling::get_service(
    SCHEDULER_ADDR, args.service.clone()
).await;
let old_instances: Vec<String> = svc_result.instances
    .split(';').map(|s| s.to_string()).collect();
let total = old_instances.len() as i32;
let batch_size = std::cmp::max(1, total / 10);

release/src/main.rs — advance_release Each call to ADVANCE_RELEASE replaces one batch. The handler tells the scheduler to scale up, waits for health, then stops one old instance:

1. Tell the scheduler to spawn new instances for the batch
2. Wait for the new instances to pass health checks
3. Tell the scheduler to stop the corresponding old instances
4. Old instances deregister from discovery via stale cleanup

// Scale up: add one replica
scheduling::scale_service(
    SCHEDULER_ADDR, release.service.clone(), current_total + 1,
).await;

// Wait for new instance to become healthy
sleep(Duration::from_secs(2)).await;

// Scale down: stop one old instance
let old_id = release.old_instances.remove(0);
scheduling::stop_instance(SCHEDULER_ADDR, old_id).await;

release.batch_progress += 1;
if release.old_instances.is_empty() {
    release.status = "deployed".to_string();
}

This process ensures that at every moment during the rollout, the service has enough healthy instances to handle traffic. The routing layer and discovery service automatically direct traffic away from stopped instances and toward new ones.

Rollback

release/src/main.rs — rollback If a deployment goes wrong, the ROLLBACK procedure marks the active release as rolled_back. Because the old instances are only stopped after new ones are confirmed healthy, a rollback during deployment simply stops the process — the remaining old instances continue serving traffic. The key to fast rollback is never removing the old before the new is proven.

Integration with the Scheduler

release/src/lib.rs The release service does not spawn processes directly. Instead, it delegates all process management to the scheduler through RPC calls: scheduling::scale_service() to add replicas and scheduling::stop_instance() to remove them. This separation of concerns means the scheduler remains the single source of truth for what is running, while the release service manages the order and pace of changes. Visit the release dashboard to create a release and step through a rolling deployment.