Chapter 4: Configuration

Every distributed system must adapt to its environment. Server addresses change, feature flags toggle, rate limits adjust, and operational parameters shift — often while the system is running. Hardcoding these values into binaries means recompiling and redeploying every time a parameter changes. A configuration service decouples runtime parameters from the code that uses them, allowing operators to change system behavior without touching source code or restarting processes.

At its core, a configuration service is a distributed key–value store with a twist: clients don't just read values, they watch for changes. When a configuration value changes, systems that depend on it should learn of the change quickly so they can adapt. This makes configuration a foundational service — nearly every other service in a planetary scale computer depends on it.

Interface

configuration/src/lib.rs The configuration service exposes five procedures through its RPC interface. The basic operations — get, set, and delete — provide standard key–value semantics. The list operation supports prefix-based enumeration of keys, useful for discovering all configuration under a namespace like storage. or caching.. The watch operation enables clients to poll for changes to a specific key.

pub const GET_PROCEDURE: ProcedureId = 1;
pub const SET_PROCEDURE: ProcedureId = 2;
pub const DELETE_PROCEDURE: ProcedureId = 3;
pub const LIST_PROCEDURE: ProcedureId = 4;
pub const WATCH_PROCEDURE: ProcedureId = 5;

#[derive(Debug, Serializable, Deserializable)]
pub struct GetArgs {
    pub key: String,
}

#[derive(Debug, Serializable, Deserializable)]
pub struct GetResult {
    pub value: String,
}

#[derive(Debug, Serializable, Deserializable)]
pub struct SetArgs {
    pub key: String,
    pub value: String,
}

#[derive(Debug, Serializable, Deserializable)]
pub struct DeleteArgs {
    pub key: String,
}

#[derive(Debug, Serializable, Deserializable)]
pub struct ListArgs {
    pub prefix: String,
}

#[derive(Debug, Serializable, Deserializable)]
pub struct ListResult {
    pub keys: String,
}

#[derive(Debug, Serializable, Deserializable)]
pub struct WatchArgs {
    pub key: String,
}

#[derive(Debug, Serializable, Deserializable)]
pub struct WatchEvent {
    pub key: String,
    pub value: String,
}

Each struct derives Serializable and Deserializable from the normalization system, enabling them to be transmitted over the network via rpc. The ListResult returns keys as a comma-separated string — a deliberate simplicity that avoids the need for list or array normalization.

Implementation

configuration/src/main.rs The server maintains a ConfigStore containing an in-memory HashMap and a broadcast channel for notifying watchers of changes. The broadcast channel uses Tokio's broadcast::channel, which supports multiple receivers and bounded buffering.

struct ConfigStore {
    data: HashMap<String, String>,
    watchers: broadcast::Sender<(String, String)>,
}

impl ConfigStore {
    fn new() -> Self {
        let (tx, _) = broadcast::channel(256);
        ConfigStore {
            data: HashMap::new(),
            watchers: tx,
        }
    }
}

The handler for the set procedure inserts the key–value pair into the hash map and then broadcasts the change through the watcher channel. This means that any system that is watching for changes to this key will be notified as soon as the set completes:

pub async fn set(payload: &str, store: &mut ConfigStore) -> Response {
    let args = SetArgs::deserialize(payload)
        .expect("Failed to deserialize payload");
    store.data.insert(args.key.clone(), args.value.clone());
    let _ = store.watchers.send((args.key, args.value));
    Response { payload: "OK".to_string() }
}

The list handler filters keys by prefix, a pattern that allows hierarchical organization of configuration. For example, a storage service might store its settings under keys like storage.compaction_interval and storage.max_size, and retrieve all storage-related keys by listing with the prefix storage..

pub async fn list(payload: &str, store: &mut ConfigStore) -> Response {
    let args = ListArgs::deserialize(payload)
        .expect("Failed to deserialize payload");
    let keys: Vec<String> = store.data.keys()
        .filter(|k| k.starts_with(&args.prefix))
        .cloned()
        .collect();
    let result = ListResult { keys: keys.join(",") };
    Response { payload: result.serialize() }
}

The watch procedure returns the current value for a key, allowing clients to poll for changes. A more sophisticated implementation might hold the connection open and push changes, but polling keeps the RPC interface simple and stateless.

Design Discussion

Several design trade-offs are worth noting in this implementation. The in-memory store provides fast reads and writes but does not survive restarts. A production configuration service would persist its data to durable storage — in fact, it could use the storage service we will examine later.

The broadcast channel for watchers is a pragmatic choice. It decouples the notification mechanism from the storage mechanism, and its bounded buffer prevents a slow consumer from blocking writers. However, if the buffer fills up, late-arriving watchers will miss events — a trade-off that favors availability over completeness.

The prefix-based listing pattern is a common technique in configuration systems. It enables namespacing (grouping related keys) without introducing a more complex data model like directories or trees. Systems like etcd and Consul use similar prefix-based enumeration in their key–value stores.

Configuration is often the first service to start and the last to stop in a distributed system. Because almost every other service depends on configuration, its availability is critical. In a production environment, the configuration service would be replicated across multiple servers using a consensus protocol to ensure that configuration data is always available even when individual servers fail.