Membership and events

Cluster shows which other nodes are up:

• members() lists them, and member(name) looks one up;
• is_reachable(name) tells whether this node can currently connect to it;
• wait_for_members(n) waits until exactly n other nodes are up;
• wait_until(condition) waits for any condition on the member list.

Events

subscribe() returns a stream of ClusterEvents:

A node restarted before anyone noticed shows up as Failed for the old incarnation, followed by Up for the new one.

The stream is a broadcast channel: a receiver that falls far behind misses events. To know the members right now and every change after that, use subscribe_with_snapshot(). Each change is then in either the snapshot or the stream, never both and never neither. Calling members() and subscribe() separately can miss a change in between.

    // Development only: use `Tls::from_pem` to authenticate cluster members.
    let mut config = ClusterConfig::new(name, Quic::new(bind, Tls::insecure_dev().unwrap()));
    for seed in args {
        let (seed_name, seed_addr) = seed.split_once('=').expect("seed must be name=addr");
        config = config.seed(Seed::new(
            seed_name,
            seed_addr.parse::<SocketAddr>().unwrap(),
        ));
    }

    let node = ClusterNode::new(Supervisor::blueprint().rand_name(), config);

    let mut events = node.cluster().subscribe();
    tokio::spawn(async move {
        while let Ok(event) = events.recv().await {
            tracing::info!("{event:?}");
        }
    });

    node.run().await

Run it in a few terminals to watch nodes join, and press Ctrl+C or kill -9 one of them to see the others notice it leave or fail:

cargo run -p zestors --features distr --example cluster -- node-a 127.0.0.1:7001
cargo run -p zestors --features distr --example cluster -- node-b 127.0.0.1:7002 node-a=127.0.0.1:7001
cargo run -p zestors --features distr --example cluster -- node-c 127.0.0.1:7003 node-a=127.0.0.1:7001

Failure detection

Membership uses SWIM: nodes probe each other at random, suspect a node that stops answering, and declare it down once the suspicion isn’t refuted in time. How quickly a crash is noticed depends on the membership settings. ClusterConfig::expected_size tunes the defaults for a cluster of that size, and foca_config replaces them entirely.

A node declared down that can’t rejoin becomes NodeStatus::Defunct. It keeps running, but no longer takes part in the cluster; restart it.

A node’s incarnation grows with every start, so peers can tell a restarted node from the old one. It is based on the clock. If the clock can be set back between restarts, use ClusterConfig::generation_store(path) to persist it.