Available Modes
ALL_INSTANCES (Default)
Every instance processes every event independently. No coordination between instances.When to use
- Local cache invalidation
- In-memory state refresh
- WebSocket push notifications (each server notifies its own clients)
- Read-model / CQRS projections where each instance maintains its own view
SINGLE_LEADER
Only one instance (the leader) processes events. Other instances are on standby and automatically take over if the leader fails.How it works
1
Leader election
On startup, each instance attempts to acquire a lock in the
_fw_locks MongoDB collection using an atomic findOneAndUpdate. The first instance to succeed becomes the leader.2
Heartbeat
The leader renews its lock every 15 seconds. The lock has a 60-second TTL.
3
Standby polling
Standby instances poll every 20 seconds to check if the lock has expired.
4
Failover
If the leader crashes or loses connectivity, it stops renewing the lock. After the TTL expires, a standby instance acquires the lock and starts the stream — resuming from the last checkpoint.
5
Graceful shutdown
On application shutdown (
@PreDestroy), the leader releases the lock immediately so a standby can take over without waiting for TTL expiry.Lock document
FlowWarden stores one document per stream in the_fw_locks collection:
The
_fw_locks collection is created automatically. No migration or manual setup required. For non-MongoDB lock backends — including the Redis-backed implementation from flowwarden-redis — see the LockService SPI reference.Timing parameters
When to use
- Order processing, billing, payment webhooks
- Email/SMS notifications that must not be duplicated
- Any handler with non-idempotent side effects
- Event-sourced systems where ordering matters globally
Mixed modes
Different streams in the same application can use different deployment modes:SINGLE_LEADER stream has its own independent lock — different streams can have different leaders across your cluster.
Monitoring
When using FlowWarden Console, the stream detail page shows:- Deployment mode badge (
SINGLE LEADERorALL INSTANCES) - Leader role badge per instance (
LEADER/STANDBY) - Instance status cards with role information
StreamMetricsProvider SPI, so they appear in heartbeat data sent to the Console.
Comparison
Horizontal scaling beyond these modes
FlowWarden does not partition the change stream across worker instances. If you need event-level parallelism beyond whatSINGLE_LEADER gives you on a single MongoDB collection, the right tools live outside the lib:
- MongoDB sharding upstream — transparent to FlowWarden; the global change stream still works.
- Multiple
@ChangeStreamclasses on different collections — already supported, each runs independently. - Future sink modules (Kafka, RabbitMQ, …) — emit events to a broker and let the broker handle key-based parallelism via consumer groups.
Checkpoint & Resume
Ensure the new leader resumes from the correct position after failover.
Retry & DLQ
Handle failures gracefully with automatic retries and dead letter queues.