@Checkpoint annotation enables automatic persistence of MongoDB Change Stream resume tokens with at-least-once delivery guarantees. FlowWarden tracks two anchors of different natures — lastProcessedToken (the work anchor, advancing on every terminally settled event) and lastSeenToken (the position anchor, holding exclusively server-certified positions written by the idle heartbeat) — and applies a fixed 3-level resume cascade on restart.
Basic Usage
Attributes
Checkpoints are stored in the
_fw_checkpoints collection in your MongoDB database. The collection name is not configurable.saveEveryN
The count threshold. A settlement is any terminal outcome: handler success, rejection by @Filter, an event with no matching handler, or a terminal skip/DLQ decision. An event still being retried (or whose retry backoff was interrupted by a shutdown) never counts — it stays replayable. Higher values reduce write pressure at the cost of replaying up to N-1 settled events on crash recovery.
The
N-1 replay bound describes nominal operation. The counter means “settlements since the last confirmed persistence” and resets only on a successful write: after a failed write, the policy stays due and the very next settlement (or the timer) retries — but until a write is confirmed the replayable tail keeps growing. The bound is restored by the first confirmed write, not maintained through sustained store outages.saveIntervalSeconds
The time threshold of the same policy. On an active-but-slow stream — one that settles events regularly but too rarely for the count threshold — this caps the anchor’s age in wall-clock time, protecting it against oplog aging that a pure count threshold cannot see. On a stream that settles nothing at all it writes nothing: keeping an idle stream recoverable is the job of idleHeartbeatIntervalSeconds.
idleHeartbeatIntervalSeconds
The oplog-rollover protection for streams that stop settling events, and the sole writer of lastSeenToken. When the main cursor has settled nothing for this long, FlowWarden opens an ephemeral, bounded probe chained from the last settled position — same collection, same resolved @Pipeline stages and fullDocument options as the stream, stamped with a flowwarden:heartbeat:<stream> comment for $currentOp attribution. When the server certifies the interval empty, the returned post-batch resume token is persisted as lastSeenToken: the resume point keeps tracking the oplog head with zero traffic.
A delivered event token is never written as
lastSeenToken, and a certified position never crosses unsettled work: if events sit undelivered, the probe abstains and writes nothing (repeated abstentions log a WARN, and feed stuckAfterProbes). Both rules together make resume safety structural — a restart that replays old events cannot regress the certified position.stuckAfterProbes
Stuck-stream detection (since 1.0.0-rc.7). A cursor can stop delivering without the driver ever saying so: a half-open connection with the client’s default socketTimeoutMS = 0 leaves the getMore hanging forever, in both modes. The stream then reports running, the idle probe keeps its resume point fresh, and nothing is processed.
The idle probe already sees the symptom. When it abstains, matching events exist that the stream has not delivered. stuckAfterProbes turns a streak of such abstentions into a verdict:
- only abstentions with no delivery in between count — any settled event resets the streak;
- only idle probes count — the startup diagnostic and recovery probes never do;
- a stream with a handler in flight is never declared stuck. A slow handler or a retry in backoff holds the settled position still exactly like a dead cursor would, and must not be restarted mid-handler; its settlement resets the streak, a dead cursor’s never will.
CRASHED and a StreamStuckException as cause, then resubscribed on the @RestartPolicy cadence, exactly like after a cursor error. Where it resumes is startPosition’s call, as for every restart: under RESUME the cascade restarts from lastProcessedToken and nothing is lost; under LATEST the new cursor starts from now and the events the hung cursor never delivered are skipped — the loss LATEST accepts on any restart.
Detection takes about
stuckAfterProbes × idleHeartbeatIntervalSeconds (15 minutes with the defaults) and needs traffic on the collection: a dead cursor with nothing to deliver is undetectable, and harmless. Implicitly off when idleHeartbeatIntervalSeconds = 0.startPosition
Determines where the stream starts consuming when it is first created or when no checkpoint exists.
Resume cascade
On restart withstartPosition = RESUME, FlowWarden applies a fixed 3-level cascade to decide where to resume the Change Stream:
The cascade only runs when
startPosition = RESUME (the default). With startPosition = LATEST, both persisted anchors are ignored and the stream starts from the current moment.
Checkpoints written by versions before 1.0.0-rc.4 may hold an event token in
lastSeenToken; it remains a valid level-2 resume position, and the field adopts the certified-only semantics at the first successful certification.onHistoryLost
Determines the recovery strategy at cascade level 3 — when neither anchor can resume the stream. With the cascade in place, reaching level 3 means the downtime exceeded the oplog window and idle certification was disabled (or also aged out), which is genuinely abnormal.
HistoryLostException
HistoryLostException
When The exception message includes the stream name, the most recent known checkpoint activity, and the operator’s escape hatch (delete the stream’s
onHistoryLost = FAIL, the framework throws a HistoryLostException with an actionable message:_fw_checkpoints document, or switch strategy).Comprehensive Example
How It Works
FlowWarden uses a dual-anchor checkpoint model internally:
This separation enables at-least-once delivery: if the application crashes after receiving an event but before it settles, the stream resumes from
lastProcessedToken on the next start (cascade level 1) and the in-flight event is re-delivered. And because the position anchor only ever holds certified positions, a resume that replays old events can never corrupt it.
- Crash mid-handler (cascade level 1)
- Filter-heavy stream (settlements anchor)
- Idle stream (certification + level 2)
Interaction with @Pipeline
Interaction with @Pipeline
When a
@Pipeline filters out most events server-side, the stream may settle nothing for long stretches even on a busy collection. The idle heartbeat then takes over: its probe replicates the exact pipeline, so a certified “interval empty” genuinely means no matching events were skipped, and lastSeenToken keeps tracking the oplog head.Without this mechanism, a restart would force MongoDB to re-scan the oplog from lastProcessedToken, potentially scanning tens of thousands of filtered-out events — or lose the position entirely once it ages out.See the @Pipeline reference for details.Storage
Checkpoints are persisted via theCheckpointStore SPI. The MongoDB-backed implementation (MongoCheckpointStore / ReactiveMongoCheckpointStore) is auto-configured by default; a Redis-backed alternative ships in flowwarden-redis. Custom backends (JDBC, in-memory, …) plug in by declaring a @Bean CheckpointStore.
Best Practices
- Keep
saveEveryN = 1for critical streams (e.g., order processing, billing) to minimize event replay on crash. - Increase
saveEveryNfor high-throughput streams where replaying a few events is acceptable —saveIntervalSecondsstill bounds the anchor’s age in time. - Do not disable
idleHeartbeatIntervalSecondsunless you have a specific reason. It is the only mechanism that keeps an idle stream’s resume point inside the oplog window. - Use
startPosition = LATESTonly for streams where historical events are irrelevant (e.g., cache invalidation). - Alert on the age of
lastHeartbeatTimestamp— it records the last confirmed recoverable position (anchor write or certification) and never moves on probe abstentions: a frozen value signals a stuck stream or a failing store. - Alert on the
onResumeFallbackToSeencallback (surfaced by the Reporter as a resume outcome, or by your ownStreamMetricsProvider) — a recurring level-2 fallback on an active stream signals that the processed anchor ages out faster than the stream settles, typically a stuck or retry-pathological handler. - Alert on the
onResumeHistoryLostcallback — reaching level 3 means downtime overran the oplog window with no usable fallback; investigate operationally rather than silencing it.
See Also
Checkpoint & Resume Guide
Step-by-step guide to configuring checkpoint and resume behavior
@ChangeStream
The main annotation for declaring Change Stream handlers
@Pipeline
Server-side filtering and its interaction with dual-anchor checkpointing
@RetryPolicy
Configure retry behavior for failed event processing