Skip to main content
The @Checkpoint annotation enables automatic persistence of MongoDB Change Stream resume tokens with at-least-once delivery guarantees. FlowWarden tracks two anchors of different natures — lastProcessedToken (the work anchor, advancing on every terminally settled event) and lastSeenToken (the position anchor, holding exclusively server-certified positions written by the idle heartbeat) — and applies a fixed 3-level resume cascade on restart.

Basic Usage

With the default configuration, a checkpoint is saved after every settled event and the stream resumes from the last processed anchor on restart.

Attributes

Checkpoints are stored in the _fw_checkpoints collection in your MongoDB database. The collection name is not configurable.

saveEveryN

The count threshold. A settlement is any terminal outcome: handler success, rejection by @Filter, an event with no matching handler, or a terminal skip/DLQ decision. An event still being retried (or whose retry backoff was interrupted by a shutdown) never counts — it stays replayable. Higher values reduce write pressure at the cost of replaying up to N-1 settled events on crash recovery.
saveEveryN < 1 is rejected at startup with a BeanCreationException.
The N-1 replay bound describes nominal operation. The counter means “settlements since the last confirmed persistence” and resets only on a successful write: after a failed write, the policy stays due and the very next settlement (or the timer) retries — but until a write is confirmed the replayable tail keeps growing. The bound is restored by the first confirmed write, not maintained through sustained store outages.

saveIntervalSeconds

The time threshold of the same policy. On an active-but-slow stream — one that settles events regularly but too rarely for the count threshold — this caps the anchor’s age in wall-clock time, protecting it against oplog aging that a pure count threshold cannot see. On a stream that settles nothing at all it writes nothing: keeping an idle stream recoverable is the job of idleHeartbeatIntervalSeconds.
saveEveryN and saveIntervalSeconds are two thresholds of one policy over the same anchor — the write happens at whichever is reached first, and every write also refreshes lastHeartbeatTimestamp (a token just delivered from the oplog re-confirms the recoverable position). All anchor writers, the manual ctx.saveCheckpointNow() included, are serialized per stream: a slow write of an older token can never overwrite a newer one.

idleHeartbeatIntervalSeconds

The oplog-rollover protection for streams that stop settling events, and the sole writer of lastSeenToken. When the main cursor has settled nothing for this long, FlowWarden opens an ephemeral, bounded probe chained from the last settled position — same collection, same resolved @Pipeline stages and fullDocument options as the stream, stamped with a flowwarden:heartbeat:<stream> comment for $currentOp attribution. When the server certifies the interval empty, the returned post-batch resume token is persisted as lastSeenToken: the resume point keeps tracking the oplog head with zero traffic.
A delivered event token is never written as lastSeenToken, and a certified position never crosses unsettled work: if events sit undelivered, the probe abstains and writes nothing (repeated abstentions log a WARN, and feed stuckAfterProbes). Both rules together make resume safety structural — a restart that replays old events cannot regress the certified position.
The interval must stay well below the expected oplog window, with margin for transient probe failures. 300 (default) is a reasonable trade-off for typical windows, not a guarantee against an oplog that rolls faster.

stuckAfterProbes

Stuck-stream detection (since 1.0.0-rc.7). A cursor can stop delivering without the driver ever saying so: a half-open connection with the client’s default socketTimeoutMS = 0 leaves the getMore hanging forever, in both modes. The stream then reports running, the idle probe keeps its resume point fresh, and nothing is processed. The idle probe already sees the symptom. When it abstains, matching events exist that the stream has not delivered. stuckAfterProbes turns a streak of such abstentions into a verdict:
  • only abstentions with no delivery in between count — any settled event resets the streak;
  • only idle probes count — the startup diagnostic and recovery probes never do;
  • a stream with a handler in flight is never declared stuck. A slow handler or a retry in backoff holds the settled position still exactly like a dead cursor would, and must not be restarted mid-handler; its settlement resets the streak, a dead cursor’s never will.
The verdict takes the runtime-death path: the stream is reported stopped with CRASHED and a StreamStuckException as cause, then resubscribed on the @RestartPolicy cadence, exactly like after a cursor error. Where it resumes is startPosition’s call, as for every restart: under RESUME the cascade restarts from lastProcessedToken and nothing is lost; under LATEST the new cursor starts from now and the events the hung cursor never delivered are skipped — the loss LATEST accepts on any restart.
Detection takes about stuckAfterProbes × idleHeartbeatIntervalSeconds (15 minutes with the defaults) and needs traffic on the collection: a dead cursor with nothing to deliver is undetectable, and harmless. Implicitly off when idleHeartbeatIntervalSeconds = 0.
The hung reading thread may stay blocked inside the driver until the client’s socket timeout. Setting socketTimeoutMS (or timeoutMS) on the Mongo client remains the right companion — the verdict frees the stream, not the thread.

startPosition

Determines where the stream starts consuming when it is first created or when no checkpoint exists.

Resume cascade

On restart with startPosition = RESUME, FlowWarden applies a fixed 3-level cascade to decide where to resume the Change Stream: The cascade only runs when startPosition = RESUME (the default). With startPosition = LATEST, both persisted anchors are ignored and the stream starts from the current moment.
Checkpoints written by versions before 1.0.0-rc.4 may hold an event token in lastSeenToken; it remains a valid level-2 resume position, and the field adopts the certified-only semantics at the first successful certification.

onHistoryLost

Determines the recovery strategy at cascade level 3 — when neither anchor can resume the stream. With the cascade in place, reaching level 3 means the downtime exceeded the oplog window and idle certification was disabled (or also aged out), which is genuinely abnormal.
FAIL is the default because silent data loss is worse than a stream that refuses to start. Only use RESUME_FROM_NOW or RESUME_FROM_OPLOG_START for streams where missing events is acceptable (e.g., cache invalidation, non-critical notifications).
When onHistoryLost = FAIL, the framework throws a HistoryLostException with an actionable message:
The exception message includes the stream name, the most recent known checkpoint activity, and the operator’s escape hatch (delete the stream’s _fw_checkpoints document, or switch strategy).

Comprehensive Example

How It Works

FlowWarden uses a dual-anchor checkpoint model internally: This separation enables at-least-once delivery: if the application crashes after receiving an event but before it settles, the stream resumes from lastProcessedToken on the next start (cascade level 1) and the in-flight event is re-delivered. And because the position anchor only ever holds certified positions, a resume that replays old events can never corrupt it.
When a @Pipeline filters out most events server-side, the stream may settle nothing for long stretches even on a busy collection. The idle heartbeat then takes over: its probe replicates the exact pipeline, so a certified “interval empty” genuinely means no matching events were skipped, and lastSeenToken keeps tracking the oplog head.Without this mechanism, a restart would force MongoDB to re-scan the oplog from lastProcessedToken, potentially scanning tens of thousands of filtered-out events — or lose the position entirely once it ages out.See the @Pipeline reference for details.

Storage

Checkpoints are persisted via the CheckpointStore SPI. The MongoDB-backed implementation (MongoCheckpointStore / ReactiveMongoCheckpointStore) is auto-configured by default; a Redis-backed alternative ships in flowwarden-redis. Custom backends (JDBC, in-memory, …) plug in by declaring a @Bean CheckpointStore.

Best Practices

  • Keep saveEveryN = 1 for critical streams (e.g., order processing, billing) to minimize event replay on crash.
  • Increase saveEveryN for high-throughput streams where replaying a few events is acceptable — saveIntervalSeconds still bounds the anchor’s age in time.
  • Do not disable idleHeartbeatIntervalSeconds unless you have a specific reason. It is the only mechanism that keeps an idle stream’s resume point inside the oplog window.
  • Use startPosition = LATEST only for streams where historical events are irrelevant (e.g., cache invalidation).
  • Alert on the age of lastHeartbeatTimestamp — it records the last confirmed recoverable position (anchor write or certification) and never moves on probe abstentions: a frozen value signals a stuck stream or a failing store.
  • Alert on the onResumeFallbackToSeen callback (surfaced by the Reporter as a resume outcome, or by your own StreamMetricsProvider) — a recurring level-2 fallback on an active stream signals that the processed anchor ages out faster than the stream settles, typically a stuck or retry-pathological handler.
  • Alert on the onResumeHistoryLost callback — reaching level 3 means downtime overran the oplog window with no usable fallback; investigate operationally rather than silencing it.
MongoDB must be configured as a Replica Set for Change Streams (and therefore checkpointing) to work. This applies to development environments as well. Testcontainers automatically provisions a single-node Replica Set for testing.

See Also

Checkpoint & Resume Guide

Step-by-step guide to configuring checkpoint and resume behavior

@ChangeStream

The main annotation for declaring Change Stream handlers

@Pipeline

Server-side filtering and its interaction with dual-anchor checkpointing

@RetryPolicy

Configure retry behavior for failed event processing