@Checkpoint, FlowWarden persists two independent resume tokens to MongoDB and reloads them on startup to continue exactly where it left off:
lastSeenToken— advances on every event received, even ones rejected by@Filter. Persisted by the heartbeat timer (saveIntervalSeconds).lastProcessedToken— advances only after the handler returns successfully. Persisted by the per-event counter (saveEveryN).
lastProcessedToken first, heartbeat fallback if it aged out, then onHistoryLost). The resumeStrategy attribute lets you flip the primary/secondary order to restart fast from the heartbeat-fresh token on streams where the at-least-once oplog scan is too expensive.
Quick setup
Add@Checkpoint to any @ChangeStream class:
How it works
- Happy path — handler always succeeds
- Filtered / idle stream — timer advances seen only
Saving strategies
Two independent mechanisms control when the resume token is persisted. They can be used together.Every N events — saveEveryN
Persists lastProcessedToken after every N successfully processed events. The counter increments only on handler success — failed events and events rejected by @Filter do not advance it.
Periodic timer — saveIntervalSeconds
Advances only lastSeenToken on a fixed schedule, regardless of event volume — and regardless of whether the handler ran. This acts as a heartbeat on low-traffic streams, idle streams, or streams where most events are rejected by @Filter.
The timer writes
lastSeenToken only — never lastProcessedToken. This is what preserves at-least-once delivery: if the application crashes mid-handler, lastProcessedToken still points at the last confirmed success, so the in-flight event is re-delivered on restart.Combining both
The two mechanisms target different fields of the_fw_checkpoints document — they never race or overwrite each other. The counter advances lastProcessedToken on handler success; the timer advances lastSeenToken on each tick:
Start position
startPosition controls where the stream starts consuming when it boots.
| Value | Behaviour |
|---|---|
RESUME (default) | Load the checkpoint from _fw_checkpoints. If none exists (first start), begin from the latest event. |
LATEST | Ignore any existing checkpoint and start from the latest event. Previous events are never replayed. |
Resume cascade
WhenstartPosition = RESUME, FlowWarden applies a 3-level cascade on startup to pick the resume position. Each level is tried in order — if the token is unusable (typically aged out of the oplog), it falls through to the next. Which token sits at level 1 vs level 2 is controlled by resumeStrategy.
PROCESSED_FIRST (default)
| Level | Token | Behaviour | Metric |
|---|---|---|---|
| 1 | lastProcessedToken | Strict at-least-once. In-flight events at crash time are re-delivered. | — |
| 2 | lastSeenToken | Heartbeat fallback. In-flight events are not re-delivered, but the stream avoids a ChangeStreamHistoryLost. WARN log. | flowwarden.stream.resume.fallback_to_seen |
| 3 | (none) | Apply the onHistoryLost strategy. | flowwarden.stream.resume.history_lost |
SEEN_FIRST
| Level | Token | Behaviour | Metric |
|---|---|---|---|
| 1 | lastSeenToken | Fast restart from the heartbeat-fresh token. Events past lastProcessedToken but before lastSeenToken may be skipped. | — |
| 2 | lastProcessedToken | Fallback when the seen token has aged out (rare — the heartbeat keeps it fresh). At-least-once delivery is preserved for this fallback. WARN log. | flowwarden.stream.resume.fallback_to_processed |
| 3 | (none) | Apply the onHistoryLost strategy. | flowwarden.stream.resume.history_lost |
Why the heartbeat matters
lastSeenToken is what makes the cascade levels above work. Imagine a stream that received millions of events but only a few hundred passed @Filter — without the heartbeat, lastProcessedToken is the only persisted resume position, and once it drops out of the oplog window the stream cannot resume without escalating to onHistoryLost. The heartbeat keeps lastSeenToken close to the head of the oplog at all times, so a 25-hour outage on a 24-hour-retention oplog can still resume cleanly at the cost of replaying at most a few seconds of events (under PROCESSED_FIRST, as the level-2 fallback) — or restart immediately from the heartbeat (under SEEN_FIRST, as the level-1 primary).
Cascade vs startPosition = LATEST
The cascade only runs with startPosition = RESUME (the default). With LATEST, both persisted tokens are ignored and the stream always starts from “now”.
Choosing a resume strategy
resumeStrategy decides which of the two persisted tokens the cascade tries first. The trade-off is between strict at-least-once delivery and restart cost.
When to keep the default (PROCESSED_FIRST)
- The handler causes side effects that must not be skipped (financial postings, audit log writes, outbound webhooks).
- The stream’s traffic keeps
lastProcessedTokenclose to the oplog head — the at-least-once scan is bounded. - You’d rather replay a handful of in-flight events on crash than risk skipping one.
When to consider SEEN_FIRST
- The stream emits very few events (one per week) on a busy cluster, so
lastProcessedTokenis far behind the oplog head most of the time. MongoDB has to scan the cluster’s entire activity to catch up — slow. - A
@Filteror@Pipelinerejects nearly every event, solastProcessedTokenrarely advances even though the heartbeat keepslastSeenTokenfresh. - The handler is idempotent enough that skipping the one event in flight at crash time is acceptable.
SEEN_FIRST still falls back to lastProcessedToken as the cascade level 2 before escalating to onHistoryLost, so the safety net is preserved. To skip even that fallback (e.g. on a fully ephemeral stream), combine SEEN_FIRST with onHistoryLost = RESUME_FROM_NOW.Attribute reference
| Attribute | Type | Default | Description |
|---|---|---|---|
saveEveryN | int | 1 | Save after every N successfully processed events. Must be > 0. |
saveIntervalSeconds | int | 5 | Periodic save interval in seconds. Set to 0 to disable. Must be ≥ 0. |
startPosition | StartPosition | RESUME | RESUME to reload last checkpoint; LATEST to ignore it. |
onHistoryLost | OnHistoryLost | FAIL | Strategy when both persisted tokens have expired from the oplog. See @Checkpoint reference. |
resumeStrategy | ResumeStrategy | PROCESSED_FIRST | Which token the cascade tries first: PROCESSED_FIRST (strict at-least-once) or SEEN_FIRST (fast restart). See Choosing a resume strategy. |
The MongoDB-backed storage is auto-configured. For a Redis-backed deployment, drop in the
flowwarden-redis satellite — auto-configuration takes care of the rest. To plug in a different custom store (e.g. JDBC, in-memory for tests), provide your own CheckpointStore bean — see the CheckpointStore SPI reference.Internal storage
Checkpoints are stored in the_fw_checkpoints collection of your MongoDB database. The collection is created automatically on first use.
Each document is keyed by stream name and contains the resume token and metadata:
Common patterns
Exactly-once semantics
Exactly-once semantics
FlowWarden provides at-least-once delivery by default. A crash between processing and checkpointing causes the last few events to be reprocessed on restart.To achieve exactly-once, use Spring’s See the Transactions guide for full setup instructions, constraints, and best practices.
@Transactional combined with ctx.saveCheckpointNow() to commit your business writes and the checkpoint atomically:Disabling checkpoint for dev/test
Disabling checkpoint for dev/test
Omit
@Checkpoint entirely — no tokens are persisted and the stream always starts from the latest event:Resume cascade in action — handler stuck for 30 minutes (PROCESSED_FIRST)
Resume cascade in action — handler stuck for 30 minutes (PROCESSED_FIRST)
A real-world scenario where the heartbeat saves the day:
@Checkpoint(saveEveryN = 1, saveIntervalSeconds = 5)— every success persistslastProcessedToken, the heartbeat ticks every 5 seconds.resumeStrategyis the defaultPROCESSED_FIRST.- Event E1000 arrives. The handler hits an external service that has become very slow and starts retrying. The handler does not return.
- While the handler is stuck, events E1001 through E10000 keep arriving. The heartbeat keeps advancing
lastSeenToken.lastProcessedTokenremains at E999. - After 30 minutes the application is restarted hard. The MongoDB oplog window is 25 minutes.
- On startup, the cascade tries
lastProcessedToken(E999) —ChangeStreamHistoryLost, the event has aged out. - The cascade falls back to
lastSeenToken, which the heartbeat kept fresh at E9998. The stream resumes from there with aWARNlog and a bump onflowwarden.stream.resume.fallback_to_seen.
onHistoryLost = FAIL and the stream would have refused to start, requiring operator intervention.Resume cascade in action — low-volume stream on a busy cluster (SEEN_FIRST)
Resume cascade in action — low-volume stream on a busy cluster (SEEN_FIRST)
The scenario
SEEN_FIRST is designed for:@Checkpoint(saveEveryN = 1, saveIntervalSeconds = 5, resumeStrategy = ResumeStrategy.SEEN_FIRST)— the stream emits roughly one event per week, but the MongoDB cluster handles millions of unrelated writes per hour. The heartbeat ticks every 5 seconds against the most recent oplog entry, regardless of source collection.- The last
@OnInsertran 6 days ago.lastProcessedTokenpoints there. The heartbeat has continued to advancelastSeenTokento ~5 seconds ago. - The application restarts (rolling deploy, node failure, scheduled maintenance — anything).
- Under
PROCESSED_FIRST, MongoDB would resume fromlastProcessedToken(6 days behind) and scan the entire cluster’s oplog for 6 days to find the next event for this stream — minutes of catch-up on every restart. - Under
SEEN_FIRST, MongoDB resumes fromlastSeenToken(5 seconds behind). Catch-up is instant. The trade-off: if an event was somehow in flight in the last 5 seconds and the handler hadn’t returned, that event is skipped. lastProcessedTokenis still preserved as the cascade level-2 fallback in caselastSeenTokenbecomes unusable (e.g. heartbeat was disabled and the seen token expired) — at that pointflowwarden.stream.resume.fallback_to_processedfires and the at-least-once oplog scan kicks in.
Multiple streams, independent checkpoints
Multiple streams, independent checkpoints
Each stream has its own checkpoint entry in
_fw_checkpoints, keyed by stream name. Two streams on the same collection checkpoint independently:See Also
@Checkpoint reference
Full annotation reference
How it Works
Where checkpointing fits in the event processing pipeline
Filtering Events
How filtered events interact with resume tokens