> ## Documentation Index > Fetch the complete documentation index at: https://docs.flowwarden.io/llms.txt > Use this file to discover all available pages before exploring further. # Checkpoint & Resume > Persist resume tokens so streams survive restarts without missing or replaying events Without checkpointing, a restarted application starts consuming from the latest MongoDB event — all events that arrived during the downtime are silently lost. With `@Checkpoint`, FlowWarden persists **two independent resume tokens** to MongoDB and reloads them on startup to continue exactly where it left off: * **`lastSeenToken`** — advances on every event received, even ones rejected by `@Filter`. Persisted by the heartbeat timer (`saveIntervalSeconds`). * **`lastProcessedToken`** — advances only after the handler returns successfully. Persisted by the per-event counter (`saveEveryN`). On restart, a 3-level cascade decides which token wins. The default is strict at-least-once (`lastProcessedToken` first, heartbeat fallback if it aged out, then `onHistoryLost`). The `resumeStrategy` attribute lets you flip the primary/secondary order to restart fast from the heartbeat-fresh token on streams where the at-least-once oplog scan is too expensive. *** ## Quick setup Add `@Checkpoint` to any `@ChangeStream` class: ```java theme={null} @ChangeStream(documentType = Order.class) @Checkpoint public class OrderStream { @OnChange void handle(ChangeStreamContext ctx) { System.out.println(ctx.summary()); } } ``` That's it. With default settings, the resume token is saved after every processed event and reloaded automatically on the next startup. *** ## How it works ```mermaid theme={null} sequenceDiagram participant App as Application participant FW as FlowWarden participant DB as MongoDB App->>FW: startup FW->>DB: load tokens from _fw_checkpoints DB-->>FW: (lastProcessedToken, lastSeenToken) FW->>DB: subscribe Change Stream from lastProcessedToken loop for each event DB-->>FW: change event FW->>FW: in-memory: lastSeen = event.token FW->>App: invoke handler App-->>FW: success FW->>DB: saveProcessed(event.token) Note over FW,DB: lastProcessedToken advances
(every N events) end App->>FW: restart FW->>DB: load tokens DB-->>FW: lastProcessedToken FW->>DB: resume from lastProcessedToken Note over FW,DB: no events missed, no replays ``` ```mermaid theme={null} sequenceDiagram participant DB as MongoDB participant FW as FlowWarden participant T as Heartbeat timer participant CP as _fw_checkpoints DB-->>FW: event E1 (filtered out by @Filter) FW->>FW: in-memory: lastSeen = E1 Note over FW: handler NOT invoked T->>FW: tick (saveIntervalSeconds) FW->>CP: saveSeen(E1) Note over CP: lastSeenToken = E1
lastProcessedToken unchanged DB-->>FW: event E2 (filtered out) FW->>FW: in-memory: lastSeen = E2 T->>FW: tick FW->>CP: saveSeen(E2) ``` *** ## Saving strategies Two independent mechanisms control when the resume token is persisted. They can be used together. ### Every N events — `saveEveryN` Persists **`lastProcessedToken`** after every N **successfully processed** events. The counter increments only on handler success — failed events and events rejected by `@Filter` do not advance it. ```java theme={null} // Save after every event (default — safest, small overhead) @Checkpoint // Save every 10 events (lower write pressure, up to 9 events at risk on crash) @Checkpoint(saveEveryN = 10) ``` With `saveEveryN = 10`, a crash between checkpoints means the last up-to-9 events are **reprocessed** on restart. Make sure your handlers are idempotent if you increase this value. `saveEveryN < 1` is rejected at startup with a `BeanCreationException`. ### Periodic timer — `saveIntervalSeconds` Advances **only `lastSeenToken`** on a fixed schedule, regardless of event volume — and regardless of whether the handler ran. This acts as a **heartbeat** on low-traffic streams, idle streams, or streams where most events are rejected by `@Filter`. ```java theme={null} // Save every 5 seconds (default) @Checkpoint // Save every 30 seconds @Checkpoint(saveIntervalSeconds = 30) // Disable the heartbeat entirely (also disables cascade level 2) @Checkpoint(saveIntervalSeconds = 0) ``` **The timer writes `lastSeenToken` only — never `lastProcessedToken`.** This is what preserves at-least-once delivery: if the application crashes mid-handler, `lastProcessedToken` still points at the last confirmed success, so the in-flight event is re-delivered on restart. Combining `saveIntervalSeconds = 0` with `saveEveryN > 1` emits a startup warning — the combination disables both heartbeat and cascade level 2, leaving the stream with no safety net if `lastProcessedToken` ages out. ### Combining both The two mechanisms target **different fields** of the `_fw_checkpoints` document — they never race or overwrite each other. The counter advances `lastProcessedToken` on handler success; the timer advances `lastSeenToken` on each tick: ```java theme={null} // lastProcessedToken every 50 successes; lastSeenToken every 10 seconds @Checkpoint(saveEveryN = 50, saveIntervalSeconds = 10) ``` For high-throughput streams, increase `saveEveryN` to reduce write pressure and keep `saveIntervalSeconds` low so the heartbeat stays fresh. For low-traffic or filter-heavy streams, `saveIntervalSeconds` does most of the work. *** ## Start position `startPosition` controls where the stream starts consuming when it boots. | Value | Behaviour | | -------------------- | ------------------------------------------------------------------------------------------------------ | | `RESUME` *(default)* | Load the checkpoint from `_fw_checkpoints`. If none exists (first start), begin from the latest event. | | `LATEST` | Ignore any existing checkpoint and start from the latest event. Previous events are never replayed. | ```java theme={null} // Resume from last checkpoint (production default) @Checkpoint(startPosition = StartPosition.RESUME) // Always start fresh — useful for stateless consumers (e.g. cache warming) @Checkpoint(startPosition = StartPosition.LATEST) ``` `startPosition = LATEST` **deletes the effective checkpoint** on each restart. Use it only for streams where missing past events is acceptable (e.g. cache invalidation, live dashboards). *** ## Resume cascade When `startPosition = RESUME`, FlowWarden applies a 3-level cascade on startup to pick the resume position. Each level is tried in order — if the token is unusable (typically aged out of the oplog), it falls through to the next. Which token sits at level 1 vs level 2 is controlled by [`resumeStrategy`](#choosing-a-resume-strategy). ### `PROCESSED_FIRST` (default) | Level | Token | Behaviour | Metric | | ----- | -------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | | 1 | `lastProcessedToken` | Strict at-least-once. In-flight events at crash time are re-delivered. | — | | 2 | `lastSeenToken` | Heartbeat fallback. In-flight events are **not** re-delivered, but the stream avoids a `ChangeStreamHistoryLost`. WARN log. | `flowwarden.stream.resume.fallback_to_seen` | | 3 | (none) | Apply the `onHistoryLost` strategy. | `flowwarden.stream.resume.history_lost` | ### `SEEN_FIRST` | Level | Token | Behaviour | Metric | | ----- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | | 1 | `lastSeenToken` | Fast restart from the heartbeat-fresh token. Events past `lastProcessedToken` but before `lastSeenToken` may be **skipped**. | — | | 2 | `lastProcessedToken` | Fallback when the seen token has aged out (rare — the heartbeat keeps it fresh). At-least-once delivery is preserved for this fallback. WARN log. | `flowwarden.stream.resume.fallback_to_processed` | | 3 | (none) | Apply the `onHistoryLost` strategy. | `flowwarden.stream.resume.history_lost` | ### Why the heartbeat matters `lastSeenToken` is what makes the cascade levels above work. Imagine a stream that received millions of events but only a few hundred passed `@Filter` — without the heartbeat, `lastProcessedToken` is the only persisted resume position, and once it drops out of the oplog window the stream cannot resume without escalating to `onHistoryLost`. The heartbeat keeps `lastSeenToken` close to the head of the oplog at all times, so a 25-hour outage on a 24-hour-retention oplog can still resume cleanly at the cost of replaying at most a few seconds of events (under `PROCESSED_FIRST`, as the level-2 fallback) — or restart immediately from the heartbeat (under `SEEN_FIRST`, as the level-1 primary). ### Cascade vs `startPosition = LATEST` The cascade only runs with `startPosition = RESUME` (the default). With `LATEST`, both persisted tokens are ignored and the stream always starts from "now". *** ## Choosing a resume strategy `resumeStrategy` decides which of the two persisted tokens the cascade tries **first**. The trade-off is between strict at-least-once delivery and restart cost. ```java theme={null} // Default — strict at-least-once. Critical streams (billing, audit, side-effects). @Checkpoint(resumeStrategy = ResumeStrategy.PROCESSED_FIRST) // Fast restart from the heartbeat token. Low-volume / filter-heavy streams. @Checkpoint(resumeStrategy = ResumeStrategy.SEEN_FIRST) ``` ### When to keep the default (`PROCESSED_FIRST`) * The handler causes side effects that must not be skipped (financial postings, audit log writes, outbound webhooks). * The stream's traffic keeps `lastProcessedToken` close to the oplog head — the at-least-once scan is bounded. * You'd rather replay a handful of in-flight events on crash than risk skipping one. ### When to consider `SEEN_FIRST` * The stream emits very few events (one per week) on a busy cluster, so `lastProcessedToken` is far behind the oplog head most of the time. MongoDB has to scan the cluster's entire activity to catch up — slow. * A `@Filter` or `@Pipeline` rejects nearly every event, so `lastProcessedToken` rarely advances even though the heartbeat keeps `lastSeenToken` fresh. * The handler is idempotent enough that skipping the one event in flight at crash time is acceptable. `SEEN_FIRST` still falls back to `lastProcessedToken` as the cascade level 2 before escalating to `onHistoryLost`, so the safety net is preserved. To skip even that fallback (e.g. on a fully ephemeral stream), combine `SEEN_FIRST` with `onHistoryLost = RESUME_FROM_NOW`. ```mermaid theme={null} flowchart LR A[Low-volume stream?] -->|no| P[PROCESSED_FIRST] A -->|yes| B[Handler causes
side effects?] B -->|yes| P B -->|no| C[Idempotent enough
to skip in-flight events?] C -->|no| P C -->|yes| S[SEEN_FIRST] ``` *** ## Attribute reference ```java theme={null} @Checkpoint( saveEveryN = 1, // checkpoint every N events saveIntervalSeconds = 5, // periodic heartbeat (0 = disabled) startPosition = StartPosition.RESUME, // where to start on boot onHistoryLost = OnHistoryLost.FAIL, // when both tokens expire from oplog resumeStrategy = ResumeStrategy.PROCESSED_FIRST // which token the cascade tries first ) ``` | Attribute | Type | Default | Description | | --------------------- | ---------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `saveEveryN` | `int` | `1` | Save after every N successfully processed events. Must be > 0. | | `saveIntervalSeconds` | `int` | `5` | Periodic save interval in seconds. Set to `0` to disable. Must be ≥ 0. | | `startPosition` | `StartPosition` | `RESUME` | `RESUME` to reload last checkpoint; `LATEST` to ignore it. | | `onHistoryLost` | `OnHistoryLost` | `FAIL` | Strategy when **both** persisted tokens have expired from the oplog. See [@Checkpoint reference](/reference/checkpoint#onhistorylost). | | `resumeStrategy` | `ResumeStrategy` | `PROCESSED_FIRST` | Which token the cascade tries first: `PROCESSED_FIRST` (strict at-least-once) or `SEEN_FIRST` (fast restart). See [Choosing a resume strategy](#choosing-a-resume-strategy). | The MongoDB-backed storage is auto-configured. For a Redis-backed deployment, drop in the [`flowwarden-redis`](/redis) satellite — auto-configuration takes care of the rest. To plug in a different custom store (e.g. JDBC, in-memory for tests), provide your own `CheckpointStore` bean — see the [`CheckpointStore` SPI reference](/reference/checkpoint-store). *** ## Internal storage Checkpoints are stored in the `_fw_checkpoints` collection of your MongoDB database. The collection is created automatically on first use. Each document is keyed by stream name and contains the resume token and metadata: ```json theme={null} { "_id": "order-stream", "instanceId": "app-1", "lastSeenToken": { "$binary": { ... } }, "lastSeenTimestamp": "2026-02-25T10:00:01Z", "lastProcessedToken": { "$binary": { ... } }, "lastProcessedTimestamp": "2026-02-25T10:00:00Z", "metadata": {} } ``` You can inspect checkpoints directly, or delete a stream's checkpoint to force a reset on the next restart: ```js theme={null} // Inspect all checkpoints db._fw_checkpoints.find() // Force reset a specific stream db._fw_checkpoints.deleteOne({ _id: "order-stream" }) ``` *** ## Common patterns FlowWarden provides **at-least-once** delivery by default. A crash between processing and checkpointing causes the last few events to be reprocessed on restart. To achieve exactly-once, use Spring's `@Transactional` combined with `ctx.saveCheckpointNow()` to commit your business writes and the checkpoint atomically: ```java theme={null} @OnInsert @Transactional void onInsert(Order order, ChangeStreamContext ctx) { // business write + checkpoint in one MongoDB transaction mongoTemplate.save(order, "processed_orders"); ctx.saveCheckpointNow(); // included in the same transaction } ``` See the [Transactions guide](/guides/transactions) for full setup instructions, constraints, and best practices. Omit `@Checkpoint` entirely — no tokens are persisted and the stream always starts from the latest event: ```java theme={null} @ChangeStream(collection = "orders") public class OrderStream { // no @Checkpoint — stateless, always starts from latest @OnChange void handle(ChangeStreamContext ctx) { ... } } ``` A real-world scenario where the heartbeat saves the day: 1. `@Checkpoint(saveEveryN = 1, saveIntervalSeconds = 5)` — every success persists `lastProcessedToken`, the heartbeat ticks every 5 seconds. `resumeStrategy` is the default `PROCESSED_FIRST`. 2. Event E1000 arrives. The handler hits an external service that has become **very** slow and starts retrying. The handler does not return. 3. While the handler is stuck, events E1001 through E10000 keep arriving. The heartbeat keeps advancing `lastSeenToken`. `lastProcessedToken` remains at E999. 4. After 30 minutes the application is restarted hard. The MongoDB oplog window is 25 minutes. 5. On startup, the cascade tries `lastProcessedToken` (E999) — `ChangeStreamHistoryLost`, the event has aged out. 6. The cascade falls back to `lastSeenToken`, which the heartbeat kept fresh at E9998. The stream resumes from there with a `WARN` log and a bump on `flowwarden.stream.resume.fallback_to_seen`. Without the heartbeat, step 6 would have escalated to `onHistoryLost = FAIL` and the stream would have refused to start, requiring operator intervention. The scenario `SEEN_FIRST` is designed for: 1. `@Checkpoint(saveEveryN = 1, saveIntervalSeconds = 5, resumeStrategy = ResumeStrategy.SEEN_FIRST)` — the stream emits roughly one event per week, but the MongoDB cluster handles millions of unrelated writes per hour. The heartbeat ticks every 5 seconds against the most recent oplog entry, regardless of source collection. 2. The last `@OnInsert` ran 6 days ago. `lastProcessedToken` points there. The heartbeat has continued to advance `lastSeenToken` to \~5 seconds ago. 3. The application restarts (rolling deploy, node failure, scheduled maintenance — anything). 4. Under `PROCESSED_FIRST`, MongoDB would resume from `lastProcessedToken` (6 days behind) and scan **the entire cluster's oplog for 6 days** to find the next event for this stream — minutes of catch-up on every restart. 5. Under `SEEN_FIRST`, MongoDB resumes from `lastSeenToken` (5 seconds behind). Catch-up is instant. The trade-off: if an event was somehow in flight in the last 5 seconds and the handler hadn't returned, that event is skipped. 6. `lastProcessedToken` is still preserved as the cascade level-2 fallback in case `lastSeenToken` becomes unusable (e.g. heartbeat was disabled and the seen token expired) — at that point `flowwarden.stream.resume.fallback_to_processed` fires and the at-least-once oplog scan kicks in. Each stream has its own checkpoint entry in `_fw_checkpoints`, keyed by stream name. Two streams on the same collection checkpoint independently: ```java theme={null} @ChangeStream(name = "billing-stream", collection = "orders") @Checkpoint(saveEveryN = 1) public class BillingStream { ... } @ChangeStream(name = "notification-stream", collection = "orders") @Checkpoint(saveEveryN = 50, saveIntervalSeconds = 10) public class NotificationStream { ... } ``` *** ## See Also Full annotation reference Where checkpointing fits in the event processing pipeline How filtered events interact with resume tokens