Checkpoint & Resume

Without checkpointing, a restarted application starts consuming from the latest MongoDB event — all events that arrived during the downtime are silently lost. With @Checkpoint, FlowWarden persists two anchors of different natures to MongoDB and reloads them on startup to continue from the last durable position. The guarantee is at-least-once: no event is lost as long as a usable anchor exists, and the tail that was not yet durable at crash time is re-delivered — which is why handlers should stay idempotent.

lastProcessedToken — the work anchor. Advances on every terminally settled event: handler success, rejection by @Filter, no matching handler, or a terminal skip/DLQ decision. Persisted by a count-or-time policy (saveEveryN / saveIntervalSeconds), whichever threshold is reached first.
lastSeenToken — the position anchor. Holds exclusively server-certified positions written by the idle heartbeat (idleHeartbeatIntervalSeconds): when a stream settles nothing for a while, a bounded probe asks MongoDB to certify a fresh resume position, protecting an idle stream from oplog rollover for as long as probes keep succeeding within the oplog window.

On restart, a fixed 3-level cascade picks the resume position: lastProcessedToken first (strict at-least-once), the certified lastSeenToken as the safety net, then onHistoryLost.

Quick setup

Add @Checkpoint to any @ChangeStream class:

@ChangeStream(documentType = Order.class)
@Checkpoint
public class OrderStream {

    @OnChange
    void handle(ChangeStreamContext<Order> ctx) {
        System.out.println(ctx.summary());
    }
}

That’s it. With default settings, the anchor is saved after every settled event and reloaded automatically on the next startup.

How it works

Active stream — settlements anchor the position
Idle stream — the heartbeat certifies a fresh position

Saving strategies

The processed anchor — count or time, whichever comes first

lastProcessedToken is persisted by a single policy with two thresholds. All writers (the count threshold, the timer, and the manual ctx.saveCheckpointNow()) are serialized per stream, and each write also refreshes lastHeartbeatTimestamp — a token just delivered from the oplog re-confirms the recoverable position. saveEveryN — the count threshold. Persists the anchor at every N-th settled event. Settlements include handler successes, @Filter rejections, events with no matching handler, and terminal skip/DLQ decisions — an event still being retried never counts.

// Save after every settled event (default — safest, small overhead)
@Checkpoint

// Save every 10 settlements (lower write pressure, up to 9 events replayed on crash)
@Checkpoint(saveEveryN = 10)

saveIntervalSeconds — the time threshold. Bounds how long a dirty anchor waits before being persisted. On an active-but-slow stream (where the count threshold rarely triggers), this caps the anchor’s age in wall-clock time. A clean anchor is never rewritten — a quiet tick writes nothing.

// Dirty anchor persisted within 5 seconds (default)
@Checkpoint

// Within 30 seconds
@Checkpoint(saveIntervalSeconds = 30)

// Count threshold only
@Checkpoint(saveIntervalSeconds = 0)

With saveEveryN = 10, a crash between checkpoints means the last up-to-9 settled events are reprocessed on restart. Make sure your handlers are idempotent if you increase this value. The N-1 bound describes nominal operation: if a checkpoint write fails, the policy stays due and retries at the very next settlement (or the timer tick), but until a write is confirmed the replayable tail keeps growing — the bound is restored by the first confirmed write, not maintained through sustained store outages.

saveEveryN < 1 is rejected at startup with a BeanCreationException.

The certified position — `idleHeartbeatIntervalSeconds`

The sole writer of lastSeenToken. When the stream has settled nothing for this long, FlowWarden opens an ephemeral, bounded probe chained from the last settled position — same collection, same @Pipeline, same fullDocument options as the stream. If the server certifies the interval empty, the returned position is persisted: the resume point keeps tracking the oplog head with zero traffic — for as long as probes keep succeeding within the oplog window, so keep the interval well below that window.

// Certify after 5 minutes of idleness (default)
@Checkpoint

// Aggressive protection for a short oplog window
@Checkpoint(idleHeartbeatIntervalSeconds = 60)

// Opt out — an idle stream is then exposed to oplog rollover
@Checkpoint(idleHeartbeatIntervalSeconds = 0)

A delivered event token is never written as lastSeenToken. The position anchor accepts server-certified positions only. This is what makes resume safety structural: a restart that replays old events cannot regress the certified position, because replayed tokens are simply never candidates for it.

A certified position never crosses unsettled work. If events are sitting undelivered (a lagging or stuck stream), the probe abstains and writes nothing — repeated abstentions surface as a WARN so a stuck consumer is visible instead of being papered over.

Three regimes, three thresholds

Stream regime	Which mechanism keeps it recoverable
High-throughput	`saveEveryN` — the count threshold writes constantly
Active but slow	`saveIntervalSeconds` — the time threshold caps the anchor’s age
No events at all	`idleHeartbeatIntervalSeconds` — the heartbeat certifies fresh positions

// High-throughput stream: anchor every 50 settlements, at most 10s old when dirty
@Checkpoint(saveEveryN = 50, saveIntervalSeconds = 10)

Start position

startPosition controls where the stream starts consuming when it boots.

Value	Behaviour
`RESUME` (default)	Load the checkpoint from `_fw_checkpoints`. If none exists (first start), capture and persist a fresh server-certified position and start from it.
`LATEST`	Ignore any existing checkpoint and start from the latest event. Previous events are never replayed.

// Resume from last checkpoint (production default)
@Checkpoint(startPosition = StartPosition.RESUME)

// Always start fresh — useful for stateless consumers (e.g. cache warming)
@Checkpoint(startPosition = StartPosition.LATEST)

startPosition = LATEST ignores the persisted anchors on every restart. Use it only for streams where missing past events is acceptable (e.g. cache invalidation, live dashboards).

Resume cascade

When startPosition = RESUME, FlowWarden applies a fixed 3-level cascade on startup. Each level is tried in order — if the anchor is unusable (typically aged out of the oplog), it falls through to the next.

Level	Anchor	Behaviour	`StreamMetricsProvider` callback
1	`lastProcessedToken`	Strict at-least-once: everything after the last settled event is re-delivered.	—
2	`lastSeenToken`	The certified safety net. Typical for idle streams whose processed anchor aged out while the heartbeat kept certifying. WARN log — or a plain INFO when the processed anchor was simply never recorded (not a degradation).	`onResumeFallbackToSeen` (aged-out case only)
3	(none)	Apply the `onHistoryLost` strategy.	`onResumeHistoryLost`

Why the certified position matters

Imagine a stream that emits one event per week on a busy cluster. Its lastProcessedToken points at last week’s event — long gone from a 24-hour oplog window. Without the heartbeat, every restart would escalate to onHistoryLost. With it, lastSeenToken was re-certified minutes ago: the cascade falls back to a near-head position, replaying nothing and losing nothing (there was nothing to deliver in between — that’s exactly what the certification proved).

Cascade vs `startPosition = LATEST`

The cascade only runs with startPosition = RESUME (the default). With LATEST, both persisted anchors are ignored and the stream always starts from “now”.

Attribute reference

@Checkpoint(
    saveEveryN                   = 1,                    // anchor every N settlements
    saveIntervalSeconds          = 5,                    // max age of a dirty anchor (0 = count only)
    idleHeartbeatIntervalSeconds = 300,                  // idle certification cadence (0 = opt out)
    startPosition                = StartPosition.RESUME, // where to start on boot
    onHistoryLost                = OnHistoryLost.FAIL    // when both anchors expire from oplog
)

Attribute	Type	Default	Description
`saveEveryN`	`int`	`1`	Persist `lastProcessedToken` at every N-th terminally settled event. Must be > 0.
`saveIntervalSeconds`	`int`	`5`	Maximum time a dirty `lastProcessedToken` waits before being persisted. `0` disables the time threshold.
`idleHeartbeatIntervalSeconds`	`int`	`300`	Idle certification cadence — the sole writer of `lastSeenToken`. `0` opts out (idle streams are then exposed to oplog rollover).
`startPosition`	`StartPosition`	`RESUME`	`RESUME` to apply the cascade; `LATEST` to ignore both anchors.
`onHistoryLost`	`OnHistoryLost`	`FAIL`	Strategy when both persisted anchors have expired from the oplog. See @Checkpoint reference.

The MongoDB-backed storage is auto-configured. For a Redis-backed deployment, drop in the flowwarden-redis satellite — auto-configuration takes care of the rest. To plug in a different custom store (e.g. JDBC, in-memory for tests), provide your own CheckpointStore bean — see the CheckpointStore SPI reference.

Internal storage

Checkpoints are stored in the _fw_checkpoints collection of your MongoDB database. The collection is created automatically on first use. Each document is keyed by stream name and contains the two anchors and metadata:

{
  "_id": "order-stream",
  "instanceId": "app-1",
  "lastSeenToken": { "$binary": { ... } },
  "lastSeenTimestamp": "2026-02-25T10:00:01Z",
  "lastProcessedToken": { "$binary": { ... } },
  "lastProcessedTimestamp": "2026-02-25T10:00:00Z",
  "lastHeartbeatTimestamp": "2026-02-25T10:00:02Z",
  "metadata": {}
}

lastHeartbeatTimestamp records the last time a recoverable position was confirmed — an anchor write or a successful certification. Its age is the operational signal for resume-point health: it never moves on probe abstentions or failures.

Checkpoints written by versions before 1.0.0-rc.4 may hold an event token in lastSeenToken. They stay fully readable — the token remains a valid resume position — and the field adopts the certified-only semantics at the first successful certification.

You can inspect checkpoints directly, or delete a stream’s checkpoint to force a reset on the next restart:

// Inspect all checkpoints
db._fw_checkpoints.find()

// Force reset a specific stream
db._fw_checkpoints.deleteOne({ _id: "order-stream" })

Common patterns

Exactly-once semantics

FlowWarden provides at-least-once delivery by default. A crash between processing and checkpointing causes the last few events to be reprocessed on restart.To achieve exactly-once, use Spring’s @Transactional combined with ctx.saveCheckpointNow() to commit your business writes and the checkpoint atomically:

@OnInsert
@Transactional
void onInsert(Order order, ChangeStreamContext<Order> ctx) {
    // business write + checkpoint in one MongoDB transaction
    mongoTemplate.save(order, "processed_orders");
    ctx.saveCheckpointNow();  // included in the same transaction
}

ctx.saveCheckpointNow() is an advanced early acknowledgement: it persists the processed anchor while the handler is still running. Outside an atomic transaction, you take responsibility for having completed your effects before the call — if the handler fails afterwards, the event will not be re-delivered. Only a confirmed write counts: on a store failure the anchor stays dirty and the automatic policy retries it.See the Transactions guide for full setup instructions, constraints, and best practices.

Disabling checkpoint for dev/test

Omit @Checkpoint entirely — no anchors are persisted and the stream always starts from the latest event:

@ChangeStream(collection = "orders")
public class OrderStream {
    // no @Checkpoint — stateless, always starts from latest
    @OnChange
    void handle(ChangeStreamContext<?> ctx) { ... }
}

Resume cascade in action — a week of silence on a busy cluster

The scenario the idle heartbeat is designed for:

@Checkpoint(saveEveryN = 1) — the stream emits roughly one event per week, but the MongoDB cluster handles millions of unrelated writes per hour. The oplog window is about 24 hours.
The last event settled 6 days ago. lastProcessedToken points there — far outside the oplog window.
Every 5 minutes of idleness, the heartbeat probe certified a fresh position: lastSeenToken is minutes old.
The application restarts (rolling deploy, node failure — anything).
The cascade tries lastProcessedToken — ChangeStreamHistoryLost, aged out.
The cascade falls back to the certified lastSeenToken and resumes instantly near the oplog head, with a WARN log and the onResumeFallbackToSeen callback (surfaced by the Reporter or your own StreamMetricsProvider). Nothing was missed: every certification proved the interval behind it empty.

Without the heartbeat, step 6 would have escalated to onHistoryLost = FAIL and the stream would have refused to start, requiring operator intervention.

What happens when a handler gets stuck

The heartbeat protects idle streams — deliberately not stuck ones. If a handler hangs while events keep arriving, those events sit undelivered, and the probe abstains rather than certifying past them: a certified position must never cross unsettled work, or a crash would silently skip the deliveries a resume owes you.What you observe instead: lastProcessedToken and lastHeartbeatTimestamp stop moving, and after repeated abstentions FlowWarden logs a WARN (“matching events keep sitting undelivered”). That frozen heartbeat age is the operational signal — alert on it.

Multiple streams, independent checkpoints

Each stream has its own checkpoint entry in _fw_checkpoints, keyed by stream name. Two streams on the same collection checkpoint independently:

@ChangeStream(name = "billing-stream", collection = "orders")
@Checkpoint(saveEveryN = 1)
public class BillingStream { ... }

@ChangeStream(name = "notification-stream", collection = "orders")
@Checkpoint(saveEveryN = 50, saveIntervalSeconds = 10)
public class NotificationStream { ... }

@Checkpoint reference

Full annotation reference

How it Works

Where checkpointing fits in the event processing pipeline

Filtering Events

How filtered events interact with resume tokens

Getting Started

FAQ

Guides

Quick setup

How it works

Saving strategies

The processed anchor — count or time, whichever comes first

The certified position — `idleHeartbeatIntervalSeconds`

Three regimes, three thresholds

Start position

Resume cascade

Why the certified position matters

Cascade vs `startPosition = LATEST`

Attribute reference

Internal storage

Common patterns

See Also

@Checkpoint reference

How it Works

Filtering Events

​Quick setup

​How it works

​Saving strategies

​The processed anchor — count or time, whichever comes first

​The certified position — idleHeartbeatIntervalSeconds

​Three regimes, three thresholds

​Start position

​Resume cascade

​Why the certified position matters

​Cascade vs startPosition = LATEST

​Attribute reference

​Internal storage

​Common patterns

​See Also

@Checkpoint reference

How it Works

Filtering Events

Quick setup

How it works

Saving strategies

The processed anchor — count or time, whichever comes first

The certified position — `idleHeartbeatIntervalSeconds`

Three regimes, three thresholds

Start position

Resume cascade

Why the certified position matters

Cascade vs `startPosition = LATEST`

Attribute reference

Internal storage

Common patterns

See Also