776 ms versus about 0.03 ms is the pagination contrast worth paying attention to. In a published PostgreSQL experiment on a five-million-row table, offset latency rose from 182 ms at page 100,000 to 776 ms at page 200,000, while keyset latency stayed near 0.03 ms in that setup, as described in GitLab's pagination guidance. Those are workload-specific measurements, not a service-level promise.

The bigger lesson isn't just to replace OFFSET. API pagination best practices define a consistency and cost contract: which records a client can expect, what each page costs, and how a worker resumes when its cursor expires. A fast query that misses records is still a broken export.

Table of Contents

Cursor or Offset for Your Collection

Pagination methods and when they fail

Method Fits when Typical failure Client rule
Offset with page and limit Small static lists and jump-to-page UIs Skips or duplicates after inserts and deletes; latency grows with depth Cap page size and maximum depth
Keyset or opaque cursor Large or frequently changing collections No random page access; wrong results if filters or sort change Send the opaque cursor with the same filters and sort
Link header next prev first last Public REST collections with server-built URLs Rebuilt query strings drop filters or sort Follow the server URL until rel next is absent
Opaque token plus hasMore Feeds walked from newest backward Replaying a token on a different query Pass the token back and stop when hasMore is false

Use indexed keyset pagination behind an opaque cursor for live profile, company, post, comment, and reaction collections. Keep offset for cases where arbitrary page access matters and the collection is small enough to bound its cost.

Separate database access from the public contract

These methods aren't mutually exclusive layers. Keyset describes how the database finds the next records; a cursor describes how the API carries continuation state. A Link header can carry a URL containing that cursor, while a response body can expose a token and hasMore.

Offset earns its place in an admin screen where users need to jump to a particular page. It's simple to inspect and works with familiar page controls. But the server still has to scan and discard preceding rows, and a changing collection can shift beneath the client between requests.

Don't make exact totals mandatory just because the interface has page controls. Counting the entire filtered collection can be separate work from retrieving a bounded page. Decide whether the user needs an exact count, an approximate count, or a reliable next action.

For a broader implementation checklist, PolyBackTest's guide to handling pagination in REST APIs is useful context when reviewing the client-facing contract alongside the database strategy.

Test the expensive path

A first-page benchmark doesn't exercise the problem that makes offset dangerous. Test shallow and deep traversal, then repeat while writers insert, delete, and update records.

  • Measure database work: Inspect rows examined, CPU, and the actual query plan, not just response duration.
  • Measure client impact: Track median and tail latency, timeouts, and retries across a complete walk.
  • Exercise query changes: Try reusing a continuation with altered filters, ordering, or resource scope.
  • Verify completion: Compare the records committed by the worker with the collection boundary it was promised.

A cursor doesn't automatically make the database efficient. If the implementation decodes a token into an offset, you've hidden the expensive scan without removing it.

Practical rule: If rows can appear, disappear, or reorder during the walk, don't ship offset as the live traversal contract.

How to Build a Cursor Contract

A cursor is server-owned pagination state, not a client-editable page number. RFC 8977 describes cursor-based continuation state and rejecting cursors that are inconsistent with the remaining query parameters. Apply that principle before worrying about token formatting.

The implementation needs a stable ordering, a matching access path, and an explicit client obligation.

A four-step guide on how to build a cursor contract for API pagination using database indexes.

Define ordering and validation

  1. Publish a deterministic total order. For a newest-first feed, use created_at DESC, id DESC. The stable, unique identifier breaks timestamp ties, so the server can identify an unambiguous continuation position.

  2. Build the index around the actual query. Put equality-filter columns before the ordered columns where that matches the access pattern. A collection scoped by company might need an index beginning with company_id, followed by created_at DESC, id DESC. Check the execution plan for each supported filter combination rather than assuming a single index covers every query.

  3. Sign the continuation state. Include the last returned ordering values, resource identity, a hash of canonicalized filters, sort, applied page size, and expiry. Reject tampered, expired, or incompatible state with a structured error. Base64 encoding alone doesn't establish integrity, and signing doesn't make sensitive contents confidential.

Canonicalization matters: equivalent query parameters should produce the same query identity regardless of their textual order. Apply defaults before calculating the hash, and validate authorization independently on every request.

Define continuation and completion

  1. Return a continuation only when more results exist. Use next_cursor or a server-built next URL, with an explicit terminal value. The JSON:API cursor-pagination profile defines opaque cursors and forward and backward traversal, while recommending first and last links when they're inexpensive to compute.

  2. Require clients to return the token unchanged. Our Posts API takes a professional profile URL or a company URL, plus an optional cursor for the next page. It walks backward through the timeline using the token returned with each page; reachable depth depends on what that profile or company has published and kept public.

  3. Keep the query context fixed. For our engagement results, clients pass back paginationToken and continue according to hasMore, rather than incrementing a start index. Filters, ordering, and the applied page size belong to the traversal identity. If clients need a different query, they should begin a new traversal.

Document whether the cursor references a moving feed or a short-lived snapshot. That distinction belongs in the public contract, not in an implementation comment, because it determines whether consumers can promise a reproducible export.

The minimum response should expose items, the page size applied, the next cursor or null, and hasMore. Keep these fields consistent: contradictory continuation signals force every SDK author to invent a different recovery rule.

Example Implementation

Start with the database, not the token encoder. For an illustrative PostgreSQL posts collection filtered by company, the index should match the equality predicate followed by the complete ordering.

CREATE INDEX posts_company_created_id_idx
ON posts (company_id, created_at DESC, id DESC);

For a continuation request, the keyset query is:

SELECT id, created_at, body
FROM posts
WHERE company_id = $3
  AND (created_at, id) < ($1, $2)
ORDER BY created_at DESC, id DESC
LIMIT $4;

Here, $1 and $2 are the validated cursor's last timestamp and identifier, $3 is the authorized company scope, and $4 is the applied page size plus an extra row. An initial request uses the same filtering and ordering without the continuation predicate.

Validate before querying

The request handler should perform these operations in order:

  • Authenticate and authorize: Establish the caller's permitted resource scope before accessing collection data.
  • Validate the cursor envelope: Decode defensively, verify the signature, and check expiry.
  • Compare query identity: Match the resource and canonical filter, sort, and page-size hash. In this example, return 400 Bad Request with a specific error code on a mismatch.
  • Execute the bounded query: Retrieve the requested page plus the extra row that establishes whether another page exists.

Don't use the extra row as the next cursor position. Return only the requested rows, then build the cursor from the last returned row. Otherwise, the client skips the record you fetched solely to detect continuation.

Return a response the worker can persist

For an illustrative applied size of 50, the response shape is:

{
  "items": [],
  "page_size": 50,
  "next_cursor": "opaque-token",
  "hasMore": true
}

The empty array stands in for the returned records in this shape example. On a terminal response, return next_cursor: null and hasMore: false.

Persist each page's records before advancing the durable checkpoint, ideally atomically with that checkpoint. If the process crashes between storage and checkpoint advancement, replay should be safe through idempotent writes.

This separation also applies to content integrations such as Podzone's programmatic access to podcast transcripts: keep resource processing separate from transport and continuation handling. Our API design example provides additional context for fitting request validation and response schemas into a product-facing API.

Page Size Under Cost and Rate Limits

A page is a billable unit, not the entire timeline. Our post search endpoint costs one credit per page and returns up to 50 posts, whether the page contains 50 results or none. Our paginated endpoints charge for each requested page, and failed API calls are never billed.

That makes page size a product decision as well as an engineering parameter. Tiny pages increase round trips and credits per completed record; oversized requests can increase memory pressure, timeout exposure, and retry work. The JSONic pagination guide explains why page sizing must account for payloads and request overhead, not just database execution.

Page size against cost and tail latency

The ranges below are starting points for testing, not latency guarantees. Provider-specific limits still take precedence over the requested size.

Page size Call and cost effect Failure effect Use it for
10 to 25 Many round trips; can consume a self-serve request budget quickly Smaller retry payload and lower per-request memory pressure Synchronous UI previews
50 One credit per post search page, full or empty A useful starting payload to test against an approximately one-second median call Post and engagement walks
100 Fewer calls where supported; GitHub's usual per_page ceiling Higher memory exposure, tail latency, and retry waste Bulk export only if measured tail latency remains acceptable
Above the provider max No extra rows; GitHub silently clamps oversized per_page requests The client may assume it received more than it did Never rely on it; inspect the returned count

Our self-serve capacity starts around 5 requests per second, with dedicated capacity up to 100 requests per second. Those limits constrain request frequency, not the number of workers you should start blindly. Use a shared limiter across workers so retries and new work don't compete without coordination.

Tune for completed records

Begin conservatively, often with 25 to 50 records where the endpoint supports that range. Measure p50, p95, error rate, and credits per completed record, then increase the size until throughput stops improving or tail latency worsens. Step back rather than optimizing only the median.

Keep the applied size fixed within a cursor chain when size forms part of query identity. Apply adaptive changes to the next traversal, or use an explicitly supported renegotiation mechanism.

  • Retry narrowly: Repeat only the failed page with its valid cursor.
  • Write idempotently: Avoid turning a transport retry into duplicate downstream processing.
  • Read continuation metadata: An empty page with a next cursor isn't the same as end-of-results.
  • Observe the whole job: Track completion time and committed records, not merely successful requests.

If you integrate monitoring through an API, as Fivenines supports, include traversal failures and stalled checkpoints in the operational view. Our API rate-limit guide is useful when coordinating request budgets across enrichment workers.

Optimize credits per finished record, not rows requested per call.

For a hypothetical export containing exactly 2,000 returned posts, full pages of 50 require 40 successful requests and 40 credits. Sequential calls taking about a second apiece fit comfortably below the self-serve request ceiling, although actual completion depends on latency and retries. If an endpoint supported pages of 10 under the same per-page billing, that workload would require 200 requests and 200 credits.

Errors Retries and Changing Data

Choose a consistency promise before writing retry code. A cursor preserves a position more reliably than an offset, but it doesn't freeze a live record. The LoadFocus pagination guide provides the broader pagination context; the operational requirement is to state the read boundary, expiry, and recovery behavior explicitly.

Name the endpoint contract

  • Moving feed: Each request reflects the collection as it exists at that moment. This fits monitoring posts and reactions, but it doesn't promise a duplicate-free, point-in-time export.
  • Short-lived snapshot: The token references a bounded, reproducible read view for bulk enrichment. A timestamp cutoff alone isn't a full snapshot if existing rows can still change or disappear.
  • Best-effort traversal: The API preserves ordered continuation but acknowledges that updates during the walk can move records across its boundary.

For a B2B data API that fetches the live page on every call, freshness and reproducibility compete. Our calls retrieve the record as it stands at request time rather than reading a stored copy. The practical distinction is explained further in our guide to freshness of data.

For bulk people or company exports, I would offer a snapshot capability separately from a moving feed. Don't label ordinary live cursors as snapshots unless the storage or upstream retrieval strategy can preserve that view.

Use this checklist to keep positional consistency separate from record freshness.

A list showing best practices for handling errors, retries, and changing data during API pagination processes.

Failure cases and the client action

The credit effects below describe our failed-call policy. Duplicate processing is a separate downstream cost, and a successful empty response remains subject to normal per-page billing.

Failure Client action Credit effect
Expired or malformed cursor Restart from the last durable checkpoint or request a new snapshot Failed call isn't billed
Cursor sent with a different filter or sort Treat as 400 and rebuild from the original parameters Failed call isn't billed
Rate limit Wait out the window, then repeat the same cursor Failed call isn't billed
Transient 5xx Retry that page only Not billed; the cost is time
Duplicate delivery Key the sink on a stable item id Duplicate insert or processing costs if unkeyed
Missing next or hasMore false Stop according to the documented completion signal No additional call required
Empty items with a next cursor Continue No error surcharge; successful pages are billed normally

A durable checkpoint is more than an expired token. Store enough original query context and a stable application boundary to initiate a supported recovery path. If the API can't resume that boundary, restart the traversal and deduplicate rather than inventing an offset.

Rate limits still apply during recovery. We enforce limits to prevent abuse, so use bounded retries, backoff, and any server-provided retry delay instead of launching replacement workers.

Our synchronous and asynchronous delivery options serve different workflows: a user request usually needs a bounded page, while a long traversal belongs in background processing. Asynchronous execution doesn't itself create snapshot consistency.

Document three explicit answers: what the cursor guarantees, when it expires, and how the client resumes. If the service can't recover a traversal without restarting, say so.

Habits and the Shipping Decision

Reliable clients follow server-issued continuation state instead of interpreting it. Make that behavior visible in SDK examples and integration tests, not just in prose documentation.

The next action should come from the response.

A hand clicks a next button in an API client showing the concept of link header pagination.

Keep traversal mechanical

GitHub's REST pagination documentation exposes next, prev, first, and last relations through the HTTP Link header, and documents a usual per_page maximum of 100, with oversized values automatically reduced. Follow rel="next" until it's absent; a short page isn't sufficient proof that traversal has finished.

Microsoft Graph's paging documentation instructs clients to GET the complete @odata.nextLink, including its original query parameters. Don't extract $skiptoken and reconstruct a request, because the complete URL preserves the server's continuation context.

The JSON:API specification places pagination relations in the collection's links object and permits unavailable relations to be omitted or null. Different wire formats can support the same disciplined client behavior.

If the next-page URL isn't something you would pass unchanged, the contract is already wrong.

Make silent failures observable

Log cursor expiry and cross-query rejection separately from upstream timeouts. A rise in validation failures suggests a client contract bug; timeouts need a different investigation.

Persist the completion signal with the token so a restarted worker can distinguish finished work from missing state. Include tied timestamps, replayed pages, and interrupted checkpoint writes in integration tests.

Use the video alongside a review of your client's continuation loop.

The shipping decision should be short enough to enforce in review:

  • Static lists: Allow offset only with hard page-size and depth caps.
  • Live collections: Require opaque continuation, deterministic ordering, and an explicit completion signal.
  • Consistency: Publish whether the traversal is a moving feed, snapshot, or best effort.
  • Economics: Tune supported page sizes using tail latency and credits per completed record.

Before the next client ships, add the composite index and a test that rejects a cursor whose filter hash doesn't match. That catches a preventable correctness failure before it becomes an incomplete customer export.

Fetchin is a real-time B2B data API that turns professional profile and company URLs into structured JSON and provides token-based traversal for posts and engagement results. Visit Fetchin to explore the endpoints and build a worker that preserves continuation state, handles failed pages safely, and tracks per-page credit use.