Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.emergedata.ai/llms.txt

Use this file to discover all available pages before exploring further.

Query API responses can contain thousands of records. Use pagination to retrieve data in manageable chunks, and delta queries to fetch only new data since your last sync.

​
Cursor-based pagination

Every response includes a next_cursor for fetching the next page: 
{
  "data": [...],
  "count": 100,
  "has_more": true,
  "next_cursor": "eyJsYXN0X2lkIjogMTIzNH0="
}
Pass the cursor to get the next page: 
async function fetchAllSearchHistory(uid: string) {
  const allData: SearchEntry[] = [];
  let cursor: string | undefined;

  do {
    const params = new URLSearchParams({ uid });
    if (cursor) {
      params.set('cursor', cursor);
    }

    const response = await fetch(
      `https://query.emergedata.ai/v1/sync/get_search?${params}`,
      { headers: { 'Authorization': `Bearer ${API_TOKEN}` } }
    );

    const result = await response.json();
    allData.push(...result.data);

    cursor = result.has_more ? result.next_cursor : undefined;
  } while (cursor);

  return allData;
}

​
Page size

Control the number of records per page with the limit parameter: 
curl 'https://query.emergedata.ai/v1/sync/get_search?uid=psub_c3d4e5f6789012345678901234abcdef&limit=500' \
  -H 'Authorization: Bearer your_api_token'
ParameterDefaultMaximum
limit1001000

​
Delta queries

Fetch only data that’s been ingested since your last sync using ingested_begin and ingested_end: 
curl 'https://query.emergedata.ai/v1/sync/get_search?uid=psub_c3d4e5f6789012345678901234abcdef&ingested_begin=2024-01-15T00:00:00Z' \
  -H 'Authorization: Bearer your_api_token'

​
Time-based filtering

ParameterDescriptionFormat
ingested_beginRecords ingested at or after this timeISO 8601
ingested_endRecords ingested before this timeISO 8601
ingested_begin and ingested_end filter by when data was added to Emerge, not when the user performed the action. This is useful for incremental syncs.

​
Response fields for delta sync

The response includes additional fields to help with delta syncing: 
{
  "data": [...],
  "count": 50,
  "has_more": false,
  "next_cursor": null,
  "applied_ingested_end": "2024-01-15T12:00:00Z"
}
FieldDescription
applied_ingested_endThe actual end time used for filtering. Use this as your next ingested_begin value to ensure no gaps.

​
Delta sync pattern

interface SyncState {
  lastIngestedEnd: string;
}

async function deltaSync(uid: string, state: SyncState): Promise<SyncState> {
  const newData: SearchEntry[] = [];
  let cursor: string | undefined;

  do {
    const params = new URLSearchParams({
      uid,
      ingested_begin: state.lastIngestedEnd
    });
    if (cursor) {
      params.set('cursor', cursor);
    }

    const response = await fetch(
      `https://query.emergedata.ai/v1/sync/get_search?${params}`,
      { headers: { 'Authorization': `Bearer ${API_TOKEN}` } }
    );

    const result = await response.json();
    newData.push(...result.data);

    // Use applied_ingested_end for next sync to avoid gaps
    const appliedEnd = result.applied_ingested_end;

    cursor = result.has_more ? result.next_cursor : undefined;

    // After last page, update state
    if (!cursor) {
      return {
        lastIngestedEnd: appliedEnd
      };
    }
  } while (cursor);

  // Process new data
  await processNewData(newData);

  return state;
}

// Usage
let syncState: SyncState = {
  lastIngestedEnd: '2024-01-01T00:00:00Z'
};

// Run periodically
syncState = await deltaSync('psub_c3d4e5f6789012345678901234abcdef', syncState);

​
Best practices

Always use the applied_ingested_end from the response as your next ingested_begin value. This ensures no gaps between syncs.
An empty data array with has_more: false means no new data since your last sync.
500 records per page balances throughput and memory usage for most applications.
If you hit rate limits or errors, back off exponentially before retrying.
Each record includes an event_id for deduplication. Use this to handle any potential duplicates across pagination boundaries.

​
Async job pagination

For async queries, results are returned as a single Parquet file. The file contains all records for all users in the request. If you need to paginate through very large async results, use the begin and end date parameters to narrow the time range: 
// First batch: January 1-15
const firstWindow = new URLSearchParams({
  user_ids: 'psub_a1b2c3d4e5f6789012345678901234ab',
  begin: '2024-01-01T00:00:00Z',
  end: '2024-01-15T23:59:59Z'
});

const job1 = await fetch(
  `https://query.emergedata.ai/v1/search?${firstWindow.toString()}`,
  { headers: { 'Authorization': `Bearer ${API_TOKEN}` } }
).then(r => r.json());

// Second batch: January 16-31
const secondWindow = new URLSearchParams({
  user_ids: 'psub_a1b2c3d4e5f6789012345678901234ab',
  begin: '2024-01-16T00:00:00Z',
  end: '2024-01-31T23:59:59Z'
});

const job2 = await fetch(
  `https://query.emergedata.ai/v1/search?${secondWindow.toString()}`,
  { headers: { 'Authorization': `Bearer ${API_TOKEN}` } }
).then(r => r.json());