Backfill historical data

Backfill with the API

Observations endpoints accept a date range, a page limit and a cursor. FX rates use /v1/fx/rates/{base}/observations; CPI uses /v1/economics/{country}/cpi/observations; commodity bars use /v1/commodities/spot/{commodity}/{currency}/bars. Date parameters are start and end, or start_date and end_date, depending on the endpoint. Each response has next_cursor, which is null on the last page.

1. Set up your API key Get a free API key and keep it in the environment variable NARWHAL_API_KEY. The script reads that variable and sends the key as Authorization: Bearer YOUR_KEY to https://api.narwhalapi.com.
2. Set the series and date range The example requests USD/IDR from 1 November 2025 to 23 October 2026 with limit set to 100. Keep the same filters on every page when you send the cursor returned by the preceding response.
3. Save each page to a file Use the script to append each observation to usd_idr.jsonl as a JSON line. It sends another request while next_cursor has a value and stops when that value is empty.
   Python, a year of daily USD/IDR rates to a file

   import json, os
   import requests
   
   H = {"Authorization": f"Bearer {os.environ['NARWHAL_API_KEY']}"}
   params = {"currencies": "IDR", "start_date": "2025-11-01", "end_date": "2026-10-23", "limit": 100}
   rows, pages, cursor = 0, 0, None
   with open("usd_idr.jsonl", "a") as out:
       while True:
           if cursor:
               params["cursor"] = cursor                 # the cursor carries the filters; resend them unchanged
           r = requests.get("https://api.narwhalapi.com/v1/fx/rates/USD/observations", params=params, headers=H, timeout=15)
           r.raise_for_status()
           body = r.json()
           for day in body["observations"]:
               out.write(json.dumps(day) + "\n")
               rows += 1
           pages += 1
           cursor = body.get("next_cursor")
           if not cursor:
               break
   print(pages, "pages,", rows, "days")

4. Read the returned dates For the 1 November 2025 to 23 October 2026 range, the script prints 3 pages, 249 days. Read the first stored date as well as the last, since the response can begin later than the start you requested.

The script keeps separate counts for pages and days. It writes each observation before it requests the following page, then prints the totals after the cursor loop ends. The file opens in append mode, and each saved line contains one observation. The request reads the IDR series with USD as its base currency.

Each page counts as one successful request against your quota; Free has 5,000 successful requests per UTC month and a limit of 30 requests per minute. A backfill across many series needs room between pages to stay under the minute limit, even when each individual series needs only a few pages.

Responses carry X-RateLimit-Remaining. If a page request returns 429 rate_limit_exceeded, use its Retry-After value before requesting another page. A 429 quota_exhausted response means waiting for the next UTC month. Errors use application/problem+json and include status, code, detail and request_id, so read the error code when deciding whether the backfill can continue.

Use a sheet or a no-code workflow

A Sheets import is fine for a few hundred rows. For a one-off backfill, a script is simpler than a workflow that loops over pages. Use the file route when you want the observations written as they arrive; the Python example appends rows throughout the backfill.

n8n, Make or Zapier can loop over pages with a cursor. The workflow needs to send the same date and series filters with each cursor, then stop when next_cursor is null. An ongoing feed can begin with a backfill and continue with a daily data pull.

Choose a backfill approach

What if the backfill stops halfway?

Restart from the last date you stored. The script appends JSON lines as it receives observations, so rows already written remain in the file if the job stops. Use that stored date as the place to resume the range.

A saved cursor can expire. If a request returns 409 cursor_expired, restart without the cursor and resume from the last stored date. A cursor carries the filters for its range; keep those filters unchanged while paging through that range.

How many rows can I request per page?

Set limit up to the endpoint's maximum: 366 for FX rate observations, 1,000 for economics observations, 500 for commodity bars and 100 for the economic calendar. The default is 100. The limit controls page size; you still follow next_cursor until the response has no further page.

Why does my backfill start later than requested?

A start date before your plan window moves up to the window's first day without an error. Free includes the last 12 months, Basic includes 5 years and Pro includes full history. The first returned date tells you where the response actually begins, so an earlier requested start does not guarantee earlier rows.

403 plan_upgrade_required applies when the whole requested range ends before the plan window, or when you read a single date or period outside it. A range that crosses the window's first day can return data from that day onward. See plans and limits if you need history beyond your window.

Worked example: USD/IDR from 1 November 2025 to 23 October 2026

Request USD/IDR with a start of 1 November 2025, an end of 23 October 2026 and a page limit of 100. Follow each returned cursor with the same filters. Across that range, the response contains 249 rows over 3 pages, so the backfill uses 3 requests.

The first row is Monday 3 November 2025 because 1 and 2 November 2025 are a weekend. On 3 November 2025, the USD/IDR rate is 16,710.3; on 23 October 2026, it is 18,107.8.