Export data in bulk with Python

Some questions end in a table too large to page through: full history for a brand, a whole metro's series, the training set for a model. The export API is built for exactly that: it runs the same query server-side as an asynchronous, high-throughput job and returns downloadable files, free of the 90-second per-request timeout that interactive queries run under.

The request shape (dataReferenceName, fields, filters, filterLogic) is identical to :runQuery, so anything built there ports directly. Exports return up to 100 million records by default (limit caps it lower; result sets beyond the cap fail with RESOURCE_EXHAUSTED rather than truncating silently).

1. Create the export

One POST starts the job. This example pulls a brand's June-2025 month in one city. Swap in your own filters, or none at all for a full dataset pull:

import os, requests

BASE = "https://api.pine59.com"
auth = {"Authorization": f"Bearer {os.environ['PINE59_API_KEY']}"}

job = requests.post(f"{BASE}/v3/exports", headers=auth, json={
    "dataReferenceName": "visitation_usa_places.foot_traffic_month",
    "fields": [{"name": "location_id"}, {"name": "street_address"}, {"name": "visits_sum"}],
    "filters": [
        {"fieldName": "brands", "operator": "==", "value": "H-E-B"},
        {"fieldName": "city", "operator": "==", "value": "San Antonio"},
        {"fieldName": "observation_start_date", "operator": "==", "value": "2025-06-01"},
    ],
    "format": "CSV",
}).json()

print(job["id"], job["status"])   # export_..., RUNNING

format takes CSV, JSON, AVRO, or PARQUET. Parquet is the right choice for anything headed into pandas or a warehouse at volume. fileConfig.filenamePrefix names the output files when you organize many exports.

2. Poll until it completes

Exports move through RUNNING (and sometimes COPYING) to COMPLETED or FAILED:

import time

while job["status"] in ("RUNNING", "COPYING"):
    time.sleep(5)
    job = requests.get(f"{BASE}/v3/exports/{job['id']}", headers=auth).json()

print(job["status"])

The small run above completed in under twenty seconds; full-dataset exports take longer, so poll every few seconds rather than continuously.

3. Download the files

A completed export lists its files under fileExportInfo.files, each with a url, name, sizeBytes, and format metadata. The URLs are time-limited (fileExportInfo.expireTime says until when), so download promptly and re-create the export if the links lapse:

import pandas as pd
from io import BytesIO

frames = []
for f in job["fileExportInfo"]["files"]:
    content = requests.get(f["url"]).content
    frames.append(pd.read_csv(BytesIO(content)))

df = pd.concat(frames, ignore_index=True)
print(len(df), "rows")

For June 2025, the example above returns 45 rows, one per H-E-B store in San Antonio, ready for the same notebook work the examples build on. Large exports split across multiple files; the loop handles either case.

4. Manage export jobs

The remaining endpoints round out the lifecycle: GET /v3/exports lists your exports, and DELETE /v3/exports/{id} removes one. For exports that should land directly in your own BigQuery project instead of files, the create request takes a bigqueryConfig. The API reference documents the destination options.

When to export, when to query

:runQuery is built for interactive work: filtered questions, aggregations, anything a person or agent iterates on. As data volumes grow, the export API becomes the more effective path: the same query, run server-side at full throughput, delivered as files sized for notebooks and warehouses. The two compose naturally: explore and refine interactively, then export the refined query at scale.