vgrid fetch

Fetch transaction data from financial service APIs and convert it to VisiGrid’s canonical truth format.

vgrid fetch <source> [options]

VisiGrid connects to 14 financial data sources. Each adapter produces the same canonical output format below.

Output format

All fetchers produce the same canonical 9-column CSV:

Column	Type	Required	Description
effective_date	string	yes	ISO 8601 date (YYYY-MM-DD)
posted_date	string	no	Settlement/posting date
amount_minor	int	yes	Minor units (cents). Never float.
currency	string	yes	ISO 4217 uppercase (USD, EUR)
type	string	yes	Transaction type (charge, refund, …)
source	string	yes	Adapter name
source_id	string	yes	Unique ID from upstream system
group_id	string	no	Grouping key (payout ID, invoice, …)
description	string	no	Human-readable memo

Output is deterministically sorted (default: group_id, effective_date, source_id) so identical data always produces byte-identical CSV.

---

Bring-your-own-API adapter (http)

Connect any REST API that returns JSON. A mapping file tells VisiGrid how to extract the 9 canonical columns from each JSON item.

vgrid fetch http \
  --url https://api.example.com/v1/transactions \
  --auth bearer-env:EXAMPLE_API_KEY \
  --map mapping.json \
  --from 2026-01-01 --to 2026-02-01 \
  --out transactions.csv

Mapping file

{
  "root": "$.data.transactions",
  "params": {
    "from": { "query": "start_date", "format": "iso" },
    "to":   { "query": "end_date",   "format": "iso" }
  },
  "columns": {
    "effective_date": "$.created_at",
    "posted_date":    { "path": "$.settled_at", "optional": true },
    "amount_minor":   { "path": "$.amount_usd", "transform": "dollars_to_cents" },
    "currency":       { "path": "$.currency",   "transform": "upper" },
    "type":           { "path": "$.category",   "map": { "payment": "charge", "refund": "refund", "*": "adjustment" } },
    "source":         { "const": "example_api" },
    "source_id":      "$.id",
    "group_id":       { "path": "$.invoice_id", "optional": true },
    "description":    { "path": "$.memo",       "optional": true }
  },
  "sort_by": ["effective_date", "source_id"]
}

Key concepts:

• root — dot-path to the response array ($.data, $.payments[0].items, etc.)
• params — maps --from/--to to query parameter names. Supports iso, unix_s, unix_ms date formats.
• columns — maps each canonical column. Use a string for simple path extraction, or an object for transforms/constants/optional handling.

Auth methods

Auth credentials are resolved from environment variables only — never inline secrets.

Method	Flag	Example
None	--auth none	Public APIs
Bearer token	--auth bearer-env:VAR	bearer-env:FORTE_TOKEN
Custom header	--auth header-env:NAME:VAR	header-env:X-API-Key:MY_KEY
Basic auth	--auth basic-env:USER:PASS	basic-env:API_USER:API_PASS

Transforms

Transform	Input	Output	Use case
cents	5000 (int)	5000	Already in minor units
dollars_to_cents	"50.00" (string)	5000	Dollar amounts → cents
upper	"usd"	"USD"	Currency normalization
lower	"CHARGE"	"charge"	Type normalization

Value mapping

Map upstream enum values to your canonical types:

"map": { "payment": "charge", "refund": "refund", "*": "adjustment" }

The * wildcard catches any value not explicitly mapped.

Pagination

Most APIs paginate responses. Add a pagination section to your mapping file to automatically fetch all pages.

Two strategies are supported:

Cursor-based (Stripe, Brex style):

{
  "root": "$.data",
  "pagination": {
    "strategy": "cursor",
    "param": "starting_after",
    "page_size_param": "limit",
    "page_size": 100,
    "next_cursor_path": "$.data[-1].id",
    "has_more_path": "$.has_more"
  },
  "columns": { ... }
}

Offset-based (Mercury style):

{
  "root": "$.transactions",
  "pagination": {
    "strategy": "offset",
    "param": "offset",
    "page_size_param": "limit",
    "page_size": 500,
    "has_more_path": "$.meta.has_more"
  },
  "columns": { ... }
}

Pagination field	Description	Required
strategy	"cursor" or "offset"	yes
param	Query param name for cursor/offset value	yes
page_size_param	Query param name for page size	yes
page_size	Items per page	no (default: 100)
next_cursor_path	JSONPath to next cursor value (cursor only)	cursor: yes
has_more_path	JSONPath to boolean “has more” flag	no (inferred from page size)

If has_more_path is omitted, pagination stops when a page returns fewer items than page_size.

Safety guards:

• --max-pages caps the number of pages fetched (default: 100)
• --max-items caps the total number of items across all pages (default: 10,000)
• Stuck cursor detection: if the cursor value doesn’t change between pages, the adapter errors instead of looping forever
• Empty page with has_more=true is treated as an error, not silent truncation

Options

Flag	Description	Default
--url	API endpoint (HTTPS only)	required
--auth	Auth method	none
--map	Path to mapping JSON	required
--from / --to	Date range (YYYY-MM-DD)	required
--out	Output CSV path	stdout
--sample	Print raw JSON response and exit	off
--save-raw	Save raw JSON to file for audit	off
--timeout	Request timeout in seconds	15
--max-items	Safety cap on total items	10,000
--max-pages	Safety cap on pages fetched	100
--quiet	Suppress progress messages	off
--fingerprint	Write signed request fingerprint JSON	off

Request fingerprinting

Add --fingerprint <path> to produce a signed JSON sidecar that records what was requested, when, and from where. This is useful for audit trails — you can later prove that a specific CSV was produced by a specific API call.

vgrid fetch http \
  --url https://api.vendor.com/v1/payments \
  --auth bearer-env:VENDOR_TOKEN \
  --from 2026-01-01 --to 2026-01-31 \
  --map mapping.json --out payments.csv \
  --fingerprint payments.fingerprint.json

The fingerprint is a SignedEnvelope (same format as vgrid sign) containing:

Field	Description
request.url	Full URL with query params (no secrets — tokens go in headers)
request.auth_method	Auth flag as passed (e.g. bearer-env:VENDOR_TOKEN), not the secret
request.from / to	Date range
request.pages_fetched	Number of pages fetched (useful for detecting truncation)
mapping.path	Mapping file path
mapping.blake3	BLAKE3 hash of the mapping file
output.row_count	Total rows written
output.csv_blake3	BLAKE3 hash of the output CSV (only when --out is used)

The signing key is auto-generated at ~/.config/vgrid/proof_key.json on first use (same key as vgrid sign).

Explore first, then map

Use --sample to inspect the raw API response before writing a mapping:

vgrid fetch http \
  --url https://api.example.com/v1/transactions \
  --auth bearer-env:EXAMPLE_API_KEY \
  --from 2026-01-01 --to 2026-01-02 \
  --sample

Tip

Fetch adapters are for pulling live data from APIs. For offline verification workflows, skip fetch and provide transaction CSVs directly to vgrid export truth.