Open source · Apache-2.0

Data pipelines,
written in YAML.

Fetch from APIs, files and databases, shape with SQL, and write to files, warehouses or back to ad platforms. StreamWright handles sign-in, paging, retries and state.

connectors
19
to shape data
SQL
ways to run
3
# streams/orders.yaml: fetch -> shape -> write
requests:                          # fetch
  - name: raw_orders
    operation:
      path: /v1/orders
      params:
        since: "{{ config.start_date }}"

transform:                         # shape
  mode: page
  steps:
    - name: orders
      select: |
        SELECT record->>'id'               AS order_id,
               record->>'customer'         AS customer,
               (record->>'amount')::DOUBLE AS amount
        FROM raw_orders
        WHERE record->>'status' = 'active'

export:                            # write
  orders:
    step: orders
    primary_key: [order_id]
    file_name: "{{ source }}/orders/{{ today }}.{{ format }}"

Reads from and writes to

  • Google Ads
  • Meta Ads
  • Microsoft Ads
  • LinkedIn Ads
  • Amazon Ads
  • Apple Ads
  • OpenAI Ads
  • PostgreSQL
  • MySQL
  • MongoDB
  • Amazon S3
  • Cloud Storage
  • Azure Blob
  • Delta Lake
  • Google Sheets
  • Kafka
  • Amazon SQS
  • DuckDB
  • Spark
  • Parquet

How it works

Fetch, Shape, Write: One Stream

Every pipeline does the same three things. Here is one stream doing them, with the data at each step.

streams/orders.yaml

requests:
  - name: raw_orders
    operation:
      path: /v1/orders
      params:
        since: "{{ config.start_date }}"
# base URL, sign-in, paging and retries: source.yaml

GET /v1/orders?since=2026-01-01 · 2 pages · 3 records

# page 1 · meta.next_cursor: "c2"{"id": "A-1001", "customer": "acme", "amount": "129.90", "status": "active"}{"id": "A-1002", "customer": "globex", "amount": "42.00", "status": "cancelled"}# page 2 · ?cursor=c2 · no next cursor: done{"id": "A-1003", "customer": "initech", "amount": "310.50", "status": "active"}

Connectors

Many connectors. One YAML grammar.

A connector is a small package that knows one system. Install the ones you need; your YAML looks the same for every one of them.

Write to · run on

  • Apache Spark
  • DuckDB / DuckLake
  • Apache Parquet
  • Singer Targets
  • dlt destinations
  • Dagster
  • Docker Container
  • Kubernetes (K8s)

Need another API? The REST connector reads most JSON APIs from YAML alone.All connectors

Reverse ETL

Send data back to ad platforms

The job that reads your data can also write it back. You write a query; StreamWright sends each row to the platform's API and records what happened to it.

  • Rows come from SQL

    Join your payload, files or warehouse tables; look up IDs on the platform first if you need to.

  • Sent in batches

    Rows go out in calls of the size you choose. Each call is logged with its row count and time.

  • Every row has an outcome

    Rejected rows keep the platform's reason, and you can query the outcomes with SQL.

  • Applied rows are not resent

    A re-run skips rows that were already applied, and calls that may have landed wait for your decision.

Run it from the CLI, on Spark, or behind streamwright serve so other systems can post rows over HTTP. Works with Google, Microsoft, Meta, LinkedIn, Amazon, Apple and OpenAI ads, and REST APIs.

jobs/conversions/upload.yamlDuckDBJob
outputs:
  - name: upload
    connection: ads                     # provider: google_ads
    step: conversions                   # one row = one conversion
    operation:
      service: ConversionUploadService
      method: upload_click_conversions
      arguments: {customer_id: "{{ config.customer_id }}"}
    apply:
      on_row_error: continue            # a bad row fails alone
    log:                                # say why rows failed
      sql: |
        SELECT gclid, outcome.error_code AS code
        FROM upload WHERE outcome.status = 'failed'
      message: "gclid {gclid} rejected: {code}"
run log
output upload: call 1/3: 10 row(s), 10 applied, 0 failed, 0.9 soutput upload: call 2/3: 10 row(s), 9 applied, 1 failed, 0.8 soutput upload: call 3/3: 8 row(s), 8 applied, 0 failed, 0.7 soutput upload: gclid Cj0KCQ... rejected: EXPIRED_EVENT

Why StreamWright

Less code to write, less to run

How StreamWright compares with connector frameworks and with scripts you write yourself.

 StreamWrightConnector frameworksCustom scripts
Adding an APIA YAML file, checked against a schemaA connector class per API, or a UI builderA script per API
Transforming dataDuckDB SQL while records moveUsually later, in the warehouseHand-written Pandas or dict code
Writing back to ad APIsBuilt in, with an outcome for every rowA separate reverse ETL toolHand-written batching and retries
What it takes to runOne Python processOften a JVM, a Docker stack or a serverLittle to run, a lot to maintain
Where data goesFiles, DuckDB, Delta, Singer, dlt, ad platformsThe tool's own destinationsWhatever the script was written for
AI assistantsAn MCP server: sw mcpRarely built inNone

Get started

Your first pipeline, in minutes.

Three steps to run the offline example on your machine. Then point a source at your own API.

  1. 1Install the CLI and a connector
    $ pip install streamwright$ sw connectors -i files
  2. 2Get the examples
    $ git clone https://github.com/karthick-jaganathan/streamwright.git$ cd streamwright
  3. 3Run the offline example
    $ sw run examples/sources/readers/files_demo \  --set data_root=examples/sources/readers/files_demo/data \  --output jsonl:out
  4. run finished in 0.5 s: 3 stream(s), 11 record(s) written