Skip to content

Data Output

You want to save results - scoring a full dataset and writing the output to parquet, CSV, a database table or a Databricks table for downstream analysis.

When to use

Use this for batch scoring - processing a full dataset and saving the results. For live API responses, use Quote Response instead.

This node takes a single input and passes it on unchanged: running or previewing the pipeline passes data through a Data Output without writing anything. It writes only when you click Write.

A Data Output cannot be reused with Create Instance: two writers to the same destination are refused. Add a separate Data Output with its own destination.

The CONFIG tab

Field What it does
PROVIDER Where the data goes: File (the default), Database, Lakehouse or Databricks. Changing it keeps only the settings the new provider uses.
FORMAT The format to write, from the formats the provider supports (see the table below). A format whose writer needs a package that is not installed is listed with needs one of: and the package, and a format Polars marks as unstable is listed with (unstable).
MODE sink (streaming) or write (in memory). Shown when the format supports more than one mode, or once a mode is set (a new node starts on sink). Each format offers only the modes Polars supports for it.
FILENAME OR PATH File provider: the file to write. Type a name or pick one from the file browser. Paths are resolved from your project folder, not from the pipeline file's folder. A bare file name goes in the project's outputs/ folder, and the format's extension is added when the name has none: scored_policies with Parquet selected writes outputs/scored_policies.parquet. A relative path such as exports/scored.parquet is taken from the project folder, and an absolute path must stay inside the project. Missing folders are created.
TABLE LOCATOR Lakehouse provider: the table to write.
CONNECTION ENVIRONMENT REFERENCE Database provider: the name of an environment variable that holds the database URI.
CREDENTIAL-FREE URI Database provider: a database URI without credentials, used instead of the connection reference.
TABLE Database provider: the table to write.
SQL WAREHOUSE Databricks provider: the SQL warehouse that runs the write. The list loads when the panel opens and marks serverless warehouses; if the workspace has only one, it is chosen for you. Enter HTTP path manually... takes an all-purpose cluster's HTTP path instead.
TABLE (Databricks) Databricks provider: choose a Catalog and Schema, then type a Table name or pick one of the schema's tables. A name that does not exist yet is created by the first write. Names may use letters, digits, underscores and hyphens.
WRITE MODE Databricks provider: Replace table (the default) replaces the table's rows and columns with the output; Append rows adds the output's rows, matched by column name, and creates the table if it is missing.
ARGUMENTS Extra keyword arguments for the Polars writer. Add argument adds a row with a name (suggested from the arguments the writer accepts) and a JSON value (for example "," or true). Only arguments the writer accepts are allowed; an unknown name or a value that is not valid JSON is flagged.

The formats each provider offers:

PROVIDER FORMAT
File Parquet, CSV, NDJSON, JSON, Arrow IPC / Feather, Arrow IPC stream, Avro, Excel
Lakehouse Delta Lake, Iceberg
Database Database (URI)
Databricks none: the output is always loaded as a table

Under the fields, the panel says how the format is written (a streaming sink or an eager writer) and how it is published.

Writing

When the settings are complete, the panel shows the resolved Destination: before you write, and warns if the destination's extension does not match the selected format.

  • Write runs the pipeline up to the node and writes the full dataset to the destination. It is disabled until the settings are complete, and reads Writing... while it runs. When it finishes, the panel reports the result with the number of rows and the path.
  • A Databricks write streams the output to a Parquet file, uploads it to a haute_staging volume in the target table's schema (the first write creates the volume), loads it into the table on the chosen SQL warehouse, then deletes the uploaded file. It uses the same DATABRICKS_HOST and DATABRICKS_TOKEN (or service-principal) credentials as a Databricks Data Input. Your token needs permission to create a volume and tables in the schema, or a haute_staging volume you have created for it. WRITE MODE decides what happens to an existing table, so there is no separate confirmation.
  • If the destination file already exists, the panel asks you to confirm with Replace existing file before overwriting it. For a lakehouse or database destination, what happens when the table already exists is the Polars writer's own behaviour, which you can set with ARGUMENTS (for example if_table_exists for a database table).

The COLUMNS tab

By default every input column is written; to write only some of them, untick the others on the COLUMNS tab, with no upstream node needed (see Working with any node).

Example

Writing a scored dataset to parquet:

  1. Connect the scored data to a Data Output node.
  2. Leave PROVIDER on File, FORMAT on Parquet and MODE on sink.
  3. Type scored_policies in FILENAME OR PATH. The panel shows the destination, outputs/scored_policies.parquet.
  4. Click Write.

Not part of a deployed API

A deployed pricing API never writes: its Data Output nodes pass data through without invoking the writer.

Multiple outputs

You can have multiple Data Output nodes in a pipeline - for example, to write both a parquet file and a CSV, or to save results at different stages.

In the pipeline file

The node's settings are stored in a JSON sidecar, config/data_output/<node name>.json, which the pipeline's .py file names in the node's decorator: @pipeline.data_output(config="config/data_output/<node name>.json"). Every configuration chooses exactly one destination with output_type.

Setting in the editor Stored as
PROVIDER output_type: "file", "lakehouse", "database" or "databricks"
FORMAT format: "parquet", "csv", "ndjson", "json", "ipc", "ipc_stream", "avro" or "excel" for a file; "delta" or "iceberg" for a lakehouse; "database" for a database
MODE mode: "sink" or "write"; absent uses the format's usual mode
FILENAME OR PATH, TABLE LOCATOR path
CONNECTION ENVIRONMENT REFERENCE connection
CREDENTIAL-FREE URI uri
TABLE table (catalog.schema.table for Databricks)
SQL WAREHOUSE http_path
WRITE MODE write_mode: "overwrite" or "append"; absent replaces the table
ARGUMENTS arguments, a map of argument name to value
COLUMNS tab selected_columns

The column metadata any node may carry, column_renames, has no editor control on this node.

A Databricks output has no format, mode or arguments:

{
  "output_type": "databricks",
  "http_path": "/sql/1.0/warehouses/abc123",
  "table": "main.pricing.scored_policies",
  "write_mode": "overwrite",
  "arguments": {}
}

The example above is stored as:

{
  "output_type": "file",
  "format": "parquet",
  "mode": "sink",
  "path": "scored_policies"
}

Every key the config may hold:

Key Type Description
arguments map of text to any JSON value Arguments passed to the Polars write or sink call, by parameter name.
mode "sink" or "write" Whether the frame is streamed to the destination (sink) or collected and written (write). With output_type "file", "database", "lakehouse".
output_type "file" The destination: a file.
format text The file format, such as parquet or csv. With output_type "file", "database", "lakehouse".
path text The file written, relative to the project root or absolute. With output_type "file", "lakehouse".
connection text The named connection in connections.toml; exactly one of connection and uri. With output_type "database".
uri text A credential-free database URI; exactly one of connection and uri. With output_type "database".
table text The table written. With output_type "database", "databricks".
http_path text The SQL warehouse's HTTP path. With output_type "databricks".
write_mode "overwrite" or "append" Whether a load replaces the table's rows or adds to them. With output_type "databricks".
input_mapping map of text to text The input names the node's code uses, each mapped to the connected input it reads.
contract "opaque" or object The node's column contract as its decorator declares it: "opaque", or the columns it reads and produces.
selected_columns list of text The columns the node passes downstream, chosen on its Columns tab; absent passes all.
column_renames map of text to text New names for columns passed downstream, from each column's name to its new one.

See also: