Data Input

You have tabular data you want to bring into your pipeline - historical policies, external enrichment data, lookup tables. The Data Input node reads it from a file, a lakehouse table, a database, a Databricks table, or records typed straight into the node.

When to use

• Loading historical data for analysis or model training.
• Bringing in reference data to join with your quotes (e.g. postcode lookups, external scores).
• Use Quote Input instead when building the live API entry point.

A Data Input has no inputs and one output: the data it reads. Its panel has three tabs: CONFIG, where you choose the source, TRANSFORM and COLUMNS.

The CONFIG tab

PROVIDER, at the top, chooses where the data comes from: File, Database, Lakehouse, Databricks or Inline. A new node starts on File with the Parquet format. Choosing another provider starts that provider's settings afresh; the node's steps or code on the TRANSFORM tab and its COLUMNS selection are kept.

The fields below PROVIDER depend on the provider. A field marked * is required.

File

Field	What it does
FORMAT	The file format: CSV, JSON, NDJSON (.jsonl/.ndjson), Parquet, Arrow IPC / Feather, Arrow IPC stream, Avro, Excel, OpenDocument spreadsheet or Text lines (one text line per row). A format Polars marks as unstable says (unstable), and a format whose reader needs a package that is not installed says "needs one of:" and the packages. Changing the format keeps the path and clears ARGUMENTS.
PATH *	The file, chosen in the file browser, which lists the folders of your project and the files with the format's extensions. Once a file is chosen, change opens the browser again. The path is stored relative to your project folder.

For a CSV file, the tab also shows the columns and types Haute detects in the file. Use detected schema copies them into ARGUMENTS as the reader's schema, which fixes each column's type; until you do, the tab shows "A schema mapping is required for this bounded input." Without a declared schema, Haute infers each column's type from the whole file when it copies the file into its snapshot (see Snapshots). Detection reads the file with the node's ARGUMENTS (all but schema), so a ; separator or a missing header row shapes the detected columns, and changing an argument detects them again. If the file cannot be read, the tab says "Could not detect schema:" with the reason, and Retry schema tries again.

Database

Field	What it does
FORMAT	Database (URI). This release reads SQLite databases.
CONNECTION ENVIRONMENT REFERENCE	The name of an environment variable that holds the database URI, so credentials stay out of the pipeline.
CREDENTIAL-FREE URI	A database URI with no user name, password or secret parameter, used instead of a connection reference. Filling in one of the two clears the other.
QUERY *	The read-only SQL query that selects the data.

Lakehouse

Field	What it does
FORMAT	Delta Lake or Iceberg (unstable).
TABLE LOCATOR *	The table's location in your project: a Delta table's folder, or an Iceberg table's metadata file (metadata/<version>.metadata.json). For Delta, open the table's folder in the file browser and click Use this folder.

Databricks

Field	What it does
SQL WAREHOUSE	The SQL warehouse's HTTP path, such as /sql/1.0/warehouses/abc123; your Databricks administrator can provide it. Type it, or click Browse to list the workspace's warehouses (each with its state and size) and pick one.
TABLE	The table to read, chosen in three lists: Select catalog..., then Select schema..., then Select table.... The full name, catalog.schema.table, shows underneath.
SELECT CLAUSE	Optional. A SELECT clause without FROM, such as SELECT policy_id, premium; Haute appends FROM and the chosen table ("Optional. A SELECT list without FROM, such as SELECT policy_id, premium. Haute adds FROM and the chosen table."). It must start with SELECT and must not contain FROM, semicolons, SQL comments or write keywords. Leave it empty to read every column.

A Databricks input has no format.

Inline

Field	What it does
FORMAT	Inline records.
RECORDS *	The rows typed straight into the node, as a JSON list of row objects, such as [{"area": "London", "factor": 1.25}]. Invalid text is not saved: the box says "Invalid JSON - changes have not been saved." or "Records must be a JSON array."

ARGUMENTS

Every provider's fields end with ARGUMENTS: extra keyword arguments for the Polars reader, such as separator for a CSV. Add argument adds a row with a name (the box suggests the arguments the reader accepts) and a value written as JSON, for example ";" for a text value or true; the bin removes a row. A database or Databricks input accepts only batch_size.

• A name the reader does not accept is kept, with "name is not a supported … argument. It is kept so the backend can reject it explicitly."
• A value that is not valid JSON keeps the previous value: "Invalid JSON. The previous value is kept until this parses."

A CONFIGURATION ERRORS box at the top of the fields lists anything in the node's settings that the tab cannot use, such as "Select a valid format for this provider." or "Databricks batch_size must be a positive integer."

Column names

A file's columns take their names from its header row, or from ARGUMENTS, never from Polars:

• A file without a header row. A CSV, Excel or ODS file with has_header set to false must name every column, in order, with new_columns, for example ["policy_id", "region", "premium"]. A CSV can instead declare a full schema, whose names are given to the columns in order. Without either, the node reports "A CSV file without a header must name every column, in order, with new_columns, or a full schema". Polars would otherwise call the columns column_0, column_1 and so on, and a pipeline that relied on those names would read the wrong column after an upgrade. A new_columns list must name every column the file has: one naming fewer or more is refused with both counts.
• A CSV schema. The names in a schema are matched to the CSV's header by name, so you can list the columns in any order. A schema must name exactly the header's columns: to rename columns, use new_columns; to set the type of some columns, use schema_overrides, which may name any of them.
• Several files at once. A path matching several CSV files takes its column types from the first ten files.

The TRANSFORM tab

See Transform.

The COLUMNS tab

The COLUMNS tab chooses which of the loaded columns the node passes on (see Working with any node).

In the data preview

A parquet file is read directly. Every other source - another file format, a database, a lakehouse or Databricks table, inline records - is first copied into a cached parquet snapshot, and the pipeline reads the snapshot. Haute refreshes a snapshot when it detects that the source has changed.

A map column, such as a Databricks MAP column or a map in a parquet or Arrow file, arrives as a list of key-value structs, [{key, value}, ...], holding every entry, including a key the map repeats. A parquet file with a map column cannot be read directly, because that read would keep only the last value of a repeated key, whatever schema the node declares: the node reports "has map column(s)" and names them, and the file must be copied into a snapshot instead (mode "read", set in the pipeline file, as In the pipeline file below describes). A folder or glob of parquet files with a map column is refused too: give each file its own Data Input. A lakehouse table's maps keep only the last value of a repeated key; the tools that write lakehouse tables keep a map's keys unique.

For a snapshot source, Import sits beside Refresh in the node's data preview. It re-reads the source and caches it as a new snapshot, and its tooltip shows when the source was last imported. Haute cannot detect changes to a database, lakehouse or Databricks table, so for those sources Import is the only way to pick up new data. To delete cached snapshots, use Cached data in Pipeline settings.

While an import runs, the button counts the rows written so far where the source is written in steps: a database or Databricks table, a parquet or IPC file. A CSV is written in one pass and shows Importing… until it finishes.

Imports run in a separate process with a memory limit, so a source too large for memory stops that process, not Haute. On macOS the limit is watched rather than enforced: see macOS: memory limits are watched, not enforced.

Example

A parquet file in your project:

1. Leave PROVIDER on File and FORMAT on Parquet.
2. Under PATH, open the data folder and click policies.parquet.

The preview shows the file's first rows.

A CSV file that uses ; as its separator:

1. Choose CSV as the FORMAT, and data/claims.csv as the PATH.
2. Under ARGUMENTS, click Add argument, type separator as the name and ";" as the value.

The detected schema splits the columns on ;. Click Use detected schema to fix their types, or leave the types to inference when Haute copies the file into its snapshot.

A Databricks table:

1. Choose Databricks as the PROVIDER.
2. Under SQL WAREHOUSE, click Browse and pick the warehouse, or type its HTTP path, such as /sql/1.0/warehouses/abc123.
3. Under TABLE, choose the catalog pricing, the schema motor and the table policies.
4. Preview the node. When the table changes later, click Import in the data preview to read it again.

Credentials never go in the pipeline

A CREDENTIAL-FREE URI must not contain a user name, password or secret query parameter. Put a URI with credentials in an environment variable and give that variable's name as the CONNECTION ENVIRONMENT REFERENCE.

In the pipeline file

The node's settings are stored in a JSON sidecar, config/data_input/<node name>.json, which the node's decorator in the pipeline's .py file names: @pipeline.data_input(config="config/data_input/<node name>.json"). Steps on the TRANSFORM tab are stored in the sidecar as steps, and the code they generate is the body of the node's function, which takes the loaded data as df; code the other steps cannot hold goes in a Free code step.

Setting in the editor	Stored as
PROVIDER	input_type: "file", "database", "lakehouse", "databricks" or "inline"
FORMAT	format (none for Databricks): see the table below
PATH / TABLE LOCATOR	path, relative to your project folder
CONNECTION ENVIRONMENT REFERENCE	connection
CREDENTIAL-FREE URI	uri
QUERY (Database)	query
SQL WAREHOUSE	http_path
TABLE (Databricks)	table (catalog.schema.table)
SELECT CLAUSE	query (absent when empty)
RECORDS	records: a list of row objects
ARGUMENTS	arguments: an object of argument names and values
TRANSFORM tab	the function body, which Haute reads the steps back from
COLUMNS tab	selected_columns

The formats each provider accepts:

input_type	format
"file"	"parquet", "csv", "ndjson" (.jsonl/.ndjson), "json", "ipc" (Arrow/Feather), "ipc_stream", "avro", "excel", "ods", "lines" (one text line per row)
"lakehouse"	"delta", "iceberg"
"database"	"database" (SQLite in this release)
"inline"	"records"

With no editor control: mode, for file, lakehouse and inline sources, is "scan" (lazy) or "read" (eager). Choosing a format sets it to the format's usual mode, and a format offers only the modes Polars supports for it. A parquet file with mode "read" is copied into a snapshot like the other sources; a parquet file with a map column needs it. Such a file's snapshot applies only the columns and n_rows arguments, and refuses any other argument that changes which rows or columns it holds. Database and Databricks inputs have no mode.

Every key the config may hold:

Key	Type	Description
code	text	The Polars code the node's function runs; when steps are present, their rendering.
steps	list of map of text to any JSON value	The low-code steps, in order, rebuilt from the node's code each time it loads; the config file does not hold them.
arguments	map of text to any JSON value	Arguments passed to the Polars read or scan call, by parameter name.
mode	"" or "read" or "scan"	Whether the file is read eagerly or scanned lazily; empty or absent takes the format's default. With input_type "file", "lakehouse", "inline".
input_type	"file"	The provider: a file.
format	text	The file format, such as parquet, csv or ndjson. With input_type "file", "database", "lakehouse", "inline".
path	text	The file, relative to the project root or absolute. With input_type "file", "lakehouse".
connection	text	The named connection in connections.toml; exactly one of connection and uri. With input_type "database".
uri	text	A credential-free database URI; exactly one of connection and uri. With input_type "database".
query	text	The SQL query whose rows the node reads. With input_type "database", "databricks".
http_path	text	The SQL warehouse's HTTP path. With input_type "databricks".
table	text	The table, as catalog.schema.table. With input_type "databricks".
records	list of map of text to any JSON value	The rows, each an object from column name to value. With input_type "inline".
input_mapping	map of text to text	The input names the node's code uses, each mapped to the connected input it reads.
contract	"opaque" or object	The node's column contract as its decorator declares it: "opaque", or the columns it reads and produces.
selected_columns	list of text	The columns the node passes downstream, chosen on its Columns tab; absent passes all.
column_renames	map of text to text	New names for columns passed downstream, from each column's name to its new one.

See also: Transform for steps and code. Sharing pipelines across operating systems, WSL, or network mounts? See Filesystem Portability.