Skip to content

Transform

This is the general-purpose node for shaping your data. Joining two datasets, calculating a new column, filtering rows - if there isn't a specialised node for it, you do it here, as a list of steps or as Polars code. This is the node you'll use most often.

When to use

  • Joining two datasets together (e.g. quotes with external enrichment data).
  • Creating derived columns (age from date of birth, vehicle age from year of manufacture).
  • Filtering or reshaping data in ways the specialised nodes don't cover.

A Transform node takes any number of inputs and has one output. Each connection brings in the upstream node's data under the upstream node's name (a Quote Input table arrives under its label, and a submodel output under its port's name). The panel has two tabs: TRANSFORM, where you build the node, and COLUMNS.

Building the node from steps

The TRANSFORM tab is the step builder. You build the node as a list of steps named in plain English, and Haute turns them into Polars code. The same builder is on the TRANSFORM tab of the Data Input, Load File, Rating Step, Expander and Model Scoring nodes, and in the TRANSFORM pane of an Explore node; Steps on other nodes says what differs there.

From top to bottom, the tab shows the node's inputs, the STEPS (a Start from card, then one card per step), the Add step button and the Generated code panel.

Inputs

INPUT (INPUTS when there are several) shows one chip per connection, named as the steps and code see it. The × on a chip removes that connection from the canvas. A chip shown in amber with a warning sign comes from a Quote Input table that is no longer emitted; hover it for the reason.

Start from

The first card, Start from, chooses the input the node starts from. With one input connected, Haute chooses it for you. With several, pick one from the list, which reads Choose an input until you do; an input that has since been disconnected shows as "name (not connected)". Once the start is chosen the node can be previewed: with no further steps it passes that input through unchanged.

With no input connected, the tab shows "Connect an input to start building steps." instead.

Add step

Add step, under the last card, opens the step chooser in place:

  • The search box ("Search steps and functions") finds steps and calculations by name, by what they do, or by their Polars method, best match first: typing join finds Join another input and Join text, clip finds Clip, sum finds Window: sum before Group and aggregate, and to_date finds Text to date. A match that adds a different step says which, such as Add column or Change types. Enter adds the first match.
  • The arrow keys move between the entries, and the foot of the chooser describes the entry you point at, with its Polars method.
  • Esc, or the × beside the search box, closes the chooser without adding a step.

The entries are grouped as follows:

Group Entries
Add column Formula, If-then, Window, Coalesce, Number function, Text function, Date function
Rows Filter rows, Sort rows, Remove duplicates, Limit rows
Columns Keep columns, Drop columns, Rename columns, Change types, Fill missing values
Combine Join another input, Append inputs, Group and aggregate, Pivot to columns, Unpivot to rows
Other Define variable, Free code

Each entry in the Add column group adds an Add column step of that kind, and the step keeps it: a Formula, If-then, Window or Coalesce step shows just its own form, and a Number function, Text function or Date function step has a Function list of the functions of its kind only (Number function starts as Round, Text function as Trim whitespace, Date function as Shift date). Search finds the other calculations too, such as Clip, Sum across or Join text. Choosing an entry adds its card at the end of the list, open, with the cursor in its first field.

Step cards

Each step is a numbered card. Its header shows the number, the step's icon and its name; an Add column card is named after what it computes, such as Formula, If-then or Clip. A closed card also shows a one-line summary of the step and the change it makes to the columns: +loss_ratio for a column it adds, − before columns it removes, or → 3 columns after a grouping or pivot. Click the header to open the card and edit the step, and click it again (or press Esc) to close it.

The buttons on the right of each card:

  • The up and down arrows move the step one place. Alt+Up and Alt+Down on the header do the same, and you can also drag a card by its header onto another card. Steps run in list order, top to bottom; the Start from card always stays first.
  • The bin deletes the step.

Notes under a closed card tell you what needs attention:

  • "Not in the data at this step:" followed by column names, when the step names a column the data does not have at that point, for example one an earlier step dropped or renamed.
  • What an unfinished step still needs, such as "Needs a formula." or "Needs a name for the new column." These are neutral reminders while you build. They turn into errors only after the pipeline has run and failed on this node, and then the failing step's card shows the error.

A card that Haute cannot read (for example after the pipeline file was edited by hand) shows as Invalid step, with "This step cannot be edited here. Delete it to repair the step list."

While you work in the step builder, Delete and Backspace outside a text box, and the canvas's copy, paste, select-all and group shortcuts, stay in the step builder instead of acting on the canvas.

What each step does

Each card reads as a sentence or a short form. The column boxes suggest the columns the data has at that step.

Step What it does What you fill in
Filter rows Keeps rows that match conditions (filter) "Keep rows where", then one condition per row: a column, a test (equals, is greater than, is one of, is missing, contains text, matches pattern (regex) and more) and a value. Add condition adds a row; with several, choose whether all or any of them must be true. An is one of list of numbers is written as the kind of number its column holds (see Column types).
Add column Computes a new or replaced column (with_columns) Column name, then the form of the step's kind: a formula box, an If-then's branches, a Window's aggregate, or, for a function step, its Function (Number: Round, Clip, Divide and more; Text: Trim whitespace, Lower case, Replace all matches and more; Dates: Shift date, Duration in days, Year and more) with its value and arguments. Changing the Function keeps what you built: the column being computed carries over, and going back to an earlier choice brings it back as you left it. A value set to Expression has its own Computed as list, in four groups (General, Number, Text, Dates), for any calculation, such as a Clip of a Sum across. Types are changed, and missing values filled, by their own steps (Change types, Fill missing values); in a formula cast, try_cast, to_date and fill_null still work. A formula is typed as text, e.g. (premium + commission) * tax / 12, and the box completes column names, earlier variables, global constants and function names, also by what a function does: typing trim offers strip. A function's extra arguments are plain values, except that clip's bounds and fill_null's value can also be a column, a variable or a formula, for example clip(irpm_total, -irpm_cap, irpm_cap). divide(premium, limit) divides like / but gives a missing value where the divisor is zero or missing, where / gives infinity (or NaN for 0 / 0). sum_horizontal(load_a, load_b, load_c) adds up two or more values, skipping missing ones, so a row with every value missing gives 0 (with + one missing value makes the total missing); Computed as Sum across builds the same from a list of parts. A Coalesce step lists two or more parts under Parts, in order (Add part adds another), each a column, a value, a constant or an Expression, and gives each row the first part that isn't missing, trying them in order: premium, then quoted_premium, then 0 gives 0 only where both columns are missing. A row with every part missing stays missing, unless the last part is a value. In a formula the same is coalesce(premium, quoted_premium, 0). An If-then gives each row the Then value of the first branch whose conditions hold, or the Otherwise value: Add branch adds another, led by "Otherwise, when", for bands or limits that pick one value per row. A Window gives each row an aggregate of its group (Over), of a column or of an Expression such as limit * basic_load; for the aggregates that sum up a group, such as sum, mean or row count, More options adds Only rows where…, so each group aggregates only its matching rows (a group with none sums to 0).
Keep columns Keeps only the listed columns (select) The columns to keep; More options adds And every other column of type.
Drop columns Removes the listed columns (drop) The columns to drop; More options adds And every other column of type.
Rename columns Gives columns new names (rename) "Rename column to new name"; Add rename adds another.
Change types Casts columns to another data type (cast) "Change column to type", for example Int64 (whole number), Float64 (decimal number), String (text) or Date (date). A text column becoming a date also takes the format its dates are written in (see Column types). Values that don't convert become missing gives a missing value where a value can't change type, or text doesn't match the format, instead of stopping the run.
Sort rows Orders rows by one or more columns (sort) "Sort by column", "then by" further columns (Add sort column), each ascending or descending, and Missing values last. Rows that tie on every sort column keep the order they arrived in.
Remove duplicates Keeps one row per key (unique) Rows are duplicates when they match on (empty means every column), and which row to keep: the first row, the last row, any one row or no row (drop every duplicate).
Group and aggregate Summarises rows into one row per group (group_by().agg()) Group by (empty summarises the whole table) and Aggregations: a name, a function (sum, mean, count of values, row count and more) and a column each. More options adds a row filter per aggregation (Only rows where…).
Join another input Combines columns from another input (join) The join kind (for example left: every row here, with matches added) and the input to join, then Match on pairs ("a column here = a column there"). More options holds Check key cardinality, Output row order and Suffix for clashing column names. A new join keeps its inputs' order (left, then right; right, then left for a right join; left input order for a semi or anti join), and changing the join kind moves the order with it unless you chose another.
Append inputs Stacks rows from other inputs (pl.concat) Append rows from: tick the inputs. Columns either must match exactly or may differ (fill missing with null).
Fill missing values Replaces nulls with a value or strategy (fill_null) Fill missing values in (empty means every column), then With a value or a strategy such as the previous row's value (forward) or the column's mean (mean).
Limit rows Keeps the first N rows (head) "Keep the first 100 rows".
Define variable Names a value for later steps "name = value"; later steps and Free code can use the name.
Pivot to columns Makes one column per value of a category (pivot) One row per (index columns), "Spread the values of", "Each cell is the function of column", and the New columns, one per value.
Unpivot to rows Stacks several columns into name/value rows (unpivot) Stack these columns, Keep as index, and the two new column names ("Into … and …"). The stacked rows come out in no set order; add a Sort rows step after it when a later step or your output depends on the order.
Free code Runs Python statements against the current table A code box: see Free code.

A Join another input card on a node with a single input says "Connect the table to join on the canvas.": connect the other table to the node first.

Column types

Where the editor knows a column's type, which it learns once the input has been previewed, it offers only what Polars can do with that type, so a step that would fail never gets built.

  • Text that holds dates. Change types turns a text column into a Date or Datetime by reading it with a format: a format box appears beside the type, starting at %Y-%m-%d (2024-01-31), or %Y-%m-%d %H:%M:%S for a date and time. Change it to match your data: %d/%m/%Y reads 31/01/2024. %Y is the four-digit year, %m the month, %d the day, %H, %M and %S the hour, minute and second. In a formula, to_date(start, '%d/%m/%Y') and to_datetime(...) read text the same way and stop the run on a value that doesn't match; try_to_date(...) and try_to_datetime(...) give a missing value instead, as Values that don't convert become missing does for a whole Change types step. cast and try_cast can't turn text into a date.
  • Which types a column can change to. A category column changes only to String (text) or Categorical (category): to make a number or a date of it, change it to String (text) first. A text column can't become Boolean (true/false): compare it with a value in a condition instead. A number or a date can't become Categorical (category) directly.
  • Numbers in "is one of". A list of numbers is written the way its column holds numbers: whole numbers for a whole-number column (a fraction can never match, so the list refuses one), numbers with a decimal point for a decimal column, and exact decimals for a Decimal column, which also refuses a value with more decimal places than the column has. Where the column's type isn't known yet, a Number type select chooses.
  • Functions and aggregations. log and exp are for numbers, and std and var aren't offered for a duration: turn it into days first with total_days.

Where the editor doesn't know a column's type it offers everything. A step that then fails on its column's type stops the run on that step's card, and the message starts with what to change.

Free code

When no step does what you need, choose Add step → Other → Free code to write Python between the other steps. The code starts with the current table in df and Polars available as pl; assign your result back to df:

df = df.with_columns(
    (pl.col("premium") * 1.05).alias("loaded_premium")
)
  • You can add more steps afterwards, and move or delete the free-code step like any other.
  • Do not use return: the next step needs to run. The card says "Assign the result to df instead of using return." if you do.
  • Variables from earlier Define variable steps are available in your code, and global constants as global_constants.<name>.
  • If your code creates columns, type their names into the fields of later steps: the builder cannot see inside your code.
  • Use df for the current table so upstream renames keep working. Input names written in free code need updating by hand when those inputs change.
  • Code that is not valid Python is reported on the card, for example "Invalid Python on line 2: …".

The code box suggests column names as you type, and you can drag its lower edge to make it taller.

Generated code

Under Add step, the Generated code panel shows the Polars code your steps produce, line-numbered and read-only. Click its title to fold it away or open it again.

  • Point at a card and its lines are tinted in the code; point at a line to tint its step, and click it to open that step's card.
  • While a step is unfinished, the panel keeps the last code that worked, dimmed and marked out of date, with a note such as "Step 2 isn't finished: it needs a formula."
  • After a run that failed on this node, the panel names the failing step in red with Go to error, which opens that step's card, and highlights the line that failed.

Steps on other nodes

On the TRANSFORM tab of a Data Input, Load File, Rating Step, Expander or Model Scoring node (and in Explore's TRANSFORM pane), the steps start from a table the node already holds in df - the loaded data, the rated data, and so on:

  • There is no Start from card, and the cards are numbered from 1.
  • Join another input and Append inputs are offered only on a Load File node, whose other inputs the steps can name. On the other nodes the steps see only df.
  • The loaded object of a Load File node, obj, is available only in a Free code step.
  • The column boxes suggest the node's input columns and the columns the node itself produced in its last preview, such as a Model Scoring node's prediction column or a Data Input's columns.

What df holds on each node, and the hint its tab shows:

Node df holds Hint
Data Input The loaded data "df = the opened input snapshot"
Load File The first input; the loaded file is obj, which only a Free code step, or the node's code when it is not steps, can use, for example to score with obj.predict(...) "obj = loaded file, assign to df"
Rating Step The rated data "use df for rated data"
Expander The expanded data "use df for expanded data"
Model Scoring The scored data, with the prediction in the node's output column "Post-processing Code (optional)"
Explore (TRANSFORM pane) The node's input "assign to df"

Steps on these nodes do what a Transform node straight after them would do. Use them to shape the data where the node produces it, such as filtering a Data Input's rows as they load or deriving expected_claims from a prediction; use a Transform node when the work combines several inputs.

Writing the node as code

A node whose code Haute cannot read as steps, such as a Transform with keyword-only parameters or code that is not a list of Python statements, shows a code box, POLARS CODE, on the TRANSFORM tab instead of the steps. Anything else you want to write as code goes in a Free code step. The hint beside the label reads "use input names, assign to df" ("assign to df" when nothing is connected):

  • Each input is available by the name of the node it came from. For example, if you connect a node called policies, you reference it as policies in your code.
  • df is not an input - it's the variable your code must assign its result to (reading df before assigning it is an error). Haute passes whatever df holds to the next node, so do not end your code with return df: Haute adds that line itself, and shows it dimmed under the code box.
  • The code box suggests column names inside quotes: the inputs' columns, then the columns the node produced in its last preview, including the ones your code creates. On the TRANSFORM tab of another node these include what the node adds, such as a Model Scoring node's prediction column. A column your code renames or drops is suggested under the name the preview showed.
  • After a failed run, the code box highlights the line that failed.
df = policies.join(claims, on="policy_id", how="left", maintain_order="left_right")
df = df.with_columns(
    (pl.col("claim_amount") / pl.col("premium")).alias("loss_ratio")
)

Reading Polars code

If you're coming from a drag-and-drop pricing tool, the code on this page may look unfamiliar. Here's a quick cheat-sheet.

Polars syntax What it means
df A dataframe - a table of data. df is the node's output: assign your result to it. Each input is available by the name of the upstream node it came from.
pl.col("column_name") Refers to a column by name.
.alias("new_name") Gives the result a new column name.
.with_columns(...) Adds or replaces columns in the table.
.filter(...) Keeps only rows that match a condition.
pl.when(...).then(...).otherwise(...) An IF statement.
pl.lit("value") A fixed/literal value.

A custom join in code

In the code above, the canvas has inputs named policies and claims. Haute reads the code to find the columns it uses: the join key and the columns used to calculate loss_ratio.

Keep the source names and join keys in sync with the canvas. Haute can project the columns the code uses and the join keys, but custom code with a column contract it cannot prove is handled conservatively as an execution boundary. That is safe, but it can scan more columns or require an admitted materialisation. See Execution Strategy for the boundary and diagnostic details.

Common patterns

The examples below assume one upstream node called policies; start from the input's name and assign the result to df.

Calculate a derived column:

df = policies.with_columns(
    (pl.col("premium") * pl.col("expense_loading")).alias("loaded_premium")
)

Filter rows:

df = policies.filter(pl.col("cover_type") == "comprehensive")

Conditional logic:

df = policies.with_columns(
    pl.when(pl.col("driver_age") < 25)
      .then(pl.lit("young"))
      .otherwise(pl.lit("standard"))
      .alias("driver_category")
)

The COLUMNS tab

The COLUMNS tab chooses which of the node's output columns it passes on, as on any node (see Working with any node).

Reusing the node with instances

If you have the same logic applied to different inputs, you don't need to duplicate the node. An instance reuses the original node's steps or code with different inputs, and a change to the original updates every instance. See Reusing a node.

Example

A node that adds each policy's loss ratio, from two inputs called policies and claims:

  1. Connect policies and claims to a new Transform node. Its TRANSFORM tab lists both under INPUTS.
  2. In Start from, choose policies.
  3. Click Add step, type join and press Enter to add Join another input. Choose left: every row here, with matches added, join claims, and under Match on pick policy_id = policy_id.
  4. Click Add step and choose Formula. Type loss_ratio as the Column name and type claim_amount / premium as the formula.

The closed cards read as a summary of the node, and the Generated code panel shows:

df = policies
df = df.join(
    claims,
    left_on=["policy_id"],
    right_on=["policy_id"],
    how="left",
    suffix="_right",
    maintain_order="left_right",
)
df = df.with_columns(
    (pl.col("claim_amount") / pl.col("premium")).alias("loss_ratio")
)

A node that is not finished stops the run

You can save a node whose steps are not finished; saving warns which step is incomplete. Running the pipeline then stops at that node with "This node's steps are incomplete. Complete or remove them before running." A node with no starting input and no code stops with "This transform has no code yet. Add code that defines what it returns."

Hand edits to the pipeline file

The steps are read from the code in the .py file each time Haute loads the pipeline. A statement you edit by hand, or one Haute doesn't recognise, becomes a Free code step, and the steps around it stay as they are.

In the pipeline file

A Transform node is a function in the pipeline's .py file, decorated with @pipeline.polars. Its parameters are the node's inputs, and its body is the node's code followed by the return df Haute adds. The steps are stored only as the code they generate, in the body, and Haute reads them back from it. While a step is unfinished, the body raises an error instead and the steps are kept in the pipeline's .haute.json until they are complete.

A node written as code can also take keyword-only parameters after its inputs, each with a literal default, such as def priced(policies, *, loading=1.05). The code reads loading as it reads an input, and it takes precedence over a value of the same name defined above the pipeline. A default must be None, True or False, a number or a string; a list or dictionary default is refused, because a function shares one copy of it between runs. Haute will not open a node whose keyword-only parameter has no default, or has one it cannot write back (a name or a calculation, for example), and names the line; a node built from steps, and every other node type, takes none.

Setting in the editor Stored as
The steps on the TRANSFORM tab the function body, read back as steps in the node's configuration: a list with one object per card, each with an id and a kind ("source" for Start from, then "filter", "with_column", "join", "free_code" and so on) and the card's fields
The code of a node whose code is not steps the function body (code in the node's configuration)
Keyword-only parameters, written in the file the signature, after * (keyword_parameters in the node's configuration: each one's name and default)
COLUMNS tab selected_columns, a decorator argument: the columns passed on (absent passes every column)

Create Instance wraps the node in a one-node submodel and adds another occurrence of its definition; see Reusing a node.

Every key the config may hold:

Key Type Description
code text The Polars code the node's function runs; when steps are present, their rendering.
steps list of map of text to any JSON value The low-code steps, in order, rebuilt from the node's code each time it loads; the config file does not hold them.
keyword_parameters list of object The keyword-only parameters of a transform written as code, each with its literal default.
keyword_parameters[].name text The parameter's name in the function signature.
keyword_parameters[].default text or number or true or false or null Its literal default, which a run binds.
input_mapping map of text to text The input names the node's code uses, each mapped to the connected input it reads.
contract "opaque" or object The node's column contract as its decorator declares it: "opaque", or the columns it reads and produces.
selected_columns list of text The columns the node passes downstream, chosen on its Columns tab; absent passes all.
column_renames map of text to text New names for columns passed downstream, from each column's name to its new one.

See also: