Embeddings

Embeddings transform raw input columns before they reach a model backbone. Numerical embeddings operate column-wise, while temporal embeddings reserve one column for time features and concatenate that branch with the remaining numerical features.

They do not consume the grouped padding mask: embeddings map each observation independently, and MaskedModel only reattaches w at the *Attn core. See Grouped padding and masks.

Available Embeddings

• IdentityEmbedding: leaves numerical features unchanged.

• LinearEmbeddings: projects each numerical feature to a learned vector.

• PeriodicEmbeddings: expands each feature with learned sinusoidal terms before projection.

• PiecewiseLinearEmbeddings: computes feature bins from the training data and embeds the resulting piecewise-linear encoding.

• BatchNormEmbeddings: batch-normalizes raw numerical features without expanding them.

• LayerNormEmbeddings: layer-normalizes raw numerical features without expanding them.

• TemporalEmbeddings: embeds one time column with Fourier features and an optional trend term.

Configuration

Use EmbeddingLayer to combine a numerical embedding with an optional temporal branch. A dictionary form is also available for model configuration dictionaries.

EmbeddingLayer(PeriodicEmbeddings(; d_embedding=24))

EmbeddingLayer(;
    num=PiecewiseLinearEmbeddings(; bins=32),
    temp=TemporalEmbeddings(; index=1),
)

EmbeddingLayer(Dict(
    :embedding_type => :periodic,
    :d_embedding => 24,
    :temporal => Dict(:index => 1),
))

Piecewise-linear and temporal embeddings need training data when the embedding chain is built. Use needs_x_train to check that requirement for a config.

API

NeuroTabModels.Models.Embeddings.BatchNormEmbeddings Type

BatchNormEmbeddings()

Batch-normalize the raw features without expanding their dimension (one output per feature).

source

NeuroTabModels.Models.Embeddings.EmbeddingLayer Type

EmbeddingLayer(; num=IdentityEmbedding(), temp=nothing)
EmbeddingLayer(num::AbstractNumericalEmbedding; temp=nothing)
EmbeddingLayer(d::AbstractDict)

Numerical embedding over the non-time columns, optionally combined with a temporal embedding on the column at temp.index (1-based position in the feature list). num always exists (defaults to IdentityEmbedding); nothing normalizes to it. When temp is set the branches are concatenated features-first, temporal-last.

Prefer the keyword constructor so the two branches are explicit:

EmbeddingLayer(;
    num=LinearEmbeddings(; d_embedding=16),
    temp=TemporalEmbeddings(; index=2, d_embedding=16),  # column 2 is time
)

index is the time column, not the embedding width and not nfeats. build_embedding_chain(config, nfeats; x_train) still needs nfeats = number of columns.

The Dict form is for the hyperparameter harness: :embedding_type (:linear, :periodic, :piecewise, :batchnorm, :layernorm, :identity) selects the numerical type, an optional :temporal Dict gives TemporalEmbeddings kwargs, and unknown or nothing keys are ignored (missing :embedding_type → identity).

EmbeddingLayer(PeriodicEmbeddings(; d_embedding=24))
EmbeddingLayer(; temp=TemporalEmbeddings(; index=1))            # temporal only
EmbeddingLayer(Dict(:embedding_type => :periodic, :temporal => Dict(:index => 1)))

source

NeuroTabModels.Models.Embeddings.IdentityEmbedding Type

IdentityEmbedding()

No-op numerical embedding that passes features through unchanged; the default num for an EmbeddingLayer.

source

NeuroTabModels.Models.Embeddings.LayerNormEmbeddings Type

LayerNormEmbeddings()

Layer-normalize the raw features without expanding their dimension (one output per feature).

source

NeuroTabModels.Models.Embeddings.LinearEmbeddings Type

LinearEmbeddings(; d_embedding=16, activation=:relu)

Per-feature linear embedding: each feature is projected to d_embedding by its own affine map, followed by activation.

Arguments

• d_embedding::Int: Output dimension per feature (default 16).

• activation: Activation applied after the projection (default :relu).

source

NeuroTabModels.Models.Embeddings.PeriodicEmbeddings Type

PeriodicEmbeddings(; d_embedding=16, frequencies=32, frequencies_init_scale=0.01f0,
                   activation=:relu, lite=false)

Per-feature periodic embedding: each feature is expanded with learned sine/cosine components, then projected to d_embedding and passed through activation.

Arguments

• d_embedding::Int: Output dimension per feature (default 16).

• frequencies::Int: Number of sinusoidal components per feature (default 32).

• frequencies_init_scale::Float32: Std. dev. initializing the frequencies (default 0.01f0).

• activation: Activation applied after the projection (default :relu).

• lite::Bool: Share the projection across features to cut parameters (default false).

source

NeuroTabModels.Models.Embeddings.PiecewiseLinearEmbeddings Type

PiecewiseLinearEmbeddings(; d_embedding=16, bins=32, activation=:identity, version=:B)

Per-feature piecewise-linear embedding against bin edges computed from the training data, then projected to d_embedding. The bin edges are derived at fit time, so this embedding requires x_train.

Arguments

• d_embedding::Int: Output dimension per feature (default 16).

• bins::Int: Number of bins, or per-feature bin counts (default 32).

• activation: Activation applied after the projection (default :identity).

• version: Encoding variant, :A or :B (default :B).

source

NeuroTabModels.Models.Embeddings.TemporalEmbeddings Type

TemporalEmbeddings(; index, order=[4, 1, 7, 0], periods=_DEFAULT_TEMPORAL_PERIODS,
                   trend=true, d_embedding=16)

Fourier embedding of a single time column: the column at index is expanded into multi-scale sine/cosine features at periods, projected to d_embedding, and optionally augmented with a linear trend. order and periods align per band: order[i] is the harmonic count for periods[i].

Arguments

• index::Int: Required. 1-based position of the time column in feature_names.

• order::Vector{Int}: Harmonics per period; nonnegative with at least one positive entry (default [4, 1, 7, 0]).

• periods::Vector{Float32}: Base periods in column units (default _DEFAULT_TEMPORAL_PERIODS).

• trend::Bool: Append a linear trend term to the embedding (default true).

• d_embedding::Int: Projection dimension of the periodic features (default 16).

source

NeuroTabModels.Models.Embeddings.build_embedding_chain Method

build_embedding_chain(config, nfeats; x_train=nothing)

Build the embedding chain for config. A numerical-only config embeds every column; with a temporal branch the input is routed through a Lux.Parallel, applying num to the non-time columns and temp to the time column, concatenated features-first then temporal.

Arguments

• config::AbstractEmbedding: The embedding config to build.

• nfeats::Int: Number of input features.

• x_train: Training matrix (n_samples, nfeats), required when needs_x_train(config) (default nothing).

Returns

A Lux layer emitting a flat (width, batch) output; recover the width with embedding_width.

source

NeuroTabModels.Models.Embeddings.embedding_width Method

embedding_width(layer, x, rng) -> Int

Output width of layer, measured by a single forward pass on the probe x. A forward pass is robust to the layer's internal structure, unlike analytic outputsize which cannot see through a vcat connection.

Arguments

• layer: The embedding layer to measure.

• x::AbstractMatrix: Probe input of shape (nfeats, batch) (init passes (nfeats, 2)).

• rng::AbstractRNG: RNG used for parameter setup.

Returns

The flattened output width as an Int.

source

NeuroTabModels.Models.Embeddings.needs_x_train Method

needs_x_train(config) -> Bool

Whether building config requires the training matrix.

Arguments

• config::AbstractEmbedding: The embedding config to inspect.

Returns

true when config needs x_train (e.g. piecewise-linear bin edges or temporal normalization statistics), false otherwise.

source