Embeddings design

Evovest

• Setup

• Numerical embeddings

  • LinearEmbeddings

  • PeriodicEmbeddings

  • PiecewiseLinearEmbeddings

  • BatchNormEmbeddings

  • IdentityEmbedding

• Temporal embeddings

  • TemporalEmbeddings

  • EmbeddingLayer

Embeddings map each raw column into a representation the backbone consumes. Numerical embeddings act column-wise. A temporal embedding reserves one time column; EmbeddingLayer concatenates that branch last. Piecewise-linear and temporal embeddings need training data at build time (needs_x_train).

Setup

Every block below uses the same scalar over  , the same mark  , and the same init seed. Figures are split by stage:

1. raw —  

2. basis — intermediate , when the map has one

3. embedding — coordinates

Expanding embeddings use d_embedding=4 and activation=:identity so the geometry is visible; Linear and Periodic default to :relu in the constructor.

e_titles (generic function with 1 method)

Numerical embeddings

Each numerical config is applied independently to every non-time column. Expanding types emit   coordinates; BatchNorm and Identity keep width 1.

LinearEmbeddings

The simplest expansion: each embedding coordinate is an independent affine map of the same scalar.

One number becomes a vector whose coordinates are lines with different slopes and intercepts . There is no intermediate basis. With   (as plotted), the grid is those lines; the default :relu would zero the negative half of each coordinate.

conf_linear = LinearEmbeddings(; d_embedding, activation=:identity)
layer_linear, ps_linear, st_linear = setup_embedding(conf_linear)
y_linear = embed_grid(layer_linear, ps_linear, st_linear, xgrid)
embed_point(layer_linear, ps_linear, st_linear, x1)

4-element Vector{Float32}:
 -0.7141968
  0.40148556
  0.3199348
 -0.26569

PeriodicEmbeddings

Periodic embedding first lifts into learned sinusoids, then projects that basis to :

The frequencies are learned (Gaussian init, here scaled up so the waves are visible on  ). Each is a linear mixture of those sinusoids. lite=true shares across features; the figures use the default per-feature map.

K = 2
conf_periodic = PeriodicEmbeddings(;
    d_embedding, frequencies=K, frequencies_init_scale=0.4f0, activation=:identity, lite=false
)
layer_periodic, ps_periodic, st_periodic = setup_embedding(conf_periodic)
h_periodic = basis_grid(layer_periodic, ps_periodic, st_periodic, xgrid, :periodic)
y_periodic = embed_grid(layer_periodic, ps_periodic, st_periodic, xgrid)
embed_point(layer_periodic, ps_periodic, st_periodic, x1)

4-element Vector{Float32}:
  0.68084216
  0.6917712
 -0.07447859
  0.2205313

Basis — then :

Projected embedding:

PiecewiseLinearEmbeddings

Bin edges come from training quantiles. Each bin   contributes a saturating ramp, and a learned map sends that encoding to :

is left of , linear inside the bin, and right of , so each is piecewise linear with kinks at the edges. Version :A (plotted) is that map. Version :B adds a residual linear path and zero-initializes , so at init :B looks like LinearEmbeddings. This config requires x_train.

conf_piecewise = PiecewiseLinearEmbeddings(; d_embedding, bins=4, activation=:identity, version=:A)
bins = Embeddings.compute_bins(x_train; bins=4)[1]
layer_piecewise, ps_piecewise, st_piecewise = setup_embedding(conf_piecewise; x_train)
h_piecewise = basis_grid(layer_piecewise, ps_piecewise, st_piecewise, xgrid, :encoding)
y_piecewise = embed_grid(layer_piecewise, ps_piecewise, st_piecewise, xgrid)
(; bins, e_x1=embed_point(layer_piecewise, ps_piecewise, st_piecewise, x1))

(bins = Float32[-2.0, -1.0, 0.0, 1.0, 2.0], e_x1 = Float32[-0.6851754, 0.52889746, 0.77274066, -0.12734753])

Basis — one ramp per bin:

Projected embedding:

BatchNormEmbeddings

Width stays 1: the feature is standardized, then optionally rescaled.

On this figure and are the mean and variance of the displayed grid (as if that grid were one batch) and   at init, so is a centered, unit-scale copy of the raw line. During training those statistics come from each minibatch; at inference they are running averages.

conf_batchnorm = BatchNormEmbeddings()
layer_batchnorm, ps_batchnorm, st_batchnorm = setup_embedding(conf_batchnorm)
y_batchnorm = embed_grid(layer_batchnorm, ps_batchnorm, st_batchnorm, xgrid)
embed_point(layer_batchnorm, ps_batchnorm, st_batchnorm, x1)

1-element Vector{Float32}:
 0.0

IdentityEmbedding

The default numerical embedding: the backbone sees the raw column.

The curve matches the raw figure above; only the series color (embedding purple) changes.

conf_identity = IdentityEmbedding()
layer_identity, ps_identity, st_identity = setup_embedding(conf_identity)
y_identity = embed_grid(layer_identity, ps_identity, st_identity, xgrid)
embed_point(layer_identity, ps_identity, st_identity, x1)

1-element Vector{Float32}:
 0.5

Temporal embeddings

Temporal embedding is a separate branch on one time column, not a column-wise numerical map. It is always attached through EmbeddingLayer, which routes that column to temp and concatenates the result after the numerical features.

TemporalEmbeddings

Fixed harmonics (not learned frequencies) of periods are projected, then an optional standardized trend is appended:

Defaults assume a POSIX-seconds column (year, month, week, day). The figure uses short periods so the same   path shows several cycles. Build it through EmbeddingLayer with x_train (for ). Output width is   when trend=true; the last coordinate is the trend.

order = [2, 1]
periods = Float32[2, 4]
conf_temporal = EmbeddingLayer(; temp=TemporalEmbeddings(; index=1, order, periods, trend=true, d_embedding))
layer_temporal, ps_temporal, st_temporal = setup_embedding(conf_temporal; x_train)
h_temporal = temporal_basis(layer_temporal, ps_temporal, st_temporal, xgrid)
y_temporal = embed_grid(layer_temporal, ps_temporal, st_temporal, xgrid)
n_harm = sum(order)
embed_point(layer_temporal, ps_temporal, st_temporal, x1)

5-element Vector{Float32}:
 0.0
 0.0
 0.3734188
 0.0
 0.4313913

Basis — then of each harmonic:

Projected embedding —   , last panel is the trend:

EmbeddingLayer

The rest of this report built a numerical config for a single column. A real table has several numeric columns and, sometimes, one time column. EmbeddingLayer is the wrapper that says how those two kinds of columns are treated.

input columns:   [  x1   x2   x3   t  ]
                    └─────────┘    │
                         num       temp  (temp.index = 4)
                           │       │
                           └─ vcat ┘  →  backbone

• num — applied to every column except the time column. Default: IdentityEmbedding() (pass-through).

• temp — applied to one column, whose position is temp.index (1-based in the feature list). Default: nothing (no time branch).

You pass this object as embedding_config when fitting; fit then builds the chain. The snippets below call build_embedding_chain only to show widths.

Numerical only (equivalent to passing LinearEmbeddings directly, as earlier sections did):

EmbeddingLayer(; num=LinearEmbeddings(; d_embedding, activation=:identity))

EmbeddingLayer{LinearEmbeddings, Nothing}(LinearEmbeddings(4, :identity), nothing)

Temporal only — numeric columns pass through; column index is the timestamp:

EmbeddingLayer(; temp=TemporalEmbeddings(; index=1, d_embedding))

EmbeddingLayer{IdentityEmbedding, TemporalEmbeddings}(IdentityEmbedding(), TemporalEmbeddings(1, [4, 1, 7, 0], Float32[3.15576f7, 2.6298f6, 604800.0, 86400.0], true, 4))

Both — two columns, x1 then t. index=2 means “the second column is time”, not embedding width:

nfeats = 2          # columns in the table
time_column = 2     # t is column 2
conf_combo = EmbeddingLayer(;
    num=LinearEmbeddings(; d_embedding, activation=:identity),
    temp=TemporalEmbeddings(; index=time_column, order=[2], periods=Float32[4], trend=true, d_embedding),
)
x_train2 = hcat(collect(xgrid), collect(xgrid))  # (n_samples, nfeats): [x1 | t]
layer_combo = Embeddings.build_embedding_chain(conf_combo, nfeats; x_train=x_train2)

# widths: one numeric feature → d_embedding; time → d_embedding projection + 1 trend
n_numeric = nfeats - 1
width_num = n_numeric * d_embedding
width_temp = d_embedding + 1
batch = randn(Float32, nfeats, 4)  # (nfeats, batch) dummy input

(;
    width_num,
    width_temp,
    concatenated=width_num + width_temp,
    measured=Embeddings.embedding_width(layer_combo, batch, Xoshiro(seed)),
)

(width_num = 4, width_temp = 5, concatenated = 9, measured = 9)

The second argument of build_embedding_chain is nfeats (how many columns the matrix has), not temp.index. x_train is (n_samples, nfeats).

For hyperparameter search, a Dict is accepted: :embedding_type selects num (:linear, :periodic, :piecewise, :batchnorm, :layernorm, :identity), and :temporal => Dict(:index => 2, ...) builds temp. Unknown keys are ignored.