StrataFormer Architecture

StrataFormer is the backbone architecture for trading-specific foundation models.

The Analogy

General domain:          Trading domain:
─────────────────────    ─────────────────────────────────
Text corpus              OHLCV market data (any asset)
Word token               Bar token (one candle = one word)
Transformer              StrataFormer
LLM (GPT, Claude)        Trading Foundation Model

Just as GPT learns language by predicting masked words, StrataFormer learns market structure by predicting masked bars.

Architecture Diagram

Input: OHLCV sequence  (B, T, 5)  +  Asset ID  (B,)
              │
              ▼
  ┌─────────────────────────────────────────────┐
  │           StrataBarTokenizer                │
  │                                             │
  │  bar_proj(OHLCV) → embed_dim                │
  │  + asset_embed(asset_id) → embed_dim        │
  │  + pos_embed(position)   → embed_dim        │
  │                                             │
  │  Output: (B, T, embed_dim)  token sequence  │
  └─────────────────────────────────────────────┘
              │
              ▼
  ┌─────────────────────────────────────────────┐  ×N
  │      StrataCausalAttentionBlock              │
  │                                             │
  │  MultiHeadAttention (causal mask)           │
  │  → bar t can ONLY attend to bars 0..t-1     │
  │  → no future price leakage                  │
  │                                             │
  │  + FFN (expand → GELU → contract)           │
  │  + LayerNorm + residual connections         │
  │                                             │
  │  Output: (B, T, embed_dim)                  │
  └─────────────────────────────────────────────┘
              │
          take last token
              │
              ▼
  ┌─────────────────────────────────────────────┐
  │        StrataStateBottleneck                │
  │                                             │
  │  embed_dim → bottleneck_dim → 4             │
  │                                             │
  │  h[0] = bias          tanh  → [-1,  1]      │
  │  h[1] = momentum      sigmoid → [0, 1]      │
  │  h[2] = trap_risk     sigmoid → [0, 1]      │
  │  h[3] = uncertainty   sigmoid → [0, 1]      │
  │                                             │
  │  Output: (B, 4) interpretable market state  │
  └─────────────────────────────────────────────┘
              │
              ▼
  ┌─────────────────────────────────────────────┐
  │           StrataFormerHead                  │
  │                                             │
  │  → action_logits  (B, 3)  LONG/SHORT/HOLD   │
  │  → confidence     (B, 1)  [0, 1]            │
  │  → regime_logits  (B, 4)  TRENDING/...      │
  └─────────────────────────────────────────────┘

Key Design Decisions

1. Causal attention (no future leakage)

Every attention block uses an upper-triangular mask — bar t can only attend to bars 0, 1, ..., t-1. This is critical for trading:

• Prevents the model from "cheating" by looking at future prices
• Mirrors how traders actually operate — only past data is available
• Enables autoregressive inference (process bars one-by-one in real time)

2. Variable context window

Unlike StrataNet's fixed seq_len=30, StrataFormer accepts any sequence length up to max_seq_len. This allows:

• Short inference: 30-bar windows for fast signals
• Long inference: 512-bar windows for macro regime detection
• Same model weights work for both

3. Multi-asset tokenizer

The StrataBarTokenizer adds a learned asset embedding to every bar:

token = bar_projection(OHLCV)
      + asset_embedding(asset_id)   ← learned per-ticker bias
      + positional_encoding(t)

This means the model learns that TSLA bars behave differently from SPY bars, even when the OHLCV values are numerically similar.

4. Interpretable bottleneck (the key differentiator)

After the attention stack, a bottleneck layer compresses the high-dimensional attention output into the same 4-dim interpretable state as StrataNet.

This is what distinguishes StrataFormer from generic Transformers like TimeGPT:

Model	Hidden state	Interpretable?
GPT-4	12,288-dim	❌ opaque
TimeGPT	512-dim	❌ opaque
StrataNet	4-dim	✅ semantic
StrataFormer	4-dim	✅ semantic

At any inference step, you always know why the model decided LONG/SHORT/HOLD.

Pretraining: Masked Bar Modeling (MBM)

Analogous to BERT's Masked Language Modeling:

Original sequence:   [bar₀] [bar₁] [bar₂] [bar₃] [bar₄]
Masked sequence:     [bar₀] [MASK] [bar₂] [MASK] [bar₄]
                                ↓
              StrataFormer predicts:
              [bar₀] [bar₁̂] [bar₂] [bar₃̂] [bar₄]
                                ↓
              Loss = MSE(bar₁̂, bar₁) + MSE(bar₃̂, bar₃)

Why this works: to reconstruct a masked bar, the model must understand: - Price relationships between bars (momentum, mean reversion) - Volatility patterns (ATR expansion/contraction) - Volume-price relationships - Regime context (trending vs ranging)

All of this is learned without any human labels — just raw OHLCV data.

Parameter Scaling

StrataFormer is designed to scale — larger configs learn richer market representations:

Config	Layers	Heads	embed_dim	Parameters	Use case
Tiny	2	2	32	~15K	Testing, small datasets
Small	2	4	64	~92K	Single asset, short history
Base	4	8	128	~680K	Multi-asset, 1-2 years
Large	6	8	256	~5M	Foundation model, large corpus

All configs share the same 4-dim interpretable bottleneck — scaling adds capacity but never loses interpretability.

Comparison: StrataFormer vs StrataNet vs Transformer

Dimension	Transformer (GPT)	StrataNet	StrataFormer
Domain	General text	Trading	Trading
Context	Variable	Fixed (30)	Variable (up to 2048)
Multi-asset	N/A	❌	✅
Pretraining	Next-token prediction	STRATA teacher	Masked Bar Modeling
Hidden state	Opaque	4-dim semantic	4-dim semantic
Output	Text tokens	LONG/SHORT/HOLD	LONG/SHORT/HOLD
Interpretable	❌	✅	✅
Parameters	Billions	~5K	~15K–5M