Feature Catalog Example

Dit is een voorbeeld van hoe de feature metadata tabel eruit ziet na het uitvoeren van het feature
engineering systeem.

Feature Metadata Table Structure

date symbol feature period lag definition_script engine feature_type

2024-01-01 BTC close_poly2 NULL NULL [Link](close, 2) torch basic

2024-01-01 BTC close_log_log1p NULL NULL torch.log1p([Link](close)) torch basic

2024-01-01 BTC close_sqrt NULL NULL [Link]([Link](close)) torch basic

rolling_mean_torch(close,
2024-01-01 BTC close_roll_14_mean 14 NULL torch statistical
window=14)

rolling_std_torch(close,
2024-01-01 BTC close_roll_14_std 14 NULL torch statistical
window=14)

[Link]([[Link](1,
2024-01-01 BTC close_lag_1 NULL 1 torch statistical
device=device), close[:-1]])

ewm_torch(close,
2024-01-01 BTC close_ewm_20 20 NULL torch statistical
alpha=0.095)

2024-01-01 BTC close_rsi_14 14 NULL rsi(close, period=14) torch technical

2024-01-01 BTC close_sma_20 20 NULL sma(close, period=20) torch technical

2024-01-01 BTC close_multiply_volume NULL NULL [Link](close, volume) torch interaction

2024-01-01 ETH close_poly2 NULL NULL [Link](close, 2) torch basic

[Link](mean,
2024-01-01 ALL random_baseline_1 NULL NULL numpy baseline
std, size=(n_samples, 1))

Engine Mapping Examples

PyTorch (torch)

python

# Basic transforms
close_poly2 = [Link](close, 2)
close_log_log1p = torch.log1p([Link](close))
close_sqrt = [Link]([Link](close))

# Statistical features
close_roll_14_mean = rolling_mean_torch(close, window=14)
close_lag_1 = [Link]([[Link](1, device=device), close[:-1]])

# Interactions
close_multiply_volume = [Link](close, volume)
CuPy (cupy)

python

# Basic transforms
close_poly2 = [Link](close, 2)
close_log_log1p = cp.log1p([Link](close))

# Statistical features
close_roll_14_mean = [Link](close, [Link](14)/14, mode='same')

Polars (polars)

python

# Statistical features
close_roll_14_mean = [Link]('close').rolling_mean(14, min_periods=1)
close_lag_1 = [Link]('close').shift(1)

# Interactions
close_multiply_volume = [Link]('close') * [Link]('volume')

TSFresh (tsfresh)

python

# Advanced statistical features

close_fft_coefficient = tsfresh.feature_extraction.feature_calculators.fft_coefficient
close_autocorrelation = tsfresh.feature_extraction.feature_calculators.autocorrelation
close_linear_trend = tsfresh.feature_extraction.feature_calculators.linear_trend(close

PySpark (pyspark)

python

# Window functions
from [Link] import Window
from [Link] import avg, lag, stddev

window_spec = [Link]('symbol').orderBy('date').rowsBetween(-13, 0)
close_roll_14_mean = avg('close').over(window_spec)
close_lag_1 = lag('close', 1).over([Link]('symbol').orderBy('date'))

Pandas (pandas)
 python

# Statistical features
close_roll_14_mean = [Link](window=14, min_periods=1).mean()
close_ewm_20 = [Link](span=20).mean()
close_lag_1 = [Link](1)

# Interactions
close_multiply_volume = close * volume

Feature Type Categories

Basic Mathematical Transforms

• polynomial: x², x³, etc.

• logarithmic: log1p, log, log2, log10

• sqrt: √|x|

• reciprocal: 1/(|x| + ε)

• trigonometric: sin, cos, tan, tanh

Statistical Features
• rolling: mean, std, min, max, median, sum, var

• lag: shifted values with various periods

• ewm: exponentially weighted moving averages

• percentile: rolling quantiles (p25, p50, p75)

• zscore: rolling z-scores

• rank: rolling ranks

Technical Indicators
• rsi: Relative Strength Index

• sma/ema: Simple/Exponential Moving Averages

• bollinger: Bollinger Bands (upper, middle, lower)

• macd: MACD indicator

• atr: Average True Range

• stochastic: Stochastic oscillator

Interaction Features
 • arithmetic: add, multiply, divide, subtract

• ratio: col1/col2

• cross_correlation: rolling correlations between features

• polynomial_interaction: x1x2, x1x2*x3

Advanced Features (TSFresh)

• fft: Fourier transform coefficients

• autocorrelation: Autocorrelation functions

• entropy: Various entropy measures

• linear_trend: Linear trend parameters

• peak_detection: Peak and valley detection

• seasonality: Seasonal decomposition

Incremental Processing Examples

New Data Detection

sql

-- Query to find new records since last run

SELECT date, symbol, feature, COUNT(*) as feature_count
FROM feature_metadata
WHERE created_at > '2024-01-01T00:00:00'
GROUP BY date, symbol, feature
ORDER BY created_at DESC;

Engine Performance Comparison

sql

-- Average computation time by engine

SELECT
engine,
feature_type,
AVG(computation_time) as avg_time,
AVG(memory_usage_mb) as avg_memory,
COUNT(*) as feature_count
FROM feature_metadata
GROUP BY engine, feature_type
ORDER BY avg_time;

Feature Dependencies
 sql

-- Find features that depend on base 'close' price

SELECT feature, definition_script, engine
FROM feature_metadata
WHERE definition_script LIKE '%close%'
AND feature != 'close'
ORDER BY feature_type, feature;

Usage Statistics

By Engine
• torch: 65% (GPU-accelerated, fastest for large datasets)

• cupy: 15% (GPU-accelerated alternative)

• polars: 12% (CPU-optimized, memory efficient)

• pandas: 5% (Legacy compatibility)

• tsfresh: 2% (Advanced statistical features)

• pyspark: 1% (Distributed processing)

By Feature Type
• statistical: 45% (rolling, lag, ewm features)

• basic: 25% (mathematical transforms)

• interaction: 20% (feature combinations)

• technical: 8% (trading indicators)

• baseline: 2% (random baselines)

Performance Characteristics
Engine Avg Time (sec) Avg Memory (MB) Best Use Case

torch 0.034 312.5 Large datasets, GPU available

cupy 0.041 298.2 Medium datasets, GPU available

polars 0.058 145.3 Memory-efficient CPU processing

pandas 0.125 187.6 Small datasets, compatibility

tsfresh 0.892 523.1 Advanced statistical features

pyspark 0.234 89.7 Distributed, very large datasets

Incremental Update Examples

Scenario 1: New Daily Data

 bash

# Process only new day's data

python scripts/feature_engineering_incremental.py \
--current-input data/crypto_2024-[Link] \
--previous-input data/crypto_2024-[Link] \
--output data/features/incremental_2024-[Link]

Scenario 2: Modified Historical Data

bash

# Reprocess modified data with dependency tracking

python scripts/feature_engineering_incremental.py \
--current-input data/crypto_corrected.parquet \
--previous-input data/crypto_original.parquet \
--output data/features/incremental_corrected.parquet

Scenario 3: First Time Processing

bash

# Process entire dataset (no previous data)

python scripts/feature_engineering_incremental.py \
--current-input data/crypto_full.parquet \
--output data/features/initial_features.parquet

Feature Catalog Query Examples

Get All Features for Symbol

python

import polars as pl

# Load feature catalog

catalog = pl.read_csv('data/features/feature_catalog.csv')

# Get all BTC features

btc_features = [Link]([Link]('symbol') == 'BTC')
print(f"BTC has {btc_features.height} features")

Find Features by Type and Period

 python

# Get all 14-period rolling features computed with torch

rolling_14_torch = [Link](
([Link]('feature_type') == 'statistical') &
([Link]('period') == 14) &
([Link]('engine') == 'torch') &
([Link]('feature').[Link]('roll'))
)

Performance Analysis

python

# Compare engine performance for same feature types

performance = catalog.group_by(['engine', 'feature_type']).agg([
[Link]('computation_time').mean().alias('avg_time'),
[Link]('memory_usage_mb').mean().alias('avg_memory'),
[Link]('feature').count().alias('feature_count')
]).sort('avg_time')

Roadmap Implementation Status

Completed
GPU-accelerated basic transforms (torch, cupy)
Statistical features with multiple engines
Technical indicators
Feature interaction generation
Incremental processing pipeline
Feature metadata tracking
Memory-efficient chunked processing

In Progress
TSFresh integration for advanced features
PySpark distributed processing
Feature selection based on importance
Real-time streaming feature updates

Planned
Custom feature function plugins
MLflow experiment tracking integration
Feature store integration
Automated feature quality monitoring
Cross-validation aware feature engineering