Experience Embeddings and Multi-Modal Architecture

4.3 & 4.4 Experience Embeddings and Multi-Modal Architecture

Experience embeddings and RAG

Each event (or short event sequence) can be mapped to a vector embedding z ∈ ℝ^d:

• Textual description via an LLM + embedding model.
• Direct encoding from structured fields (e.g., a small MLP over numeric fields and one-hot types).

These embeddings form a vector index of experiences:

# Pseudocode: building an experience index
from some_vector_db import VectorIndex
from some_embedding_model import embed_event

index = VectorIndex(dim=768)

for event in spaxiom_event_stream():
    z = embed_event(event)
    index.add(id=event.id, vector=z, metadata=event.to_dict())

Now an agent can do experience RAG (retrieval-augmented generation):

• Query: "Similar fall events in wet restrooms with low lighting."
• Retrieve nearest neighbors in embedding space.
• Use these as context when deciding how to act or update policies.

This is distinct from web-document RAG: the corpus is sensor-grounded, and Spaxiom guarantees a consistent, typed schema.

Multi-Modal Embedding Architecture

Event tokenization strategies

Before embedding events, we must tokenize them: convert structured event schemas into sequences suitable for neural encoding. Spaxiom supports three complementary strategies:

1. Type-based tokenization

Treat each event as a discrete token based on its type and key attributes:

# Example: type-based tokens
event = {
    "type": "QueueFormed",
    "zone": "loading_dock",
    "length": 8,
    "timestamp": "2025-01-06T10:23:45Z"
}

# Tokenize as: [EVENT_TYPE, ZONE_ID, BUCKET(length), TIME_BUCKET]
tokens = [
    vocab["QueueFormed"],           # 1042
    vocab["zone:loading_dock"],     # 3521
    discretize(8, bins=[0,5,10,20]),  # bin_2 → 7892
    time_bucket("10:23", hour=True)   # hour_10 → 2341
]

This approach is simple and mirrors language modeling, but discards fine-grained numeric information.

2. Temporal binning

Aggregate events into fixed time windows (e.g., 5-minute bins) and represent each bin as a multi-hot vector:

where c_i is the count of event type i in the time window.

# Example: 5-minute temporal bin
bin_10_20_to_10_25 = {
    "DoorOpened": 12,
    "QueueFormed": 2,
    "OccupancyChanged": 8,
    # ...
}

# Encode as sparse vector
vector = sparse_vector(vocab_size=500)
vector[vocab["DoorOpened"]] = 12
vector[vocab["QueueFormed"]] = 2
vector[vocab["OccupancyChanged"]] = 8

This preserves event frequency but loses exact timing within the bin.

3. Spatial hashing

For spatially distributed events, hash zone coordinates into spatial tokens:

from spaxiom.embedding import spatial_hash

# Events with (x, y) coordinates
event = {"type": "FallEvent", "x": 12.5, "y": 8.3, "zone": "ward_b"}

# Hash to grid cell (resolution: 1m)
cell_id = spatial_hash(x=12.5, y=8.3, resolution=1.0)  # → "cell_12_8"

# Multi-scale hashing (0.5m, 1m, 2m, 4m)
tokens = [
    spatial_hash(12.5, 8.3, res=0.5),  # fine-grained
    spatial_hash(12.5, 8.3, res=1.0),
    spatial_hash(12.5, 8.3, res=2.0),
    spatial_hash(12.5, 8.3, res=4.0)   # coarse-grained
]

Multi-scale hashing enables models to learn hierarchical spatial patterns (e.g., falls occur near doorways at 0.5m scale, but cluster by wing at 4m scale).

Encoder architecture options

Once events are tokenized, we embed them into continuous vector space. Spaxiom supports multiple encoder architectures tailored to different modalities:

Transformer-based encoders (BERT-style)

Treat event sequences as "sentences" and apply masked event prediction (MEP):

from spaxiom.embedding import TransformerEventEncoder

# BERT-style encoder: 12 layers, 768-dim, 12 heads
encoder = TransformerEventEncoder(
    vocab_size=5000,        # Event types + zones + attributes
    d_model=768,
    n_layers=12,
    n_heads=12,
    max_seq_len=512         # Events in context window
)

# Input: sequence of event tokens
event_seq = [1042, 3521, 7892, 2341, ...]  # QueueFormed, zone, length, time

# Output: contextual embeddings
embeddings = encoder(event_seq)  # (seq_len, 768)

# Use [CLS] token embedding for sequence-level representation
z_seq = embeddings[0]  # (768,)

Pretraining objective: Mask 15% of events in a sequence, predict masked events from context.

Graph Neural Network encoders (for spatial graphs)

Model events as nodes in a spatiotemporal graph, with edges representing spatial proximity or temporal succession:

from spaxiom.embedding import SpatialTemporalGNN

# Graph structure:
# - Nodes: events with (type, zone, timestamp, x, y)
# - Edges: spatial (same zone), temporal (within 30s), causal (triggered by)

gnn = SpatialTemporalGNN(
    node_features=128,      # Initial node embedding dim
    edge_types=3,           # spatial, temporal, causal
    n_layers=4,             # GNN layers (message passing)
    output_dim=512
)

# Input: graph of events
graph = {
    "nodes": [
        {"type": "DoorOpened", "zone": "entrance", "t": 0},
        {"type": "OccupancyChanged", "zone": "entrance", "t": 2},
        {"type": "QueueFormed", "zone": "loading", "t": 10},
    ],
    "edges": [
        (0, 1, "temporal"),  # DoorOpened → OccupancyChanged
        (1, 2, "causal"),    # OccupancyChanged → QueueFormed
    ]
}

# Output: node embeddings after message passing
node_embeddings = gnn(graph)  # (3, 512)

# Aggregate for graph-level embedding
z_graph = node_embeddings.mean(dim=0)  # (512,)

Pretraining objective: Link prediction (predict missing edges), node attribute prediction.

Recurrent encoders (LSTM/GRU for temporal sequences)

For long event sequences with strong temporal dependencies:

from spaxiom.embedding import LSTMEventEncoder

encoder = LSTMEventEncoder(
    vocab_size=5000,
    embed_dim=256,
    hidden_dim=512,
    n_layers=2,
    bidirectional=True
)

# Input: event sequence (variable length)
event_seq = [1042, 3521, 7892, ...]  # (seq_len,)

# Output: final hidden state
z_seq = encoder(event_seq)  # (1024,) [512*2 for bidirectional]

Pretraining objective: Next-event prediction (language modeling on event streams).

Contrastive learning for event embeddings

To learn semantically meaningful embeddings without labeled data, Spaxiom uses contrastive learning inspired by SimCLR and triplet loss:

SimCLR-style contrastive learning

Generate positive pairs via data augmentation, negative pairs via random sampling:

from spaxiom.embedding import ContrastiveEventEncoder

encoder = ContrastiveEventEncoder(base_encoder=transformer_encoder)

# Augmentation strategies:
# 1. Time jitter: shift timestamps by ±30s
# 2. Zone dropout: randomly mask 10% of zone attributes
# 3. Event dropout: drop 5% of events in sequence
# 4. Spatial noise: add Gaussian noise to (x, y) coordinates

def augment(event_seq):
    return apply_random_augmentation(event_seq)

# Contrastive loss (InfoNCE)
def contrastive_loss(encoder, event_seq, temperature=0.07):
    # Create two augmented views
    z1 = encoder(augment(event_seq))  # (batch, 768)
    z2 = encoder(augment(event_seq))  # (batch, 768)

    # Compute similarity matrix
    sim_matrix = cosine_similarity(z1, z2) / temperature

    # Loss: maximize similarity of positive pairs, minimize negatives
    labels = torch.arange(batch_size)  # Diagonal = positive pairs
    loss = cross_entropy(sim_matrix, labels)
    return loss

Training: Sample 1M event sequences from production deployments, train encoder to maximize agreement between augmented views.

Triplet loss for fine-grained ranking

Learn embeddings that respect semantic similarity:

# Example triplet:
# Anchor:   QueueFormed(zone=loading, length=8, wait_time=120s)
# Positive: QueueFormed(zone=loading, length=9, wait_time=135s)  # Similar
# Negative: DoorOpened(zone=entrance)  # Different event type

anchor_event = {"type": "QueueFormed", "zone": "loading", "length": 8}
positive_event = {"type": "QueueFormed", "zone": "loading", "length": 9}
negative_event = {"type": "DoorOpened", "zone": "entrance"}

z_a = encoder(anchor_event)
z_p = encoder(positive_event)
z_n = encoder(negative_event)

loss = max(0, ||z_a - z_p||² - ||z_a - z_n||² + 0.5)

Triplet mining: Use hard negatives (events that are spatially/temporally close but semantically different) to improve discrimination.

Multi-modal fusion in embedding space

Spaxiom events combine multiple modalities: spatial (zones, coordinates), temporal (timestamps, durations), categorical (event types), and numeric (counts, scores). Fusion strategies:

Early fusion (concatenation)

# Encode each modality separately
z_spatial = spatial_encoder(zone, x, y)          # (128,)
z_temporal = temporal_encoder(timestamp, dur)    # (128,)
z_type = type_encoder(event_type)                # (128,)
z_numeric = numeric_encoder([count, score])      # (128,)

# Concatenate and project
z_concat = torch.cat([z_spatial, z_temporal, z_type, z_numeric])  # (512,)
z_fused = projection_head(z_concat)  # (768,)

Late fusion (cross-attention)

# Encode each modality as sequence
spatial_seq = spatial_encoder(zones)      # (n_zones, 128)
temporal_seq = temporal_encoder(events)   # (n_events, 128)

# Cross-attention: temporal attends to spatial
attn_output = cross_attention(
    query=temporal_seq,
    key=spatial_seq,
    value=spatial_seq
)  # (n_events, 128)

# Aggregate
z_fused = attn_output.mean(dim=0)  # (128,)

Hybrid fusion (modality-specific then joint)

from spaxiom.embedding import MultiModalFusion

fusion = MultiModalFusion(
    spatial_encoder=spatial_gnn,
    temporal_encoder=lstm_encoder,
    fusion_method="cross_attention",
    output_dim=768
)

event_data = {
    "zones": [...],        # Spatial graph
    "event_seq": [...],    # Temporal sequence
    "metadata": {...}      # Event types, attributes
}

z = fusion(event_data)  # (768,)

Dimensionality and scalability

Embedding dimensions

Trade-off between expressiveness and computational cost:

• 128D: Fast retrieval, suitable for edge devices (Raspberry Pi)
• 512D: Balanced performance for cloud-based RAG
• 1024D: High-fidelity for research, multi-modal fusion

# Dimensionality reduction (optional)
from sklearn.decomposition import PCA

# Train encoder at 1024D for expressiveness
z_high = encoder(event)  # (1024,)

# Reduce to 128D for deployment
pca = PCA(n_components=128)
pca.fit(z_high_dataset)
z_low = pca.transform(z_high)  # (128,) ~10x faster retrieval

Approximate nearest neighbor search

For large-scale RAG with millions of events, use efficient vector search:

import faiss

# Build FAISS index (HNSW for fast ANN)
index = faiss.IndexHNSWFlat(d=768, M=32)  # M = graph connectivity
index.add(embeddings)  # (n_events, 768)

# Query: find k=10 nearest events
query_embedding = encoder(query_event)  # (768,)
distances, indices = index.search(query_embedding[None, :], k=10)

# Retrieve events
similar_events = [event_store[i] for i in indices[0]]

Index size: For 1M events at 768D (float32), FAISS HNSW requires ~3.5 GB RAM. Quantization (e.g., IVF+PQ) reduces to ~500 MB with minimal recall loss.

Pre-training strategies

Masked event prediction (MEP)

# BERT-style pretraining on event sequences
def masked_event_prediction(encoder, event_seq, mask_prob=0.15):
    # Randomly mask 15% of events
    masked_seq = mask_random(event_seq, p=mask_prob)

    # Predict masked events
    logits = encoder.predict(masked_seq)  # (seq_len, vocab_size)

    # Loss: cross-entropy on masked positions
    loss = cross_entropy(logits[masked_positions], true_labels)
    return loss

# Train on 10M event sequences from 1000 sites
for epoch in range(10):
    for batch in event_dataloader:
        loss = masked_event_prediction(encoder, batch)
        loss.backward()
        optimizer.step()

Next-event prediction (NEP)

# Autoregressive pretraining (GPT-style)
def next_event_prediction(encoder, event_seq):
    # Predict next event given history
    for t in range(len(event_seq) - 1):
        context = event_seq[:t+1]
        z = encoder(context)
        logits = prediction_head(z)  # (vocab_size,)
        target = event_seq[t+1]
        loss += cross_entropy(logits, target)
    return loss / len(event_seq)

Spatial-temporal jigsaw

Shuffle event order, train model to reconstruct correct temporal sequence:

# Jigsaw pretext task
def spatiotemporal_jigsaw(encoder, event_seq):
    # Shuffle events (break temporal order)
    shuffled, permutation = shuffle(event_seq)

    # Predict original order
    z = encoder(shuffled)
    predicted_order = permutation_head(z)  # (seq_len, seq_len)

    # Loss: predict permutation matrix
    loss = cross_entropy(predicted_order, permutation)
    return loss

Fine-tuning for downstream tasks

After pretraining, fine-tune embeddings for specific applications:

Task 1: Fall risk prediction

# Fine-tune encoder for binary classification
pretrained_encoder = load_checkpoint("spaxiom_pretrained.pt")

# Add task-specific head
classifier = nn.Sequential(
    pretrained_encoder,
    nn.Linear(768, 256),
    nn.ReLU(),
    nn.Linear(256, 2)  # Binary: fall / no fall
)

# Fine-tune on labeled data (10k events with fall labels)
for epoch in range(5):
    for event, label in fall_dataset:
        z = classifier(event)
        loss = cross_entropy(z, label)
        loss.backward()
        optimizer.step()

Task 2: Event retrieval for RAG

# Fine-tune with in-batch negatives (DPR-style)
def retrieval_fine_tuning(query_encoder, event_encoder, query, positive_event, batch):
    q = query_encoder(query)               # (768,)
    e_pos = event_encoder(positive_event)  # (768,)
    e_neg = event_encoder(batch)           # (batch_size, 768)

    # Dot product similarity
    sim_pos = (q * e_pos).sum()
    sim_neg = (q @ e_neg.T)  # (batch_size,)

    # Loss: positive should rank higher than negatives
    loss = -log_softmax([sim_pos, *sim_neg])[0]
    return loss

Production deployment: embedding pipeline

from spaxiom.embedding import EmbeddingPipeline

# End-to-end pipeline: events → embeddings → vector DB
pipeline = EmbeddingPipeline(
    encoder=pretrained_encoder,
    tokenizer=event_tokenizer,
    index=faiss_index,
    batch_size=256,
    device="cuda"
)

# Stream events from Spaxiom runtime
for event in runtime.event_stream():
    # 1. Tokenize
    tokens = pipeline.tokenize(event)

    # 2. Encode
    z = pipeline.encode(tokens)

    # 3. Index
    pipeline.add_to_index(event_id=event["id"], embedding=z, metadata=event)

# Query at inference time
query = "Find similar queue events in loading zones during peak hours"
results = pipeline.search(query, k=10)

for result in results:
    print(f"Event: {result['type']}, Similarity: {result['score']:.3f}")
    print(f"Zone: {result['zone']}, Timestamp: {result['timestamp']}")

Evaluation metrics

Case study: hospital fall prediction with multi-modal embeddings

A 500-bed hospital deployed Spaxiom embeddings for fall risk prediction:

# Pretraining: 2M events from 10 hospitals (3 months)
encoder = TransformerEventEncoder(vocab_size=5000, d_model=768)
pretrain(encoder, dataset=hospital_events_2M, objective="masked_event_prediction")

# Fine-tuning: 8K labeled fall events from target hospital
classifier = FallRiskClassifier(encoder)
finetune(classifier, dataset=labeled_falls_8K, epochs=5)

# Deployment: real-time inference on edge (Jetson Nano)
for event in runtime.event_stream():
    if event["type"] in ["GaitInstability", "SlowWalking", "StandingStill"]:
        z = encoder(event)
        risk_score = classifier(z)

        if risk_score > 0.8:
            alert_staff(event["zone"], risk="HIGH")

# Results (6-month trial):
# - 67% reduction in falls (from 12/month to 4/month)
# - 82% precision, 74% recall for high-risk alerts
# - <20ms latency for inference (acceptable for real-time)

Future directions

• Foundation models: Train large-scale event encoders (10B+ events) as "experience foundation models"
• Cross-modal alignment: Align event embeddings with vision (camera frames), language (incident reports), and audio (alarms)
• Hierarchical embeddings: Learn multi-scale representations (event → sequence → episode → deployment)
• Continual learning: Update embeddings online as new event types emerge (zero-shot generalization)
• Privacy-preserving embeddings: Federated learning for embeddings without sharing raw events

Summary: Spaxiom's multi-modal embedding architecture transforms structured events into dense vector representations optimized for retrieval, prediction, and reasoning. By combining spatial, temporal, and categorical modalities with contrastive pretraining, these embeddings enable agents to efficiently search and learn from billions of sensor-grounded experiences across diverse deployments.