Formal Analysis of Compression Bounds

3.4 Formal Analysis of Compression Bounds

The intuitive token-counting arguments in Sections 3.1–3.2 demonstrate order-of-magnitude savings, but they leave open several theoretical questions:

• What is the information-theoretic lower bound on tokens required to make a decision?
• How close does Spaxiom's event-based representation get to this bound?
• Under what conditions does compression fail or degrade?
• Can we characterize the rate-distortion tradeoff between event vocabulary size and decision quality?

This section provides a more rigorous framework for analyzing Spaxiom's compression, drawing on information theory, rate-distortion theory, and algorithmic compression.

Information-theoretic lower bound

Consider an agent making a sequence of decisions D₁, D₂, ..., D_N over a time horizon T, based on sensor observations X₁, X₂, ..., X_M where M = S · f · T (S sensors, f Hz sampling, T seconds).

By the data processing inequality, any representation Z of the sensor stream (whether raw tokens or Spaxiom events) must satisfy:

where I(·;·) denotes mutual information. This states that any compressed representation Z cannot convey more information about decisions D than the raw observations X.

The minimum description length (in bits, convertible to tokens via tokens ≈ bits / log₂(vocab_size)) required to represent sufficient information for decision D is lower-bounded by:

where H(D | context) is the conditional entropy of the decision given any prior context (previous decisions, world model, task specification).

In practice, this lower bound is unattainable because:

• We don't know a priori which sensor observations are decision-relevant.
• The agent's decision function is unknown (learned via RL or prompted from LLM).
• Encoding must be online and causal: we cannot look ahead to future observations to compress past ones.

Rate-distortion analysis

Rate-distortion theory formalizes the tradeoff between compression rate (bits or tokens) and reconstruction distortion (decision quality loss).

Let:

• X = raw sensor stream (high-dimensional, continuous)
• Z = compressed representation (discrete events or tokens)
• R = |Z| = rate (number of tokens)
• D = 𝔼[d(X, decode(Z))] = expected distortion (reconstruction error or decision quality loss)

The rate-distortion function R(D) defines the minimum rate required to achieve distortion ≤ D:

For Gaussian sources with squared-error distortion, this has a closed form:

where σ² is the source variance. This logarithmic relationship means that each additional bit of compression exponentially increases distortion. Conversely, modest increases in event vocabulary can dramatically improve decision quality.

Spaxiom as lossy compression with semantic preservation

Spaxiom differs from classical lossy compression (JPEG, MP3) in that the distortion metric is not pixel-error or waveform-error, but decision-relevant semantic loss.

Define a semantic distortion metric:

where D̂ = policy(Z) is the decision made from compressed representation Z, D = policy(X) is the oracle decision from full observations, and ℓ is a task loss (e.g., regret, value gap, safety violations).

Spaxiom's design hypothesis is that:

In other words, for the same token budget, Spaxiom's semantically-aware event compression incurs much lower decision-quality loss than generic compression algorithms optimized for reconstruction error.

Event saturation and sublinear scaling

Figure 1 (Section 3.3) shows that Spaxiom's token count grows sublinearly with time horizon T, eventually saturating. We can model this mathematically.

Let E(T) be the number of events emitted over horizon T. Assume events are triggered by salient state transitions with rate λ(t) per second. Then:

Key insight: in many real-world domains, salient events occur at a bounded rate that does not scale with sensor count or sampling frequency. Examples:

• Occupancy: people enter/exit at ~0.1–10 Hz regardless of floor grid resolution (1000+ tiles sampled at 10 Hz).
• HVAC: temperature setpoint changes occur 1–10 times per day, not continuously.
• Queue: queue-length state transitions happen ~1–10/min even with 100 Hz depth cameras.
• Safety: near-miss events are rare (~0.01–1 per hour) despite ms-resolution collision detection.

In steady-state, λ(t) → λ_∞, a constant. This implies:

Thus tokens_intent ≈ λ_∞ · T · k_event, which is linear in T but with a slope determined by event rate, not sensor count.

Contrast with raw sensor tokens: tokens_raw ≈ S · f · T · k_value, linear in both T and S·f.

The compression ratio is:

This ratio is constant in T for large T, meaning Spaxiom provides consistent compression regardless of time horizon. When S·f ≫ λ_∞ (many sensors, sparse events), compression can be 100–10,000×.

Worst-case scenarios: when compression fails

Spaxiom's compression degrades or fails in several scenarios:

1. High-entropy environments: if sensor state changes unpredictably (e.g., turbulent fluid dynamics, molecular simulations), event rates λ approach or exceed sensor sampling rates S·f, eliminating compression.
2. Adversarial inputs: a malicious actor could inject sensor noise designed to trigger spurious events, inflating event count E(T).
3. Poorly designed event vocabularies: if events are too coarse-grained, they may not capture decision-relevant distinctions (underfitting). If too fine-grained, event count explodes (overfitting).
4. Continuous control: tasks requiring closed-loop control at sensor sampling rates (e.g., quadcopter stabilization at 1 kHz) cannot tolerate the latency of event abstraction. Here, raw sensor streams or model-based state estimation are more appropriate.

For these cases, we expect:

Compression vanishes. However, these represent a minority of embodied-agent scenarios. Most human-scale environments (buildings, hospitals, warehouses) exhibit the sparse-event structure that Spaxiom exploits.

Theoretical best-case compression

Assume an idealized scenario:

• Sensors: S = 1000, f = 10 Hz, k_value = 4 tokens/sample
• Events: λ_∞ = 1 Hz (1 event/second), k_event = 50 tokens/event
• Time horizon: T = 3600 s (1 hour)

Then:

This 800× compression is achievable when events are truly sparse. In practice, λ varies across domains:

• Elder care ADL: λ ≈ 0.01–0.1 Hz (sparse)
• Retail queue: λ ≈ 0.1–1 Hz (moderate)
• Warehouse safety: λ ≈ 1–10 Hz (dense)
• Data center thermal: λ ≈ 0.001–0.01 Hz (very sparse)

Resulting compressions range from 10× (dense events) to 10,000× (very sparse events).

Learning optimal event vocabularies

The choice of event types and granularity is currently manual (domain expert designs INTENT patterns). Future work could learn optimal event vocabularies via:

1. Vector quantization: treat events as discrete codes in a VQ-VAE. Learn codebook that minimizes reconstruction loss for downstream tasks.
2. Mutual information maximization: learn events that maximize I(E; D) (mutual information with decisions) while minimizing |E| (event count).
3. Reinforcement learning: train a meta-policy that proposes event types, evaluated by agent performance on downstream tasks.

Preliminary experiments suggest learned event vocabularies can achieve 1.5–3× better compression than hand-designed ones, at the cost of interpretability.

Connection to Kolmogorov complexity

From an algorithmic information theory perspective, Spaxiom's event stream can be viewed as a succinct program that generates decisions.

Let K(D | X) be the Kolmogorov complexity of decision sequence D given observations X: the length of the shortest program (in bits) that outputs D when given X as input.

Spaxiom's claim is effectively:

That is, the Spaxiom event abstraction preserves the algorithmic information relevant to decisions, despite massive compression of the raw observation stream.

This is analogous to how JPEG preserves the "semantic content" of an image (recognizable objects, scenes) while discarding high-frequency details irrelevant to human perception.

Empirical validation: token savings in production

To validate these theoretical arguments, we deployed Spaxiom in three production environments and measured actual token usage:

Key observations:

• Elder care achieves extreme compression (3286×) due to very sparse ADL events (λ ≈ 0.01 Hz).
• Retail has moderate compression (608×) with λ ≈ 1 Hz (queue state changes, customer movements).
• Warehouse has lowest compression (64×) due to dense safety events (λ ≈ 10 Hz) from forklift near-misses and trajectory monitoring.

All three deployments achieve 60–3000× compression, validating the theoretical model. The warehouse case demonstrates that even in dense-event scenarios, Spaxiom provides meaningful token savings.

Summary: compression bounds

To summarize the theoretical framework:

• Lower bound: H(D | context) bits (unattainable without oracle knowledge of decision function)
• Spaxiom achieves: λ_∞ · T · k_event tokens, linear in T but independent of sensor count S and sampling rate f
• Raw baseline: S · f · T · k_value tokens, linear in both T and S·f
• Compression ratio: C ≈ (S·f·k_value) / (λ_∞·k_event), constant for large T
• Typical range: 10–10,000× depending on event sparsity
• Failure modes: high-entropy environments, adversarial inputs, continuous control (λ → S·f)

This formal analysis grounds the intuitive token-counting arguments from earlier sections and provides a predictive model for when Spaxiom will (and won't) provide compression benefits.