Catalyst Voronoi Model
SUBLATE

Catalyst

Parquet-Based ETL Engine

The universal columnar substrate and embedded Lakehouse pipeline for zero-copy data interchange, in-memory analytics, and agentic AI.

Origin & Motivation // From the Founder

Building Local AI for eDiscovery from First Principles

As an eDiscovery attorney specializing in supervised machine learning, I set out to run a local AI on my Mac. After a year of testing a variety of "proven" approaches to build a reliable Retrieval-Augmented Generation (RAG) pipeline, the proper architecture emerged by simply eliminating complexity and returning to first principles.

Most enterprise RAG is built on vibes. With no consensus on best practices and zero evidentiary standards, pipelines rely on arbitrary text chunking and loose vector lookups, leaving autonomous agents prone to hallucination and semantic drift.

Applying the rigid requirements of EDRM ingestion to the Extract, Transform, Load (ETL) engine needed for agentic RAG simply makes the most sense. The Electronic Discovery Reference Model spent over twenty years under federal court scrutiny solving this exact problem. When you ground an agent in strictly typed metadata, cryptographic SHA-256 custody, deterministic PII redaction, and structured columnar context, you don't have to hope the agent stays in line—the data architecture enforces it.

Traditional client-server Database Management Systems (DBMS) proved to be the biggest failure point at scale. As Turing Award laureate Michael Stonebraker demonstrated in The End of an Architectural Era (VLDB 2007) and OLTP Through the Looking Glass (SIGMOD 2008), legacy relational engines spend upwards of 96% of CPU cycles on bureaucratic overhead—buffer pool latching, lock managers, and tuple unpacking—leaving only ~4% for actual computation. When tasked with high-dimensional analytics and unstructured text, row-based architectures fail catastrophically. Worse, they mechanically forfeit modern CPU acceleration: while Single Instruction Multiple Data (SIMD) has existed for decades, systems like SQL Server interleave wide text and metadata across CPU cache lines, forcing hardware into 1980s scalar loops.

The solution is eliminating the standalone DBMS entirely. By adopting Apache Arrow and the Parquet columnar storage format, the file itself becomes the database. Catalyst runs 100% locally, air-gapped, and aggressively tuned to Apple Silicon. Instead of relying on a monolithic DBMS, we use ephemeral compute engines like DuckDB, LanceDB, and Tantivy. Because they run embedded directly in-process, they read only the exact columns they need—delivering an immediate 21x speed multiplier on every core with zero-copy speed.

Architectural Briefing // The Zero-Copy Columnar Substrate

The Columnar Shift: Why Catalyst Replaced MS SQL Server with Apache Parquet

Twenty years ago, Microsoft SQL Server was the only real database option for enterprise eDiscovery. For over two decades, enterprise discovery platforms have bolted proprietary components on top of this transactional engine—adding dtSearch for keywords, Lucene for clustering, and external vector stores for embeddings. As proved in seminal MIT/Brown research on Column-Stores vs. Row-Stores (Abadi et al., SIGMOD 2008) and C-Store (Stonebraker et al., VLDB 2005), attempting to execute analytical workloads on row-oriented substrates results in a fragile relational Frankenstein plagued by inter-process serialization, out-of-process join crashes, and 8KB page fragmentation across costly server farms.

Catalyst re-engineered the discovery pipeline from first principles around Apache Arrow, Apache Parquet, and embedded analytical engines—delivering instant multi-modal search, one-third the storage footprint, and up to 99.7% query I/O and data-scan compute reduction on sovereign, air-gapped local workstation hardware.

The Open Standard // Industry Consensus

Leave SQL Server to Banks. Parquet is the Analytical Format of Choice.

The entire modern data science, AI, and hyperscale analytics ecosystem has standardized on Apache Parquet for disk storage and Apache Arrow for in-memory compute. Even Microsoft itself chose Parquet over SQL Server to power its modern analytics in Microsoft Fabric and Azure Synapse. Google, Amazon, Snowflake, and Databricks all store and query data in Parquet.

Transactional Row OLTP

Where MS SQL Server Belongs

Engineered for atomic, single-row ledger updates, credit card processing, and ATM transactions (OLTP). Demands synchronous row-level write locks and 8KB page fetches. Forcing a 20-year-old banking database to ingest, embed, and search 1,000,000 legal documents with high-dimensional AI vectors causes massive latency, page fragmentation, and multi-day bottlenecks.

Designed For Retail Banks, ATM Networks, SAP / ERP Financial Ledgers, Point-of-Sale (POS) Checkouts.
Columnar Analytical OLAP & AI

The Universal Standard (Parquet & Arrow)

Engineered specifically for bulk analytical queries, vector embeddings, high-throughput token crunching, and wide-table projections. Queries read only the requested columns directly off disk with zero-copy shared memory across Python, Rust, and Swift.

Standard Relational Stack

The Row-Based Relational Stack

  • Substrate: Microsoft SQL Server (Row-oriented OLTP). Built for transactional banking; forces every query to drag entire 8KB pages off disk into memory even when projecting a single field.
  • Bolted-On Search: dtSearch for keyword indexing, Lucene/Elasticsearch for text clusters, external vector stores (Pinecone, Milvus) for embeddings.
  • Multi-Modal Latency: Tri-engine searches require separate queries across SQL metadata, dtSearch keywords, and vector databases—incurring constant network RPC, serialization/deserialization, and lock contention.
  • Page Bloat & Joins: Relational page segmentation, clustered index thrashing, and cascading join overhead across millions of rows.
  • Infrastructure Burden: Demands multi-node server farms, dedicated DBAs, expensive proprietary per-core SQL Server licenses, and third-party dtSearch seat fees.
Catalyst Modern Architecture

The Zero-Copy Columnar Engine

  • Substrate: Apache Parquet (columnar analytical storage) and Apache Arrow (zero-copy shared memory). Serverless, embedded, and 100% open-standard.
  • Co-Located Multi-Modal Engines: Unifies DuckDB (relational/metadata SQL), LanceDB (dense vector similarity), and Tantivy (full-text search) directly over the same Arrow stream.
  • Direct Column Projection: Queries read only the requested columns directly off disk. Reading 2 columns out of a 150-column table costs literally zero RAM or I/O for the other 148 columns.
  • Predicate Pushdown & Footers: Parquet file footers store row-group min/max statistics and bloom filters. Engines skip up to 99.7% of data without loading chunks into memory.
  • 1/3 Storage Footprint: Columnar dictionary encoding, RLE, and Zstandard compression reduce raw corpora to ~33% of traditional database file sizes.
Evidence src/catalyst/services/ingestion/ui_manifest_builder_service.py
Python // DuckDB In-Process Projection
Zero-Copy Parquet Serving: Ephemeral DuckDB analytical views mount contiguous Parquet partitions directly off NVMe without data copying, deserialization, or row-tuple unpacking.
def _build_views(self) -> list[str]:
    # Declarative virtual views mounted directly over zero-copy Parquet files
    static_views = {
        self.visual_analytics_dir / "va_comm_threads.parquet":
            "CREATE OR REPLACE VIEW va_comm_threads AS SELECT * FROM read_parquet('{basePath}/visual_analytics/va_comm_threads.parquet');",
        self.visual_analytics_dir / "va_cluster_hierarchy.parquet":
            "CREATE OR REPLACE VIEW va_cluster_hierarchy AS SELECT * FROM read_parquet('{basePath}/visual_analytics/va_cluster_hierarchy.parquet');",
        self.visual_analytics_dir / "va_document_registry.parquet":
            "CREATE OR REPLACE VIEW va_document_registry AS SELECT * FROM read_parquet('{basePath}/visual_analytics/va_document_registry.parquet');",
    }
    for file_path, view_sql in static_views.items():
        if file_path.exists():
            views.append(view_sql)
    return views
Evidence src/catalyst/services/vector_buffer_service.py
Python // Zero-Copy PyArrow C-Buffer
Bypassing .tolist() Object Unpacking: Transferring 1.25M dense embedding vectors (256D) directly from contiguous C-memory into Arrow FixedSizeListArray eliminates hundreds of millions of ephemeral Python heap allocations for a 21× ingestion speedup.
def to_arrow_fixed_size_list(embeddings: np.ndarray, dim: int = 256) -> pa.FixedSizeListArray:
    """Zero-copy contiguous C-buffer wrap into Arrow FixedSizeListArray without .tolist() heap unpacking."""
    contiguous = np.ascontiguousarray(embeddings, dtype=np.float32)
    flat_buffer = pa.py_buffer(contiguous.ravel())
    value_array = pa.Array.from_buffers(pa.float32(), contiguous.size, [None, flat_buffer])
    return pa.FixedSizeListArray.from_arrays(value_array, dim=dim)
Interactive Memory Architecture Simulator // 1,041,752 Document Benchmark
Physical Disk I/O & Memory Bus Projection
Benchmark Query:
> EXPLAIN ANALYZE SELECT custodian, COUNT(*), SUM(size_bytes) FROM 'showcase_manifest.parquet' GROUP BY custodian ORDER BY COUNT(*) DESC;
Parquet Column Chunks (Contiguous On-Disk Partitions) 1,041,752 Records • evidence/showcase_manifest.parquet
custodianActive Stream
421.6 KBRLE Dictionary
size_bytesActive Stream
1.88 MBInt64 Array
sha256Skipped (0 B)
35.58 MBUntouched on NVMe
source_pathSkipped (0 B)
9.34 MBUntouched on NVMe
showcase_pathSkipped (0 B)
9.84 MBUntouched on NVMe
source_setSkipped (0 B)
4.45 KBUntouched on NVMe
custodian_basisSkipped (0 B)
7.93 KBUntouched on NVMe
Projection pushdown active: DuckDB reads only the 421.6 KB custodian dictionary and 1.88 MB size array directly into prosumer unified memory. 233.7 MB of file paths and cryptographic hashes are never touched on disk.
Data Loaded Off Disk
2.30 MB
99.0% I/O skipped from NVMe
Cache Line Efficiency
100%
Contiguous SIMD Neon vectors
Execution Engine
In-Process DuckDB
Vectorized hash aggregate
Measured Query Latency
0.0218s 85× Faster
21.8 ms (Apple Silicon)
Hardware Microarchitecture // The Physics of Execution

30 Years of Untapped Silicon: Why Row Databases Foreclose SIMD

For nearly three decades, every personal workstation and server has shipped with supercomputer-grade vector processors sitting completely idle. Relational databases like MS SQL Server are mechanically foreclosed from using Single Instruction, Multiple Data (SIMD) because interleaved row tuples cause catastrophic CPU cache pollution. Columnar memory feeds vector registers what they were physically engineered to consume.

Mechanical Foreclosure // Scalar Loops

Legacy SQL Server: 99.8% Cache Pollution

Born in 1989, SQL Server was hardcoded around single-core, tuple-at-a-time (Volcano) loops years before MMX (1997), AltiVec (1999), or NEON (2004) brought vector registers to desktop silicon, as detailed in the MonetDB/X100 hyper-pipelining analysis.

Physical Memory Layout (Interleaved Tuples)
[4B ID] [8B Date] [20B Custodian] [15,000B Email Body]
  • Severe Cache Pollution: Querying WHERE Custodian = 'Smith' pulls 64-byte or 128-byte cache lines filled with unneeded email body text. As documented in Ulrich Drepper's What Every Programmer Should Know About Memory, over 99% of CPU bus bandwidth is squandered transferring data the query never asked for.
  • No Contiguous Vector Loading: Because matching fields are separated by thousands of bytes of text, the CPU cannot load a homogeneous vector register.
  • Scalar Execution Stalls: The CPU falls back to scalar pointer-hopping: inspect one row, branch, jump over text, inspect next row. The processor pipeline constantly stalls waiting on DRAM latency.
~300M rows / sec
Per-core throughput ceiling on 64-bit metadata scans due to cache line misses and scalar branch stalls.
Hardware Saturation // 21.3x Multiplier

Catalyst Columnar: 102.4 Billion Evals / Sec

By eliminating DBMS daemons and storing data in contiguous Apache Arrow / Parquet columns, Catalyst activates the dormant SIMD vector engines inside Apple Silicon.

Physical Memory Layout (Homogeneous Array)
[8B Date1] [8B Date2] [8B Date3] [8B Date4] [8B Date5]...
  • 100% Cache Line Density: All custodian IDs and timestamps are packed edge-to-edge. Every 64-byte L1 cache line pulled into the core contains 100% actionable query data. Zero wasted bus bandwidth.
  • Direct NEON Vector Loading: Contiguous memory slides directly into 128-bit ARM NEON vector registers on Apple Silicon without conversion, unpacking, or pointer chasing.
  • Vectorized Hardware Math: A single CPU instruction evaluates 8, 16, or 32 records simultaneously in hardware registers. Millions of filter comparisons execute in a fraction of a millisecond.
6.4B / core  →  102.4 Billion / sec
3.2 GHz × 2 (64-bit int) = 6.4B rows/sec/core (21.3× speedup). Across 16 P-Cores with zero-copy UMA: 102.4 Billion evals/sec.
Evidence src/catalyst/services/ingestion/gnn_ingestion_orchestrator.py
Python // Apple Silicon Metal MPS Tensor
Direct Vector Silicon Saturation: Tensor scoring and cumulative trapezoidal integration execute natively on Apple Silicon Metal Performance Shaders with zero CPU-bus synchronization stalls.
def compute_mps_auc_roc(pos_scores: torch.Tensor, neg_scores: torch.Tensor) -> float:
    """Compute AUC-ROC natively on the MPS device with zero CPU synchronization."""
    scores = torch.cat([pos_scores, neg_scores])
    labels = torch.cat([torch.ones_like(pos_scores), torch.zeros_like(neg_scores)])

    # Vector sorting and cumulative trapezoidal summation run edge-to-edge on Apple Silicon
    sorted_indices = torch.argsort(scores, descending=True)
    sorted_labels = labels[sorted_indices]

    tps = torch.cumsum(sorted_labels, dim=0)
    fps = torch.cumsum(1.0 - sorted_labels, dim=0)

    tpr = tps / pos_scores.size(0)
    fpr = fps / neg_scores.size(0)

    return float(torch.sum((fpr[1:] - fpr[:-1]) * tpr[:-1]).item())
Evidence src/catalyst/runtime/qos_scheduler.py
Python // Darwin LibSystem C-FFI
Escaping macOS E-Core Confinement: macOS Darwin kernel by default assigns background Python CLI tasks to low-power Efficiency Cores. Calling pthread_set_qos_class_self_np(0x21, 0) forces the OS scheduler to dispatch workers to all 16 physical Performance Cores.
def elevate_process_qos() -> bool:
    """Elevate worker QoS from background throttling to QOS_CLASS_USER_INTERACTIVE (0x21)."""
    if platform.system() == "Darwin":
        libc = ctypes.CDLL("libSystem.B.dylib")
        # QOS_CLASS_USER_INTERACTIVE = 0x21 pins execution to Apple Silicon Performance Cores
        ret = libc.pthread_set_qos_class_self_np(0x21, 0)
        os.setpriority(os.PRIO_PROCESS, os.getpid(), 0)
        return ret == 0
    return False
Why Big Data & ML Mandate Parquet/Arrow

Modern data science (PyTorch, Hugging Face, Ray, Polars, Databricks) standardized on Apache Arrow because it eliminates serialization between Python, Rust, and C++. Extracting text, 256D/1536D vectors, and graph topologies turns raw data into dense columnar Arrow arrays where models and analytics share memory at zero copy cost.

Universal Open Interoperability

Catalyst outputs pristine, strictly typed Apache Parquet tables with zero proprietary lock-in. Datasets are directly mountable in Snowflake, Databricks, DuckDB, AWS Athena, or native Swift desktop clients without conversion, re-indexing, or migration pipelines.

Sovereign Air-Gapped Local Compute

Big data processing no longer requires cloud server farms. Catalyst runs the entire multi-stage ETL pipeline locally on prosumer workstations (Apple Silicon M1/M2/M3/M4 Ultra/Max). 100% offline, zero cloud egress, immune to network failure or third-party subpoenas, and fully defensible under attorney-client privilege.

eDiscovery Run Telemetry // Physical Storage Audit

Showcase Execution Run Summary & Lakehouse Manifest

Audit metrics from processing the showcase legal corpus on prosumer workstation hardware. Pre-ingestion collection payload versus post-processing columnar lakehouse storage.

The Collection // Raw Payload

Pre-Ingestion Source Corpus

Source Files 1,041,752
Actual Raw Bytes 1,963,662,106 B
Size on Disk (APFS) ~2.1 GB
Custodians 164 accounts
SHA-256 Deduplication 94,277 dupes skipped
Unique Documents 947,475 unique
File Breakdown 517K .eml | 250K .txt
The Scalpel Pass // NLP Topology

Tokenization & Intelligence

Semantic Chunks 1,251,230
Evaluated Tokens 147,969,548
Pipeline Throughput 1,088,219 WPM
Single-Paragraph Intact 586,228 (46.9%)
Deterministic Entities 427,647 entities
Celestial Clusters 11,859 topics
PyTorch Graph Link AUC 0.8897 ROC
The Lakehouse // Analytical Substrate

Post-Processing Storage

UI Parquet Tables 14 files (1.1 GB)
Processing Parquet 3 files (7.8 GB)
DuckDB Serving Views 16 mounted views
DuckDB Index Store 6.4 GB (embedded)
LanceDB Vector Index 1.25M vectors (3.9 GB)
Icebug CSR Graph 378 MB
Total Parquet Tables 17 columnar tables
Evidence sublate-vantage/Vantage/Models/LakehouseManifest.swift
Swift // Declarative Codable Contract
Cross-Boundary Type Enforcement: Catalyst's Lakehouse topology is validated against a strict Codable contract ensuring zero silent schema drift across Python, Rust, and native Swift review UI.
public struct LakehouseManifest: Codable {
    public let createdAt: String
    public let manifestVersion: String
    public let namespace: String
    public let pipelineStages: [PipelineStage]?
    public let servingEngine: String?       // e.g. 'namespace_index.duckdb'
    public let status: String              // 'COMPLETE'
    public let tables: [String: LogicalTable]? // Physical Parquet partitions
    public let telemetry: [String: Telemetry]?
}
Ground Truth Execution Trace
Inspect the unedited 4,309-line live execution log and architectural deep-dive annotations of Catalyst processing 1,041,752 source files (1.96 GB payload, 1.25M chunks) in 6,118.83s (1h 42m) on a single prosumer Apple Silicon M1 Ultra (16 P-Cores @ 100% saturation, 0 bytes cloud egress).