Building Local AI for eDiscovery from First Principles
As an eDiscovery attorney specializing in supervised machine learning, I set out to run a local AI on my Mac. After a year of testing a variety of "proven" approaches to build a reliable Retrieval-Augmented Generation (RAG) pipeline, the proper architecture emerged by simply eliminating complexity and returning to first principles.
Most enterprise RAG is built on vibes. With no consensus on best practices and zero evidentiary standards, pipelines rely on arbitrary text chunking and loose vector lookups, leaving autonomous agents prone to hallucination and semantic drift.
Applying the rigid requirements of EDRM ingestion to the Extract, Transform, Load (ETL) engine needed for agentic RAG simply makes the most sense. The Electronic Discovery Reference Model spent over twenty years under federal court scrutiny solving this exact problem. When you ground an agent in strictly typed metadata, cryptographic SHA-256 custody, deterministic PII redaction, and structured columnar context, you don't have to hope the agent stays in line—the data architecture enforces it.
Traditional client-server Database Management Systems (DBMS) proved to be the biggest failure point at scale. As Turing Award laureate Michael Stonebraker demonstrated in The End of an Architectural Era (VLDB 2007) and OLTP Through the Looking Glass (SIGMOD 2008), legacy relational engines spend upwards of 96% of CPU cycles on bureaucratic overhead—buffer pool latching, lock managers, and tuple unpacking—leaving only ~4% for actual computation. When tasked with high-dimensional analytics and unstructured text, row-based architectures fail catastrophically. Worse, they mechanically forfeit modern CPU acceleration: while Single Instruction Multiple Data (SIMD) has existed for decades, systems like SQL Server interleave wide text and metadata across CPU cache lines, forcing hardware into 1980s scalar loops.
The solution is eliminating the standalone DBMS entirely. By adopting Apache Arrow and the Parquet columnar storage format, the file itself becomes the database. Catalyst runs 100% locally, air-gapped, and aggressively tuned to Apple Silicon. Instead of relying on a monolithic DBMS, we use ephemeral compute engines like DuckDB, LanceDB, and Tantivy. Because they run embedded directly in-process, they read only the exact columns they need—delivering an immediate 21x speed multiplier on every core with zero-copy speed.
The Columnar Shift: Why Catalyst Replaced MS SQL Server with Apache Parquet
Twenty years ago, Microsoft SQL Server was the only real database option for enterprise eDiscovery. For over two decades, enterprise discovery platforms have bolted proprietary components on top of this transactional engine—adding dtSearch for keywords, Lucene for clustering, and external vector stores for embeddings. As proved in seminal MIT/Brown research on Column-Stores vs. Row-Stores (Abadi et al., SIGMOD 2008) and C-Store (Stonebraker et al., VLDB 2005), attempting to execute analytical workloads on row-oriented substrates results in a fragile relational Frankenstein plagued by inter-process serialization, out-of-process join crashes, and 8KB page fragmentation across costly server farms.
Catalyst re-engineered the discovery pipeline from first principles around Apache Arrow, Apache Parquet, and embedded analytical engines—delivering instant multi-modal search, one-third the storage footprint, and up to 99.7% query I/O and data-scan compute reduction on sovereign, air-gapped local workstation hardware.
Leave SQL Server to Banks. Parquet is the Analytical Format of Choice.
The entire modern data science, AI, and hyperscale analytics ecosystem has standardized on Apache Parquet for disk storage and Apache Arrow for in-memory compute. Even Microsoft itself chose Parquet over SQL Server to power its modern analytics in Microsoft Fabric and Azure Synapse. Google, Amazon, Snowflake, and Databricks all store and query data in Parquet.
Where MS SQL Server Belongs
Engineered for atomic, single-row ledger updates, credit card processing, and ATM transactions (OLTP). Demands synchronous row-level write locks and 8KB page fetches. Forcing a 20-year-old banking database to ingest, embed, and search 1,000,000 legal documents with high-dimensional AI vectors causes massive latency, page fragmentation, and multi-day bottlenecks.
The Universal Standard (Parquet & Arrow)
Engineered specifically for bulk analytical queries, vector embeddings, high-throughput token crunching, and wide-table projections. Queries read only the requested columns directly off disk with zero-copy shared memory across Python, Rust, and Swift.
The Row-Based Relational Stack
- Substrate: Microsoft SQL Server (Row-oriented OLTP). Built for transactional banking; forces every query to drag entire 8KB pages off disk into memory even when projecting a single field.
- Bolted-On Search: dtSearch for keyword indexing, Lucene/Elasticsearch for text clusters, external vector stores (Pinecone, Milvus) for embeddings.
- Multi-Modal Latency: Tri-engine searches require separate queries across SQL metadata, dtSearch keywords, and vector databases—incurring constant network RPC, serialization/deserialization, and lock contention.
- Page Bloat & Joins: Relational page segmentation, clustered index thrashing, and cascading join overhead across millions of rows.
- Infrastructure Burden: Demands multi-node server farms, dedicated DBAs, expensive proprietary per-core SQL Server licenses, and third-party dtSearch seat fees.
The Zero-Copy Columnar Engine
- Substrate: Apache Parquet (columnar analytical storage) and Apache Arrow (zero-copy shared memory). Serverless, embedded, and 100% open-standard.
- Co-Located Multi-Modal Engines: Unifies DuckDB (relational/metadata SQL), LanceDB (dense vector similarity), and Tantivy (full-text search) directly over the same Arrow stream.
- Direct Column Projection: Queries read only the requested columns directly off disk. Reading 2 columns out of a 150-column table costs literally zero RAM or I/O for the other 148 columns.
- Predicate Pushdown & Footers: Parquet file footers store row-group min/max statistics and bloom filters. Engines skip up to 99.7% of data without loading chunks into memory.
- 1/3 Storage Footprint: Columnar dictionary encoding, RLE, and Zstandard compression reduce raw corpora to ~33% of traditional database file sizes.
def _build_views(self) -> list[str]:
# Declarative virtual views mounted directly over zero-copy Parquet files
static_views = {
self.visual_analytics_dir / "va_comm_threads.parquet":
"CREATE OR REPLACE VIEW va_comm_threads AS SELECT * FROM read_parquet('{basePath}/visual_analytics/va_comm_threads.parquet');",
self.visual_analytics_dir / "va_cluster_hierarchy.parquet":
"CREATE OR REPLACE VIEW va_cluster_hierarchy AS SELECT * FROM read_parquet('{basePath}/visual_analytics/va_cluster_hierarchy.parquet');",
self.visual_analytics_dir / "va_document_registry.parquet":
"CREATE OR REPLACE VIEW va_document_registry AS SELECT * FROM read_parquet('{basePath}/visual_analytics/va_document_registry.parquet');",
}
for file_path, view_sql in static_views.items():
if file_path.exists():
views.append(view_sql)
return views
def to_arrow_fixed_size_list(embeddings: np.ndarray, dim: int = 256) -> pa.FixedSizeListArray:
"""Zero-copy contiguous C-buffer wrap into Arrow FixedSizeListArray without .tolist() heap unpacking."""
contiguous = np.ascontiguousarray(embeddings, dtype=np.float32)
flat_buffer = pa.py_buffer(contiguous.ravel())
value_array = pa.Array.from_buffers(pa.float32(), contiguous.size, [None, flat_buffer])
return pa.FixedSizeListArray.from_arrays(value_array, dim=dim)
EXPLAIN ANALYZE SELECT custodian, COUNT(*), SUM(size_bytes) FROM 'showcase_manifest.parquet' GROUP BY custodian ORDER BY COUNT(*) DESC;
30 Years of Untapped Silicon: Why Row Databases Foreclose SIMD
For nearly three decades, every personal workstation and server has shipped with supercomputer-grade vector processors sitting completely idle. Relational databases like MS SQL Server are mechanically foreclosed from using Single Instruction, Multiple Data (SIMD) because interleaved row tuples cause catastrophic CPU cache pollution. Columnar memory feeds vector registers what they were physically engineered to consume.
Legacy SQL Server: 99.8% Cache Pollution
Born in 1989, SQL Server was hardcoded around single-core, tuple-at-a-time (Volcano) loops years before MMX (1997), AltiVec (1999), or NEON (2004) brought vector registers to desktop silicon, as detailed in the MonetDB/X100 hyper-pipelining analysis.
- Severe Cache Pollution: Querying
WHERE Custodian = 'Smith'pulls 64-byte or 128-byte cache lines filled with unneeded email body text. As documented in Ulrich Drepper's What Every Programmer Should Know About Memory, over 99% of CPU bus bandwidth is squandered transferring data the query never asked for. - No Contiguous Vector Loading: Because matching fields are separated by thousands of bytes of text, the CPU cannot load a homogeneous vector register.
- Scalar Execution Stalls: The CPU falls back to scalar pointer-hopping: inspect one row, branch, jump over text, inspect next row. The processor pipeline constantly stalls waiting on DRAM latency.
Catalyst Columnar: 102.4 Billion Evals / Sec
By eliminating DBMS daemons and storing data in contiguous Apache Arrow / Parquet columns, Catalyst activates the dormant SIMD vector engines inside Apple Silicon.
- 100% Cache Line Density: All custodian IDs and timestamps are packed edge-to-edge. Every 64-byte L1 cache line pulled into the core contains 100% actionable query data. Zero wasted bus bandwidth.
- Direct NEON Vector Loading: Contiguous memory slides directly into 128-bit ARM NEON vector registers on Apple Silicon without conversion, unpacking, or pointer chasing.
- Vectorized Hardware Math: A single CPU instruction evaluates 8, 16, or 32 records simultaneously in hardware registers. Millions of filter comparisons execute in a fraction of a millisecond.
def compute_mps_auc_roc(pos_scores: torch.Tensor, neg_scores: torch.Tensor) -> float:
"""Compute AUC-ROC natively on the MPS device with zero CPU synchronization."""
scores = torch.cat([pos_scores, neg_scores])
labels = torch.cat([torch.ones_like(pos_scores), torch.zeros_like(neg_scores)])
# Vector sorting and cumulative trapezoidal summation run edge-to-edge on Apple Silicon
sorted_indices = torch.argsort(scores, descending=True)
sorted_labels = labels[sorted_indices]
tps = torch.cumsum(sorted_labels, dim=0)
fps = torch.cumsum(1.0 - sorted_labels, dim=0)
tpr = tps / pos_scores.size(0)
fpr = fps / neg_scores.size(0)
return float(torch.sum((fpr[1:] - fpr[:-1]) * tpr[:-1]).item())
pthread_set_qos_class_self_np(0x21, 0) forces the OS scheduler to dispatch workers to all 16 physical Performance Cores.
def elevate_process_qos() -> bool:
"""Elevate worker QoS from background throttling to QOS_CLASS_USER_INTERACTIVE (0x21)."""
if platform.system() == "Darwin":
libc = ctypes.CDLL("libSystem.B.dylib")
# QOS_CLASS_USER_INTERACTIVE = 0x21 pins execution to Apple Silicon Performance Cores
ret = libc.pthread_set_qos_class_self_np(0x21, 0)
os.setpriority(os.PRIO_PROCESS, os.getpid(), 0)
return ret == 0
return False
Modern data science (PyTorch, Hugging Face, Ray, Polars, Databricks) standardized on Apache Arrow because it eliminates serialization between Python, Rust, and C++. Extracting text, 256D/1536D vectors, and graph topologies turns raw data into dense columnar Arrow arrays where models and analytics share memory at zero copy cost.
Catalyst outputs pristine, strictly typed Apache Parquet tables with zero proprietary lock-in. Datasets are directly mountable in Snowflake, Databricks, DuckDB, AWS Athena, or native Swift desktop clients without conversion, re-indexing, or migration pipelines.
Big data processing no longer requires cloud server farms. Catalyst runs the entire multi-stage ETL pipeline locally on prosumer workstations (Apple Silicon M1/M2/M3/M4 Ultra/Max). 100% offline, zero cloud egress, immune to network failure or third-party subpoenas, and fully defensible under attorney-client privilege.
Showcase Execution Run Summary & Lakehouse Manifest
Audit metrics from processing the showcase legal corpus on prosumer workstation hardware. Pre-ingestion collection payload versus post-processing columnar lakehouse storage.
Pre-Ingestion Source Corpus
Tokenization & Intelligence
Post-Processing Storage
public struct LakehouseManifest: Codable {
public let createdAt: String
public let manifestVersion: String
public let namespace: String
public let pipelineStages: [PipelineStage]?
public let servingEngine: String? // e.g. 'namespace_index.duckdb'
public let status: String // 'COMPLETE'
public let tables: [String: LogicalTable]? // Physical Parquet partitions
public let telemetry: [String: Telemetry]?
}