TenzorPipe — zero-VRAM media ingestion engine

See it side by side

Narrated, about three minutes. It opens with the workload TenzorPipe loses — GPU decoding wins a single first pass — and every figure on screen comes from a saved measurement record.

Source footage excerpts from Sintel, © Blender Foundation, used under CC BY 3.0. Robotics and field scenes are labelled AI-generated concept art, not real deployments.

One decode, then pure memory reads

Conventional loaders re-decode every clip on every epoch. TenzorPipe decodes once into a contiguous Arrow artifact; each epoch after that is a memory map away.

Measured, not marketed

One 20-second 1080p30 H.264 clip at 8 Mb/s, Intel i5-14400F + RTX 5060 (8 GB) under WSL2, 50 measured iterations per contender. Full method, raw numbers and caveats: BENCHMARKS.md.

Metric	TenzorPipe 0.3.0	FFmpeg pipe	NVIDIA DALI	TorchCodec (CUDA)
First-pass ingest	646 ms	554 ms	442 ms	276 ms
Every later training epoch	2.8 ms Arrow re-read	decode again	decode again	decode again
GPU memory during decode	0 MiB	0 MiB	822 MiB	650 MiB
Output reproducibility	1,056 / 1,056 byte-identical	differ from each other by colour conversion and resampling

TenzorPipe is not the fastest first pass — GPU decoders are. It wins from the second epoch onward, and it never takes VRAM away from your model.

Re-reads: why the video says 25 ms and this table says 2.8 ms

They measure different operations on different storage. The showcase figure is a full reopen + read cycle in a fresh process, including repeated mapping overhead on Windows-backed WSL storage. The table’s figure is the Arrow → torch load of an already-prepared file from /dev/shm, a RAM disk, which is what a training loop repeats on every epoch after the first.

Neither is a cold-storage measurement. The RAM-disk figure excludes disk I/O entirely, and the showcase figure is explicitly not a cold-SSD benchmark. Cold NVMe re-reads remain unmeasured, so treat both as bounds rather than as what your own hardware will do.

Target architectures & concrete use cases

Where a zero-VRAM, bit-exact, decode-once pipeline changes the engineering calculus.

Edge robotics & autonomous perception

Jetson · drones · humanoids

Eliminates VRAM decode contention on shared and unified-memory architectures, where every decoder byte is a byte the model cannot have.
Streams contiguous Apache Arrow frames straight into spatial neural networks with 0 MiB GPU overhead, conserving battery and compute bandwidth.

Multimodal LLM & VLM training

video-language models · multi-GPU clusters

Ingests training video once into the structured .tenzor format, with video and Log-Mel audio locked to the same 0.5 s epoch grid.
Re-reads tensors at sub-3 ms multi-epoch speed, keeping clusters saturated instead of repeatedly decompressing the same MP4s.

Air-gapped & sovereign pipelines

defense · medical · regulated data

Full local execution: zero cloud APIs, zero network calls, and no temporary frame dumps to disk — a libc-level write audit observed only the final artifact.
Deterministic, bit-exact ingestion suited to HIPAA and air-gapped environments, verified across 1,056 regression cases.

High-throughput local indexing & search

journaling · media ops · embeddings

Removes the disk thrashing and process-spawn latency of shelling out to FFmpeg across thousands of local clips.
Pipes contiguous frame batches straight into local vector embedding models — 16 clips in 0.41 s versus 1.95 s one at a time.

Three lines

# 1. install (Linux x86-64, CPython 3.9+)
pip install tenzorpipe

# 2. decode once into Apache Arrow
python -c "import tenzorpipe as tp; tp.ingest('clip.mp4', 'clip.tenzor')"

# 3. memory-map tensors for every epoch
python -c "import tenzorpipe as tp; d = tp.load('clip.tenzor'); print(len(d), d[0]['video'].shape)"

Tensor contract

224×224 RGB CHW float32 in [-1, 1] and 64-band Log-Mel, one row per 0.5 s epoch.

No runtime deps

OpenH264 and the AAC decoder are compiled in. No FFmpeg, no CUDA, no system codecs.

Bit-exact

Same bytes on every worker count, verified against the previous release on 1,056 cases.

Commercial & enterprise licensing

TenzorPipe is released under the Business Source License 1.1. It converts to Apache-2.0 on 16 September 2030.

Free for production

Individuals and entities under US$100,000 annual gross revenue
Revenue counted across parents, subsidiaries and affiliates under common control
Non-production use — testing, evaluation, academic research — always unrestricted

Commercial license required

Organizations at or above US$100,000 annual gross revenue
Embedded, bundled or OEM distribution on hardware devices
Terms negotiated directly with the licensor

licensing@tenzorpipe.org