LibRaw RPi Decoder Heap Overflow: A 6-Year Bug Found by AI

Background

LibRaw is the de facto open-source library for reading RAW files from digital cameras. It powers darktable, RawTherapee, digiKam, and countless imaging pipelines across research, professional, and hobbyist workflows.

The USE_6BY9RPI build flag enables Raspberry Pi RAW+JPEG (BRCM format) support. This decoder code was integrated as third-party "borrowed" code—maintained externally and contributed to libraw without the same security review as native decoders. For approximately six years, this code existed in the repository untouched.

The maintainer noted that "8- and 16-bit versions of the files were simply not found on the open internet" — meaning fuzzers like oss-fuzz and libFuzzer had no corpus to reach this code path. Continuous fuzzing cannot test what it cannot discover. This is the precise class of bug that survives longest: reachable through the documented API, invisible to fuzzers, and too subtle for pattern-based scanners. A stale loop variable reused as a write offset requires understanding of macro scope and control flow across the entire function—not a signature.

How the AI Found It

Phase 1 — Ingesting the decoder

XOR-1 ingested the LibRaw source tree and built a model of the file-processing pipeline: how BRCM files are parsed in parse_raspberrypi() (src/metadata/misc_parsers.cpp), how raw_stride is computed from attacker-controlled fields in the BRCM header, and how that value flows into rpi_load_raw8() and rpi_load_raw16() as dwide. It identified USE_6BY9RPI as a non-default code path and prioritized it precisely because it was "borrowed" code with no evidence of independent security audit in the commit history.

Phase 2 — The FORC macro and the stale variable

XOR-1 read the FORC macro definition in internal/defines.h:

#define FORC(cnt) for (c = 0; c < cnt; c++)

It recognized that after FORC(dwide) completes, c equals dwide—not zero. It then scanned every post-FORC use of c within the same function scope and found the critical line: RAW(row, col + c) = dp[c]. The RAW macro expands to raw_image[row * raw_width + col], so the actual write destination is raw_image[row * raw_width + col + dwide]—past the end of the allocated row, for every pixel, every row. The bug is not in the FORC loop itself. It is in the silent reuse of a spent loop variable on the very next line.

Phase 3 — Tracing attacker control and confirming the overflow

XOR-1 traced dwide back to its source: raw_stride, computed in parse_raspberrypi() as:

raw_stride = ((header.h_width + header.padding_right) + 31) & (~31);

Both h_width and padding_right are uint16_t fields read directly from the BRCM file header with no validation. An attacker sets padding_right. XOR-1 then verified the allocation formula at unpack.cpp:395 — malloc(raw_width * (raw_height + 8) * sizeof(ushort)) — and confirmed that when dwide exceeds 8 * raw_width + 1, writes escape the +8 row guard entirely. A minimal crafted BRCM file was generated and run under AddressSanitizer. Overflow confirmed on the first run.

XOR-1 Session Log

The Vulnerable Code

After the FORC(dwide) macro completes, c is equal to dwide, not zero. Both rpi_load_raw8() and rpi_load_raw16() reuse c directly as an offset into raw_image, causing every write to overflow past the end of each row.

rpi_load_raw8() (lines 302–303, BEFORE FIX)

/* After FORC(dwide), c == dwide — not reset to zero */
for (dp = data, col = 0; col < raw_width; dp++, col++)
    RAW(row, col + c) = dp[c];   /* writes to raw_image[row * raw_width + col + dwide] */

rpi_load_raw16() (lines 399–400, BEFORE FIX)

/* Same defect — c is dwide after FORC(dwide) */
for (dp = data, col = 0; col < raw_width; dp += 2, col++)
    RAW(row, col + c) = (dp[1] << 8) | dp[0];

The Fix

The LibRaw maintainer reviewed Vorthix's analysis, confirmed the vulnerability, and merged Vorthix's suggested fix within 24 hours of disclosure.

rpi_load_raw8() (FIXED)

- for (dp = data, col = 0; col < raw_width; dp++, col++)
-     RAW(row, col + c) = dp[c];
+ for (dp = data, col = 0; col < raw_width && col < dwide; dp++, col++)
+     RAW(row, col) = *dp;

rpi_load_raw16() (FIXED)

- for (dp = data, col = 0; col < raw_width; dp += 2, col++)
-     RAW(row, col + c) = (dp[1] << 8) | dp[0];
+ for (dp = data, col = 0; col < raw_width && col < dwide/2; dp += 2, col++)
+     RAW(row, col) = (dp[1] << 8) | dp[0];

The variable c is removed entirely from both the index and the pointer offset. An additional col < dwide bounds guard is added for extra safety on both code paths.

AddressSanitizer Output

Timeline

"A fix is a claim. Every claim has an assumption. Find the assumption."

About This Research

This vulnerability was discovered autonomously by XOR-1, Vorthix's internal research agent. The finding was validated by the Vorthix security team before disclosure. About Vorthix →

Reported by Haris Hussain, Founder & CEO of Vorthix (@hextheshadow on GitHub, LinkedIn).

Coordinated disclosure completed Jul 14, 2026. See commit 87241b4 for the fix.