Development of a 3D Visualization Testing Framework Using GPU-Agnostic Image Comparison

This project implements a testing framework for 3D rendering pipelines that evaluates visual correctness independently of the underlying GPU hardware. The framework renders identical scenes on multiple adapters using Direct3D 12 and compares the outputs with an enhanced version of FLIP — a perceptual image difference metric developed by NVIDIA Research.

The core research question is whether a perceptual metric can serve as a hardware-agnostic acceptance criterion: declaring two renders equivalent when they are indistinguishable to a human observer, even if the raw pixel values differ due to vendor-specific floating-point behaviour. The framework includes six purpose-built Direct3D 12 scenes that cover the rasterizer baseline, compute-heavy BRDF evaluation, Monte Carlo path tracing, and an HDR stress case that deliberately exposes a known FLIP limitation — along with a fix via ACES tone-mapping pre-processing.

Five algorithmic enhancements over the baseline NVIDIA FLIP metric are implemented in flip_tool_2:

• ACES tone-mapping pre-processing (--tonemap) — converts HDR linear values into the SDR [0, 1] range before comparison, so that perceptually-identical HDR frames that differ only in raw luminance scale are correctly reported as MATCH.
• Weber–Fechner perceptual masking — automatically suppresses FLIP errors in high-frequency and edge-heavy regions where the human visual system is less sensitive, using local Weber contrast derived from a Sobel gradient. No manual tuning required.
• Median + MAD statistical outlier detection — a zero-configuration adaptive per-pixel anomaly threshold that calibrates to each image pair's noise floor.
• Four configurable decision modes (And, Or, Similarity, Anomaly) that combine the global similarity score with the anomaly percentage.
• CIEDE2000 (ΔE₀₀) color difference replacing HyAB, correctly modelling the blue–purple hue-rotation effect that the baseline metric ignores.

---

Download

A pre-built standalone flip_tool_2.exe for Windows x64 is published on the Releases page. No Visual Studio or Visual C++ Redistributable required — unzip and run.

flip_tool_2.exe --help

If you also need the renderer executables, build from source as described below.

---

Requirements

• Windows 10/11
• Visual Studio 2022 (MSVC, with Desktop C++ workload)
• CMake 3.16+
• Windows SDK 10.0.22621+
• DirectX 12-capable GPU(s)
• Python 3.10+ (for automation scripts in scripts/)

---

Project Structure

thesis/
├── renderers/                  # Six Direct3D 12 scenes
│   ├── shared/                 # D3D12AppBase, D3D12Helpers (shared by all renderers)
│   ├── HelloTriangle/          # Rasterizer baseline                  → PNG
│   ├── GrazingSpec/            # Grazing-angle specular / aliasing    → PNG
│   ├── PathTracer/             # Cornell Box path tracer (8 SPP)      → PNG
│   ├── Supernova/              # HDR luminance stress scene           → EXR
│   ├── GrassField/             # 500k procedural sub-pixel blades     → PNG
│   └── ToxicShadows/           # SDF raymarch w/ trigonometric drift  → PNG
├── flip_image/                 # Shared image I/O (PNG / EXR / HDR)
├── flip_tool/                  # FLIP v1 CLI  (wraps NVlabs/flip)
├── flip_tool_2/                # FLIP v2 CLI  (custom CPU reimplementation)
├── third_party/
│   └── tinyexr/                # Single-header OpenEXR library
├── scripts/
│   ├── build.py                # Build all targets via CMake + vswhere
│   ├── bench_flip.py           # FLIP execution time benchmarking
│   ├── render_gpu.py           # Render all scenes on a chosen GPU
│   ├── test_all.py             # Render on every GPU + run FLIP comparisons
│   ├── compare_pairs.py        # Pairwise FLIP v1/v2 over existing renders
│   └── clear_results.py        # Delete all files from results/
├── results/                    # Output images and FLIP diff maps (git-ignored)
└── CMakeLists.txt

---

Build

CLion

1. Clone the repository and open the root folder as a CMake project.
2. Select the Release configuration.
3. Build → Build All (Ctrl+F9).

Binaries are placed in cmake-build-release/bin/.

Python script

python scripts/build.py                    # Release build (default)
python scripts/build.py --config Debug     # Debug build
python scripts/build.py --jobs 4           # Limit parallel jobs
python scripts/build.py --targets GrazingSpec flip_tool_2   # Specific targets

build.py auto-detects the Visual Studio installation via vswhere, runs cmake configure if the build cache is absent, and builds each target with coloured progress output.

Command line (manual)

Run from a plain cmd.exe so vcvars initialises correctly. Locate vcvars64.bat for your Visual Studio installation, then:

"C:\Program Files\Microsoft Visual Studio\<version>\<edition>\VC\Auxiliary\Build\vcvars64.bat"
cmake --build cmake-build-release --config Release

Common paths:

• VS 2022 Community — ...\2022\Community\...
• VS 2022 Professional — ...\2022\Professional\...
• VS 2019 Community — ...\2019\Community\...

---

Usage

Performance Benchmark: bench_flip.py

Measures the wall-clock execution time of the comparison tools across multiple scenes, resolutions, and decision modes. The script orchestrates a two-phase pipeline: first rendering the required reference and test frames, and then repeatedly executing the selected FLIP tool to gather statistically robust timing data (mean, min, and max execution times).

python scripts/bench_flip.py

Flag	Default	Description
--flip-version <1&#124;2>	2	FLIP tool version to benchmark
--gpu <N>	interactive	GPU adapter index for the rendering phase
--config <cfg>	Release	Build configuration
--reps <N>	5	Number of repetitions per test cell
--skip-render	off	Skip Phase 1 and reuse existing images in results/bench/
--out-csv <path>	results/bench_results_v{N}.csv	Output path for the benchmark CSV

Automated: test_all.py

Renders all six scenes on every GPU, then runs FLIP on each pair. Must be run from the project root.

python scripts/test_all.py

Parameters:

Flag	Default	Description
--adapters N [N ...]	all GPUs	Adapter indices to use (e.g. --adapters 0 1)
--config <cfg>	Release	Build config: Release, Debug, RelWithDebInfo
--width <px>	1920	Render width
--height <px>	1080	Render height
--flip-version <1&#124;2>	2	Which FLIP tool to use for comparisons
--out-dir <path>	results/	Output directory
--skip-render	off	Skip rendering, only re-run FLIP on existing images

Examples:

# FLIP v1 instead of v2
python scripts/test_all.py --flip-version 1

# 4K resolution
python scripts/test_all.py --width 2560 --height 1440

# Two specific adapters
python scripts/test_all.py --adapters 0 1

# Skip rendering, re-compare existing images
python scripts/test_all.py --skip-render

# Debug build
python scripts/test_all.py --config Debug

Outputs are saved to results/:

• {scene}_{GPUName}.png / .exr — rendered frames
• {scene}_diff_{GPU0}_vs_{GPU1}.png — FLIP error heatmaps

Single GPU: render_gpu.py

Renders all (or selected) scenes on one GPU.

python scripts/render_gpu.py               # interactive GPU picker
python scripts/render_gpu.py --gpu 0       # GPU index 0
python scripts/render_gpu.py --gpu 1 --width 2560 --height 1440
python scripts/render_gpu.py --gpu 0 --scenes grazingspec supernova

Parameters:

Flag	Default	Description
--gpu <N>	interactive	GPU adapter index
--config <cfg>	Release	Build configuration
--width <px>	1920	Render width
--height <px>	1080	Render height
--out-dir <path>	results/	Output directory
--scenes NAME ...	all	Filter: hellotriangle grazingspec pathtracer supernova grassfield toxicshadows

Pairwise comparison: compare_pairs.py

Auto-discovers all rendered images already present in results/, builds every ordered GPU pair, and runs both flip_tool and flip_tool_2 (in all four decision modes) on each pair. Writes a per-pair diff image plus an aggregate CSV summary.

python scripts/compare_pairs.py
python scripts/compare_pairs.py --config Release
python scripts/compare_pairs.py --out-dir path/to/results

Clean results: clear_results.py

python scripts/clear_results.py

Deletes all files inside results/ (preserving .gitkeep / .gitignore).

Manual: individual executables

Each renderer accepts the same flags:

--adapter N          GPU index (default: 0)
--output  <path>     Output file path
--width   <px>       Render width  (default: screen width)
--height  <px>       Render height (default: screen height)
--list-adapters      List available GPUs and exit
--help

HelloTriangle.exe --adapter 0 --output results\ht_gpu0.png
GrazingSpec.exe   --adapter 1 --output results\gs_gpu1.png
PathTracer.exe    --adapter 0 --output results\pt_gpu0.png --width 1920 --height 1080
Supernova.exe     --adapter 0 --output results\sn_gpu0.exr
GrassField.exe    --adapter 0 --output results\gf_gpu0.png
ToxicShadows.exe  --adapter 0 --output results\ts_gpu0.png

CLion Run Configurations

To run test_all.py from CLion:

Field	Value
Executable	python.exe (or full path to interpreter)
Program arguments	scripts/test_all.py --flip-version 2
Working directory	$ProjectFileDir$

Working directory must be set to $ProjectFileDir$ (project root). Without it, relative paths such as results/ will not resolve correctly.

To run a single renderer:

Field	Value
Executable	cmake-build-release\bin\Supernova.exe
Program arguments	--adapter 0 --output "$ProjectFileDir$\results\supernova_gpu0.exr"
Working directory	$ProjectFileDir$

---

FLIP Tools

Both tools compare two images and print a similarity score. Exit code: 0 = MATCH, 1 = DIFFERENT.

flip_tool (v1)

Thin wrapper around the NVlabs/flip reference library.

flip_tool.exe --test <path> --reference <path> [options]

  --threshold <0–1>        Similarity threshold for MATCH (default: 0.95)
  --output-diff <path>     Save FLIP error heatmap PNG
  --print-raw              Print mean error value in [0,1]
  --hdr                    Force HDR evaluation
  --ldr                    Force LDR evaluation
  --ppd <float>            Pixels-per-degree override
  --monitor-distance <m>   Viewer distance in metres (default: 0.7)
  --monitor-width    <m>   Display width in metres (default: 0.7)
  --monitor-res-w  <px>    Display horizontal resolution (default: 3840)
  --help

flip_tool_2 (v2)

Custom CPU-only reimplementation. Identical interface, plus advanced decision criteria and tone-mapping flags:

flip_tool_2.exe --test <path> --reference <path> [options]

  --threshold <0–1>        Similarity threshold for MATCH (default: 0.95)
  --output-diff <path>     Save FLIP error heatmap PNG
  --print-raw              Print mean error value in [0,1]
  --hdr                    Force HDR evaluation
  --ldr                    Force LDR evaluation
  --ppd <float>            Pixels-per-degree override
  --monitor-distance <m>   Viewer distance in metres (default: 0.7)
  --monitor-width    <m>   Display width in metres (default: 0.7)
  --monitor-res-w  <px>    Display horizontal resolution (default: 3840)
  --tonemap                Apply ACES filmic tone-mapping before FLIP
  --decision-mode <mode>   Logic for MATCH: AND (default), OR, SIMILARITY, ANOMALY
  --max-anomalies <%>      Max allowed statistical anomalies in % (default: 3.0)
  --help

--tonemap and the HDR Clipping problem

FLIP in HDR mode compares absolute luminance values. The Supernova scene deliberately generates floor radiance of [200, 10 000] nit via a vendor-specific FP hash. Both values clip to white on any SDR display (visually identical), but FLIP sees a 50× luminance difference and reports DIFFERENT.

The --tonemap flag applies ACES filmic tone-mapping (Narkowicz 2015) to both images before running FLIP, mapping HDR values into [0, 1] as they would appear on an SDR screen.

flip_tool_2.exe --reference results\supernova_gpu0.exr --test results\supernova_gpu1.exr
# Result: DIFFERENT (~19% mean error)

flip_tool_2.exe --reference results\supernova_gpu0.exr --test results\supernova_gpu1.exr --tonemap
# Result: MATCH

--tonemap is silently disabled on LDR images (all pixels ≤ 1.0).

---

Improvements in flip_tool_2 over flip_tool (v1)

1. Perceptual masking via Weber–Fechner contrast

Standard FLIP weighs every pixel equally, which causes false DIFFERENT verdicts in scenes with stochastic noise or high-frequency detail — regions where the human visual system is inherently less sensitive to small errors.

flip_tool_2 adds a Weber–Fechner contrast masking post-process applied directly to the raw FLIP error map before the mean is computed:

1. Luminance pre-pass — the reference image is converted to greyscale once and stored in a flat float cache, avoiding redundant conversions.

2. Sobel gradient — a 3×3 Sobel operator computes the local edge magnitude sobelMag at each pixel.

3. Weber contrast — the gradient is normalised by local luminance:

   $$W_contrast(x,y) = sobelMag(x,y) / (L(x,y) + 0.05)$$

   The epsilon 0.05 prevents division by zero in black regions and avoids over-amplifying noise in deep shadows (where the eye is also insensitive).

4. Weight modulation — the FLIP error at each pixel is suppressed proportionally:

   $$Error'(x,y) = Error(x,y) * (1 - min(W_contrast, 0.85))$$

The masking is fully automatic and scene-adaptive: bright outdoor scenes with strong edges produce naturally higher Weber contrast values and receive more suppression; dark indoor scenes with subtle gradients are treated more strictly. No manual parameter tuning is required.

2. ACES HDR tone-mapping (--tonemap)

Described in the section above. Converts HDR linear values to SDR [0, 1] before FLIP, resolving the HDR clipping false-positive that exists in v1.

3. Statistical Anomaly Detection

Traditional automated testing relies on a hardcoded similarity threshold (e.g., 95%). This approach is brittle for stochastic workloads like Monte Carlo path tracing, where inherent variance (noise) lowers the global similarity score, causing false DIFFERENT verdicts even when the render is technically correct.

flip_tool_2 introduces zero-configuration statistical outlier detection based on the Median + MAD (Median Absolute Deviation) robust estimator:

1. Computes the median and MAD of the error map: sigma_MAD = 1.4826 * MAD
2. Derives a dynamic anomaly threshold: Threshold = max(0.05, median + 3 * sigma_MAD)
3. Any pixel exceeding this threshold is flagged as a statistical anomaly.

Using the median instead of the mean makes the threshold robust to outliers — smooth scenes receive a strict threshold, noisy scenes a relaxed one, automatically.

4. Multi-Criteria Decision Pipeline

flip_tool_2 replaces rigid pass/fail logic with a flexible decision matrix via --decision-mode:

• AND (Default): Must pass both similarity threshold and anomaly limit. Strictest mode — recommended for deterministic rasterization and CI regression gates.
• OR: Passes if globally similar or anomalies within margin. Good for hybrid test suites.
• SIMILARITY: Legacy mode — evaluates only the classic FLIP similarity percentage.
• ANOMALY: Smart mode — fails only if the percentage of statistical outliers exceeds the limit.

5. CIEDE2000 Color Difference (ΔE00)

flip_tool_2 replaces the original HyAB + Hunt color difference with the industry-standard CIEDE2000 (ΔE00) formula (CIE 142-2001), implemented in pure C++17. ΔE00 correctly accounts for perceptual non-uniformity across the Lightness, Chroma, and Hue axes, including the blue–purple hue-rotation effect that HyAB ignores. This matters in practice: the ToxicShadows scene exposes floating-point divergence between GPU vendors — on NVIDIA the shadow is a physically correct soft gray, on Intel it becomes a vivid green gradient caused by vendor-specific FP32 rounding in sin(). HyAB reported 96.5 % similarity and called it MATCH; CIEDE2000 correctly measured the chromatic shift and returned 88.2 % so the AND default mode flags it as DIFFERENT, catching the hardware bug that v1 silently passed.

Metric	flip_tool v1 (HyAB)	flip_tool_2 (CIEDE2000)
Similarity	96.5 %	88.2 %
Mean error	0.0352	0.1179
Result	MATCH ❌	DIFFERENT ✓

---