Skip to content

fix(ww3d2): Prevent undefined behavior when writing dynamic vertices and reading screenshot pixels - #3671

Draft
CryoTheRenegade wants to merge 1 commit into
TheSuperHackers:mainfrom
CryoTheRenegade:refactor/vertex-buffer-casts
Draft

CryoTheRenegade wants to merge 1 commit into
TheSuperHackers:mainfrom
CryoTheRenegade:refactor/vertex-buffer-casts

Conversation

@CryoTheRenegade

@CryoTheRenegade CryoTheRenegade commented Nov 10, 2026 •

Copy link
Copy Markdown

Follow-up to #3670, from xezon's "Are there more of these?" on #3580. This collects the last pointer casts of this kind in the renderers and the screenshot writer into one PR.

  • SegLineRendererClass::Render and StreakRendererClass::RenderStreak cast the locked dynamic vertex buffer to unsigned char*, then write each vertex through Vector3*, unsigned int* and Vector2* casts at offsets taken from FVFInfoClass. The dynamic vertex buffer always has the dynamic_fvf_type format, and Get_Formatted_Vertex_Array already returns it as VertexFormatXYZNDUV2* and asserts that format. The loops now write the fields of that struct, as dazzle.cpp and dynamesh.cpp already do:

    -				Vector3 *vertex = reinterpret_cast<Vector3 *>(vb + verticesOffset);
    -				vertex->X = vArray[i].x;
    ...
    -				*reinterpret_cast<unsigned int *>(vb + diffuseOffset) = vArray[i].diffuse;
    ...
    -				vb += vbSize;
    +				vb[i].x = vArray[i].x;
    ...
    +				vb[i].diffuse = vArray[i].diffuse;
    +				vb[i].u1 = vArray[i].u1;
    +				vb[i].v1 = vArray[i].v1;
  • screenshotThreadFunc reads the copied surface, a new unsigned char[] array, through const unsigned int* and const unsigned short* casts. Reading unsigned char objects as unsigned int is undefined behavior. Each pixel is now copied out with memcpy.

The reinterpret_cast<long*> in dx8webbrowser.cpp stays. It hands the device pointer to the BrowserEngine COM interface, whose Initialize is declared with a long* parameter, and nothing in this code reads through it.

Evidence

  • Disassembly: I compared seglinerenderer.obj, streakRender.obj and W3DScreenshot.obj for both games against a build of main. In every changed function, the float arithmetic and the call targets are the same as on main.
    • VS 2026 (ClangCL) Release: in Render and RenderStreak, the vertex stores use fixed struct offsets instead of the FVFInfoClass offsets. In screenshotThreadFunc, the pixel loops load the color bytes directly, so they use fewer shifts and masks.
    • VC6 Release: Render and RenderStreak lose the offset locals, so their stack frames shrink by 4 bytes. That changes the x87 scheduling of the transform math before the vertex loop: the same multiplications and additions, in a different register order, with one operand kept on the x87 stack instead of reloaded from memory. This is render-only code, so it does not affect game logic. screenshotThreadFunc is unchanged, because the 4- and 2-byte memcpy compile to the same loads.

AI Disclosure

Claude Opus 5.5 was used to aid in implementation and to compare the disassembly.

… pointer casts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Screenshot conversion now reads 32-bit and 16-bit pixels with memcpy. Two rendering paths now assign vertex position, color, and UV data through named fields in VertexFormatXYZNDUV2.

Changes

Pixel and Vertex Data Access

Layer / File(s) Summary
Screenshot pixel reads
Core/GameEngineDevice/Source/W3DDevice/GameClient/W3DScreenshot.cpp
The 32-bit and 16-bit conversion loops read pixel values from byte-addressed rows with memcpy. RGB extraction remains unchanged.
Typed vertex assignments
Core/Libraries/Source/WWVegas/WW3D2/seglinerenderer.cpp, Core/Libraries/Source/WWVegas/WW3D2/streakRender.cpp
Both rendering paths assign vertex position, diffuse color, and UV values through VertexFormatXYZNDUV2 fields instead of FVF-offset-based byte writes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested labels: Rendering, Fix, Stability, Minor, Platform

Suggested reviewers: bobtista, xezon

Merge Risk: Minimal · up to ddcf7

No actionable merge-blocking risk was identified in the pixel and vertex access changes.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly summarizes the changes to dynamic vertex writes and screenshot pixel reads.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description explains the changes to dynamic vertex writes and screenshot pixel reads, and provides relevant implementation and testing context.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 124b4b5e-fd35-440d-a2e4-b7d31594ceef
Commits

Reviewing files that changed from the base of the PR and between 998f337 and ddcf743.

Files selected for processing: 3

  • Core/GameEngineDevice/Source/W3DDevice/GameClient/W3DScreenshot.cpp
  • Core/Libraries/Source/WWVegas/WW3D2/seglinerenderer.cpp
  • Core/Libraries/Source/WWVegas/WW3D2/streakRender.cpp

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant