Repository navigation
fix(ww3d2): Prevent undefined behavior when writing dynamic vertices and reading screenshot pixels - #3671
Conversation
… pointer casts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. WalkthroughScreenshot conversion now reads 32-bit and 16-bit pixels with ChangesPixel and Vertex Data Access
Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix Suggested labels: Suggested reviewers: Merge Risk:
|
Follow-up to #3670, from xezon's "Are there more of these?" on #3580. This collects the last pointer casts of this kind in the renderers and the screenshot writer into one PR.
SegLineRendererClass::RenderandStreakRendererClass::RenderStreakcast the locked dynamic vertex buffer tounsigned char*, then write each vertex throughVector3*,unsigned int*andVector2*casts at offsets taken fromFVFInfoClass. The dynamic vertex buffer always has thedynamic_fvf_typeformat, andGet_Formatted_Vertex_Arrayalready returns it asVertexFormatXYZNDUV2*and asserts that format. The loops now write the fields of that struct, asdazzle.cppanddynamesh.cppalready do:screenshotThreadFuncreads the copied surface, anew unsigned char[]array, throughconst unsigned int*andconst unsigned short*casts. Readingunsigned charobjects asunsigned intis undefined behavior. Each pixel is now copied out withmemcpy.The
reinterpret_cast<long*>indx8webbrowser.cppstays. It hands the device pointer to the BrowserEngine COM interface, whoseInitializeis declared with along*parameter, and nothing in this code reads through it.Evidence
seglinerenderer.obj,streakRender.objandW3DScreenshot.objfor both games against a build ofmain. In every changed function, the float arithmetic and the call targets are the same as onmain.RenderandRenderStreak, the vertex stores use fixed struct offsets instead of theFVFInfoClassoffsets. InscreenshotThreadFunc, the pixel loops load the color bytes directly, so they use fewer shifts and masks.RenderandRenderStreaklose the offset locals, so their stack frames shrink by 4 bytes. That changes the x87 scheduling of the transform math before the vertex loop: the same multiplications and additions, in a different register order, with one operand kept on the x87 stack instead of reloaded from memory. This is render-only code, so it does not affect game logic.screenshotThreadFuncis unchanged, because the 4- and 2-bytememcpycompile to the same loads.AI Disclosure
Claude Opus 5.5 was used to aid in implementation and to compare the disassembly.