Project: That Rendering Thing

[sjh26]

Draft Status

Ready to go

Category

VR/AR and Rendering

Key Investigators

• Steve Pieper (Isomics, USA)

Project Description

I've been experimenting with wgpu-py, an implementation of WebGPU that you can pip_install directly into Slicer.

The work in progress in SlicerWGPU, which is now working pretty nicely in demo mode.

Key Features

• Easy to install, can work with any recent Slicer release. Don't even need to restart Slicer.
• Can write shader code easily that works on Vulkan, DirectX, and Metal.
• It supports multi-volume rendering with full quality compositing
• It can directly render segmentations and provides instant updates in response to edits
• It supports interactive rendering of volumes under non-linear transforms
• Interactive elements like transform handles and markups are still handled by VTK and seamlessly composited

Current status

• Currently in demo mode. Some things are integrated to work with the MRML scene while, others work alongside.
• The VTK OpenGL render window is tapped into so that most rendering works cleanly, but SlicerWGPU uses a different method to composite multiple translucent surfaces (A-Buffer compared to Depth Peeling) so those aren't currently combined.
• Performance seems about the same as VTK's GPU renderer, but no extensive testing has been done.

How to try it

• Clone the SlicerWGPU repository
• Drag and drop the folder onto a running Slicer (5.10.0 or really any version should work).
• Be sure to load the Scene Rendering module (you can ignore the others)
• Go to the Scene Rendering module and try the buttons. The top block of buttons (the "VTK Injection" block) are the current demos and the "Dual View" bock are some older experiments.

Objective

1. Give demos to anyone who might be interested and collect feedback
2. Collect use-cases where existing Slicer rendering doesn't fully meet requirements but the SlicerWGPU code could
3. Consider what would be needed for a more complete integration (i.e. possible custom display nodes and dedicated GUI).

Approach and Plan

1. Experiment with an updated Volume Rendering module to take advantage of advanced features
2. Try implementing any new features or use cases people suggest
3. Think about how to render volumes that are too large for GPU memory (i.e. paging texture data)
4. Consider how to make this more easily available and useful

Progress and Next Steps

(TBD)

Illustrations

Background and References

SlilcerCL: https://github.com/pieper/SlicerCL (https://www.youtube.com/watch?v=hFxTyLPjQd0)
STEP: https://github.com/pieper/step (https://www.youtube.com/watch?v=ML9_JWAz1kY, https://youtu.be/8dputUoKBTA)
Transfer function editor: https://youtu.be/4Q9NfNmN-9w

[sjh26]

@pieper