add first prototype of texElementImage2D

[cabanier]

cc @kenrussell

[kenrussell]

One initial comment. Likely this PR will remain unmerged for a while until there are prototype implementations of this new API. In the meantime, progress can be made on the API's semantics, addressing the TBD mentioned here, etc.

[foolip]

cc @progers @schenney-chromium @szager @chrishtr

[foolip]

The answer to WICG/html-in-canvas#68 in this PR seems to be that we do keep level, and that's what we have in Chromium's implementation. Just flagging the point of discussion in case someone feels differently.

[foolip]

@cabanier what's still needed to make this ready for review? whatwg/html#11588 is making steady progress and I want to make sure the WebGL side is in sync.

Importantly, the HTML spec now has a concept of an "element image snapshot" which is what must be used in canvas APIs like this one. Can you update the spec text to use that concept? You can use a direct link to https://whatpr.org/html/11588/canvas.html#concept-canvas-element-image-snapshots until the HTML spec PR is merged.

[jakearchibald]

This seems pretty limiting compared to the 2d equivalent. Could this take optional width & height args? This seems especially important for high-dpr.

It also seems to be desirable to use a portion of an element in a texture, eg for extreme zoomed cases. This is also something you can do with the 2d API.

[szager-chromium]

I think the IDL changes in this PR are a bit behind; the current chromium implementation has four overrides for texElementImage2D which allow for specification of both source rect (for painting a section of an element, or for extending the painted area to include ink overflow) and also destination size (i.e. texture size).

@foolip can you update the PR to match the chromium implementation?

[cabanier]

The width/height was there originally but we were told that those APIs were going to be removed in the future. Has that changed?
What is the difference between passing width/height and setting it on the element?

[jakearchibald]

Setting it on the element means scaling everything else up within it, which is a pain. Trying to deal with the sub-rect case is even more painful.

[jakearchibald]

@szager-chromium that's good to hear. Although, it's a little frustrating that for some things, the explainer is more up to date, and for other things the PRs are more up to date, and for other things, like this, neither has the details.

[jakearchibald]

Where's the up to date proposal for the WebGPU API addition?

[szager-chromium]

I'm not sure there is one. I found an issue tracking this, but I'm not aware of any completed spec work. As with WebGL, I think the IDL in the chromium repository might be the most up-to-date reference.

[szager-chromium]

Now that I'm looking at the code (which ironically I wrote), I see there there is a variant that allows you specify destination width/height, and applies that as a scale factor to the element's paint record. There is also a variant that allows you to specify a source rect that behaves the same as for texElementImage2D. There is not however a variant that allows you specify both of those.

I can't remember what I was thinking there, but the WebGPU API shape has definitely not had a lot of scrutiny yet, and can still be easily changed.

[jakearchibald]

It does seem worth having an API that allows both. It'd seem odd if the WebGPU version was the least capable.

I guess this can get more review when it's properly proposed somewhere.

[szager-chromium]

The missing overload has been added to the WebGPU equivalent API.

[kenrussell]

Four overloads of this single entry point seem like a little much to me. Would it be possible to collapse this down to two by saying that passing 0 for GLsizei width/GLsizei height has default behavior of deriving the width/height from the CSS borderbox of the element at the time of rendering?

[kenrussell]

This is a duplicate of the entry on line 2240 and should be removed.

[kenrussell]

This overload - if it's necessary - should define how the width and height are computed, similarly to the overload at line 2215.

[szager-chromium]

Four overloads of this single entry point seem like a little much to me. Would it be possible to collapse this down to two by saying that passing 0 for GLsizei width/GLsizei height has default behavior of deriving the width/height from the CSS borderbox of the element at the time of rendering?

Functionally this would work, but it contrasts with texImage2D(), which does not interpret width/height of zero to mean intrinsic size. I'd be a bit concerned that this behavior might be magical and non-intuitive.

[lexaknyazev]

@szager-chromium

The existing texImage* commands support texture uploads from TexImageSource objects defined as:

typedef (ImageBitmap or
         ImageData or
         HTMLImageElement or
         HTMLCanvasElement or
         HTMLVideoElement or
         OffscreenCanvas or
         VideoFrame) TexImageSource;

Have you considered adding a new type to this list, e.g., RenderedElement, that would be returned by a brand new function like renderElement(only_relevant_opts,... ) and then could be used in all existing texture-related GL commands? Then this entire feature would be detached from multiple GL-specific semantics that otherwise must be carefully considered with the currently proposed design.

Just saying that rendering HTML to a 2D Canvas and using that Canvas as a WebGL texture source provides much more functional and convenient experience (modulo pixel redirection) and likely bypasses all issues listed below simply because they are already defined for uploads from TexImageSource objects.

---

Issue 1: Texture allocation vs texture upload

The texImage2D command that was clearly used as a base for texElementImage2D design is one of the most error-prone in GL because it does multiple things at once. Of course, implementations usually skip certain steps as appropriate but the semantics are still as listed:

1. It (re-)defines the texture state, including dimensions and format.
2. It (re-)allocates the texture memory.
3. It replaces the texture data, potentially with runtime type conversions.

Sub-region texture updates are possible with the texSubImage2D command that does only the third step from the list above but it looks like such functionality is not supported in the current proposal.

WebGL 2.0 supports a more efficient and safer workflow with immutable-format textures like this:

gl.texStorage2D(...) // forever defines all dimensions and the pixel format, zero-inits the texture
gl.texSubImage2D(...) // updates the specified texture region with provided data

The benefit here is that there won't be any implicit redefinitions and/or reallocations that are a common source of bugs both in apps and in implementations. This workflow would require setting the texture's dimensions before uploading the data, though.

Issue 2: Texture types

The texImage2D command that was clearly used as a base for texElementImage2D design supports two texture types (via the target parameter): TEXTURE_2D and six TEXTURE_CUBE_MAP_* values corresponding to cube map faces. Nothing technically prevents an application from rendering HTML elements to cube map faces because a rendered element is just an array of pixels at the end of the day. However, cube map faces must be squares and have equal dimensions, so the command variants that omit the target texture dimensions are likely less suitable for this texture type.

WebGL 2.0 also supports 2D texture arrays and sliced 3D textures via texImage3D and similar commands. As with cube maps, there's nothing wrong with uploading rendered elements to such texture types but GL supports per-slice data uploads only with the sub-region, i.e., texSubImage3D, command.

Issue 3: Texture formats

The internalformat, format, type tuple has a very specific meaning in GL that does not fully make sense for uploading rendered elements:

• internalformat is supposed to be the actual GPU format used by the texture. There're historical reasons for supporting so called "unsized" formats like RGBA for that parameter but that's obsolete and should not be used going forward. Looking at the current proposal, the RGB and RGBA values should be changed to RGB8 and RGBA8 respectively.
• format and type describe the application data supplied to the texture upload command. So when the texture data comes from some ArrayBufferView the format and type parameters are used to tell GL how to interpret raw data. In case of uploading rendered elements, there's no application-side byte data to describe so these parameters serve no purpose.

[szager-chromium]

@lexaknyazev -- Thank you for the very detailed analysis, this is very helpful.

We have discussed the possibility of adding another type to the TexImageSource union, but only briefly and without addressing the issues you've raised. The API proposal does feature an intermediate object called ElementImage, which is a static snapshot of an Element's painted output returned by a call to canvas.captureElementImage(element). However, I see two complications with adding it to TexImageSource.

• ElementImage was added to the API to facilitate canvas drawing on a worker thread, where references to Element are not permitted. For non-worker-thread usage, we allow the source argument to the API to be either Element or ElementImage. We lose some convenience if the call to captureElementImage() becomes mandatory. Not a huge deal, but we have been trying to keep the API as streamlined as possible in the common case.

• All of the existing variants of TexImageSource are bitmaps or trivially convertible to bitmap, but ElementImage is not. We rely on the other parameters to texElementImage2D (specifically sx,sy,swidth,sheight,width,height) when rasterizing the ElementImage to a bitmap. So we would actually need a different intermediate method call to produce the bitmap. So the invocation becomes something like:

const bitmap = canvas.captureElementBitmap(element_or_element_image, sx, sy, swidth, sheight, width, height);
gl.texImage2D(bitmap, ...);

I don't think this is an improvement over your suggestion of drawing the element into a 2D canvas and then calling gl.texImage2D(canvas_2d, ...), so I don't see the point in adding it.

Of the three issues you described, I think the first is the most important. There should be a way to re-upload the texture without re-allocating it. This is especially important for continuously-animating pages where the HTML content may update on every animation frame. I think this one should be addressed before the feature ships in Chrome.

Issue 2 seems a bit esoteric to me; and as you point out it's avoidable by specifying an explicit destination size. Maybe there's something we can do to make that nicer, but I wouldn't consider this a blocker.

The internalformat, format, type arguments were added just to conform to existing API's. I've observed that it's commonplace in OpenGL to have long lists of parameters that require specific static values, so I guess "when in Rome." I think we'd be happy to drop those arguments if there are no objections.

[lexaknyazev]

Issue 2 seems a bit esoteric to me; and as you point out it's avoidable by specifying an explicit destination size. Maybe there's something we can do to make that nicer, but I wouldn't consider this a blocker.

The first step with the issue 2 is to decide whether texture types other than TEXTURE_2D should be supported. The spec has to define valid target values anyway and in the worst case it can say that only TEXTURE_2D is valid. However, if cube map faces are also valid, there's no technical reason to not support TEXTURE_2D_ARRAY but that type would require a slightly different API shape to be created and/or uploaded, which would depend on the resolution of the sub-region updates.

The internalformat, format, type arguments were added just to conform to existing API's.

There are some subtleties to consider here. For example, the current proposal has this line:

Internal Format	Format	Type
RGBA16F	RGBA	HALF_FLOAT FLOAT

Setting the type to FLOAT and assuming literal GL semantics, it would mean that the browser is supposed to rasterize the element to float32 pixels and then pass them to the GPU for driver-internal conversion to float16. I don't think any implementation would intentionally support that path. Removing format and type would naturally eliminate such cases.

Another question is about all no-alpha formats. They are usually emulated in lower layers and in worst cases may involve costly per-pixel processing, e.g., to copy three-component RGB values to natively-supported RGBA with alpha set to 1.0. If the element rasterizer has an option to produce all-opaque alpha, that option could be passed explicitly to make sure that the declared texture format matches the internal representation.

[cabanier]

Of the three issues you described, I think the first is the most important. There should be a way to re-upload the texture without re-allocating it.

Some versions of this API take a width and height for the texture so if the author passes in the same values, there wouldn't be a reallocation.

This is especially important for continuously-animating pages where the HTML content may update on every animation frame. I think this one should be addressed before the feature ships in Chrome.

Do you know how this is handled inside Chromium? I suspect it has to optimized this internally to get good performance.

[szager-chromium]

Of the three issues you described, I think the first is the most important. There should be a way to re-upload the texture without re-allocating it.

Some versions of this API take a width and height for the texture so if the author passes in the same values, there wouldn't be a reallocation.

This is especially important for continuously-animating pages where the HTML content may update on every animation frame. I think this one should be addressed before the feature ships in Chrome.

Do you know how this is handled inside Chromium? I suspect it has to optimized this internally to get good performance.

This is frankly well outside my area of expertise; but in the chromium implementation we funnel into the same code path as texImage2D quite early so I would expect that any existing optimizations of that function to avoid reallocations will apply to texElementImage2D.

[szager-chromium]

Issue 2 seems a bit esoteric to me; and as you point out it's avoidable by specifying an explicit destination size. Maybe there's something we can do to make that nicer, but I wouldn't consider this a blocker.

The first step with the issue 2 is to decide whether texture types other than TEXTURE_2D should be supported. The spec has to define valid target values anyway and in the worst case it can say that only TEXTURE_2D is valid. However, if cube map faces are also valid, there's no technical reason to not support TEXTURE_2D_ARRAY but that type would require a slightly different API shape to be created and/or uploaded, which would depend on the resolution of the sub-region updates.

I would be inclined to restrict this to only TEXTURE_2D. Given the vagaries of converting from CSS pixels to the canvas grid, I think any kind of origami would be fraught, so I'm in favor of following the KISS principle.

The internalformat, format, type arguments were added just to conform to existing API's.

There are some subtleties to consider here. For example, the current proposal has this line:

Internal Format
Format
Type

RGBA16F
RGBA
HALF_FLOAT FLOAT

Setting the type to FLOAT and assuming literal GL semantics, it would mean that the browser is supposed to rasterize the element to float32 pixels and then pass them to the GPU for driver-internal conversion to float16. I don't think any implementation would intentionally support that path. Removing format and type would naturally eliminate such cases.

Removing those arguments sounds good to me!

Another question is about all no-alpha formats. They are usually emulated in lower layers and in worst cases may involve costly per-pixel processing, e.g., to copy three-component RGB values to natively-supported RGBA with alpha set to 1.0. If the element rasterizer has an option to produce all-opaque alpha, that option could be passed explicitly to make sure that the declared texture format matches the internal representation.

I'm pretty sure the web rendering system always produces RGBA (not sure if this is chromium-specific, but I doubt it). But any way you slice it, I can't think of any reason why we would want to allow users to pick a different internal format, even if it could be supported. So again, my KISS instinct is to just drop this parameter.

[lexaknyazev]

I'm pretty sure the web rendering system always produces RGBA (not sure if this is chromium-specific, but I doubt it). But any way you slice it, I can't think of any reason why we would want to allow users to pick a different internal format, even if it could be supported. So again, my KISS instinct is to just drop this parameter.

This means removing all RGB, RGB16F, and RGB32F lines from the format tables.

And while we're here, are 8-bit values supposed to be sRGB-encoded? If so, the internal format for them should be SRGB8_ALPHA8 because WebGL does not support format reinterpretation.

[szager-chromium]

On the question of excessive memory reallocations: even if we are able to optimize for the case where the size does not change with successive invocations, that still leaves a potential pitfall when animating the size or scale of an element. In that case, you would presumably want to allocate a texture at the maximum size of the animation and then draw sub-areas of it during other animation phases.

I think this could still be accomplished in a couple of ways with the API as written:

• Use constant swidth/sheight parameters that match the maximum size of the animation. This would likely require some fiddling with the sx/sy parameters to preserve alignment.

• Place the animating element inside a <div> that is statically sized to the maximum animation size, and draw the <div>. This is effectively equivalent equivalent to the first suggestion, but the alignment can be handled via CSS rather than sx/sy parameters.

Neither of those seems too onerous to me, but it does require the developer to be aware of the issue and take measures to avoid it. Requiring a call to texStorage2D() would force the issue, at the cost of a bit of extra complexity up front, even in simple cases.

[kenrussell]

Is the precise size of the Element or ElementImage known at the time the app intends to upload one or the other into the WebGL texture? My understanding from https://github.com/WICG/html-in-canvas is that if the app registers a ResizeObserver for the element's devicePixelContentBox, then it will get the most up-to-date pixel-accurate size at the point where the paint event is fired. And the ElementImage contains the width and height.

In that case it would be feasible to do a sub-image upload via a new texElementSubImage2D into a texture allocated via texStorage2D. If not - it's impractical.

It might be better to start simple regardless and only support texElementImage2D. Chrome's had longstanding performance issues where it was difficult to implement GPU-accelerated sub-image uploads for video textures given WebGL's semantics (rejecting the upload if the texture's internal format was incompatible with that of the existing WebGL texture). It was easier to redefine the texture every time with the source's internal format and size.

[szager-chromium]

I think I'd prefer to start as simple as possible and then return to the idea of texElementSubImage2D if and when there's demand for it. I'm open to other opinions though.

[cabanier]

@szager-chromium I tried to incorporate everyone's feedback in the new PR. Let me know if you'd like to see changes.

[cabanier]

@szager-chromium Any comments on the latest PR?

[ewewraw]

And while we're here, are 8-bit values supposed to be sRGB-encoded? If so, the internal format for them should be SRGB8_ALPHA8 because WebGL does not support format reinterpretation.

Looking at the canvas spec with the default set to srgb, the default internal format colorspace should probably be sRGB as well.

When placing html with RGB8 passed as internalFormat as the explainer suggests, I can see the difference in colors between regular HTML and HTML-in-canvas, so probably the default format should be a gamma corrected one?

[cabanier]

Looking at the canvas spec with the default set to srgb, the default internal format colorspace should probably be sRGB as well.

When placing html with RGB8 passed as internalFormat as the explainer suggests, I can see the difference in colors between regular HTML and HTML-in-canvas, so probably the default format should be a gamma corrected one?

I think there should be no difference when it's set to RGB8. @szager-chromium, is this a bug in Chromium's implementation?

[cabanier]

@lexaknyazev please let me know if you agree with my latest changes.

[lexaknyazev]

My view on this is as follows:

• Unpack alignment is already ignored for TexImageSource uploads so it's natural to ignore it here.
• Various unpack skip and dimension parameters cannot be reliably used because the rasterized size is not always known. Therefore, ignoring them seems reasonable. This might be further conditioned on the presence of config.width and config.height, though, because in case they are provided, the regular pixel unpack logic applies just fine.
• Unpack buffer binding is not a part of pixel state and since it generates an error on all other inapplicable commands, that error should be here as well.

[lexaknyazev]

Looking into this again, I think there's no strong reason to specify this special behavior given that implementations will internally call texImage2D anyway. Passing the input level value through is just as simple as hard-coding 0.

Any application concerns about texture mipmap completeness are resolvable using the config params if needed.

[lexaknyazev]

The spec should be very clear about the transfer functions used in each case.

I'd expect that 8-bit values are always sRGB-encoded and the choice between RGBA8 and SRGB8_ALPHA8 is only to allow apps to bypass hardware sRGB decoding during texture sampling if they wish so for some reason.

However, floating-point values are usually not sRGB encoded. If we aren't sure about the latter, maybe floating-point options could be omitted for now.