Provide a synchronous constructor for non-empty streams

[zslayton]

TL;DR: There is not currently a way to construct a non-empty stream in a synchronous context, making it impossible for a guest component to virtualize several WASI APIs. While planned features like cooperative threads will eventually make it possible to populate a stream by spawning background tasks, I believe that having a straightforward stream constructor that takes a list would be generally useful and hopefully (?) much simpler to implement.

---

(Apologies if this is not the best repo for this issue, please feel free to move it as appropriate.)

WASI-Virt currently supports WASI versions v0.2.1 and v0.2.3. Both of these are quite old relative the ecosystem; cargo build --target wasm32-wasip2 produces components that depend on WASI v0.2.9. The latest 2.x release is 0.2.12, and the first 0.3.x release landed last month. WASI-Virt also uses exact-match dependency resolution, so while a v0.2.12 export could satisfy a v0.2.3 import in theory, it gets rejected in practice. Functionally, this means that WASI-Virt cannot be used with recent toolchains.

I attempted to enable semver-compatible dependency resolution, but learned that it wouldn't be possible so long as WASI-Virt was using WASI p2. P2's use of resources to represent IO streams always lead to transitive import/export conflicts:

4: world exports wasi:io/streams@0.2.3 but it's also transitively used by an import which means that this is not valid

These conflicts should be obviated by P3's use of stream instead of resources. I set out to test this by making a simple component that would virtualize wasi:filesystem (and nothing else). However, I quickly ran into an issue when trying to implement descriptor.read-via-stream (see #wasi > Creating a `stream` in a sync context).

In short: read-via-stream is synchronous function. A guest implementation can construct a stream reader/writer pair via wit_stream::new(), but the guest cannot (meaningfully) spawn a background task to populate a stream via the writer. Cooperative threads should make this possible when they arrive early next year.

The WASI-Virt use case does not require a sophisticated writer implementation. The bytes that represent a file are known at virtualization time; the file descriptors are backed by a simple Vec<u8>. However, so far as I can tell, there is no way to construct a stream from a list<u8> in a synchronous context.

I know there's a larger design effort happening around slicing and splicing stream; I wonder if there's room for a simple synchronous constructor. (Regrettably, the lack of user generics does mean that you couldn't define a func from-list(list<T>) -> stream<T> in the near term.)

[lukewagner]

Thanks for working on WASI-Virt, doing all this exploration and writing up a nice clear issue!

So I believe that we can hopefully solve your original virtualization issue in a wasip2 context as commented here. I also think there's a path for WASI-Virt to virtualize read-via-stream in a 0.3.0 context as commented here. So hopefully those can address the immediate WASI-Virt use cases without having to add anything.

Considering just the design question of whether we should add a new built-in to create a stream from an array of bytes: I think this would end up requiring streams to maintain an internal memory buffer which we've otherwise tried to avoid: currently streams are always unbuffered and just synchronize a direct copy from a stream.writer-supplied buffer to a stream.reader-supplied buffer. So my impression is that we shouldn't add a built-in to do this.

However, one old-and-still-great idea that I think would directly address your WASI-Virt use case is #138, which proposes a built-in to (synchronously) create a (readable end of a) stream directly from a passive data segment, which would avoid the intermediate copy through linear memory entirely and simply work in a synchronous read-via-stream context. In particular, the key advantage of creating a stream from a data segment is that data segments are immutable and thus the stream wouldn't need to make an eager copy (as it would if creating the stream from a buffer in linear memory). If anyone wants to work on #138, I'd be happy to help write it up as a spec proposal.