better documentation for learners

[dominictarr]

we need lots more examples to make pull streams easy to learn.

1. simpest source
2. simplest sink
3. simplest through (mention duplex but leave it for later)
4. errors and aborting. (take(n), what happens if you don't do this: fs / streams: file descriptor leak on connection abort nodejs/node#1834
5. some useful streams: pull-cat, pull-paramap, pull-defer, pull-many, pull-merge, pull-through, pull-pushable, pull-traverse, pull-stringify, pull-level, pull-glob (maybe split these up by difficulty? with examples where they are useful?)
6. interaction with node streams? stream-to-pull-stream
7. muxrpc (maybe put this sooner?)
8. secret-handshake?

@yoshuawuyts @ahdinosaur @DamonOehlman @pfraze any other ideas?

[yoshuawuyts]

I'd put networking and fs in here too, perhaps also JSON parsing - covering a good rang of practical uses makes it more likely people will use it.

[ahdinosaur]

ooh yes, this is awesome! really keen on more accessible documentation for pull streams. 🐱

other ideas:

• composable pull streams
  • pull(through, through, through) returns the equivalent of a through stream
  • pull(source, through, through) returns the equivalent of a source stream
  • pull(through, through, sink) returns the equivalent of a sink stream
• lazy pull streams
  • @dominictarr mentioned pull-glob as an example
• functional programming with through streams (pull.map, pull.filter, pull.reduce, and such)
• common message passing architectures
  • "request response" (RPC)
  • "publish subscribe"
  • "pipeline"
• duplex streams for common transports (http, ws, cli)
• how to use pull streams with other "frameworks" to build full stack apps: react (e.g. patchwork), redux, (pull-redux?), ...

also, some examples of other nice "stream" documentation in the wild:

• ReactiveX
• RxMarbles
• rx

/cc @pietgeursen

[dominictarr]

I think the most important thing here is to find realistic examples that represent problems that are interesting and or people actually have.

Oh, another category is real time data, and streaming of that. maybe an example is an infinite scroller that uses back pressure to stop reading while until you scroll down more.

[dominictarr]

@ahdinosaur compose https://github.com/dominictarr/pull-stream-examples/blob/master/compose.js
can you think of some more examples where composition is useful?

[eugene-eeo]

@dominictarr maybe I misunderstood this because I'm still trying to learn this concept, but running the last example in pull.js, gives 100 results instead of 101. Maybe the code should be changed to:

function take(n) {
    return function(read) {
        return function(abort, cb) {
            if (!n--) return read(true, cb);
            read(null, cb);
        };
    };
}

[dominictarr]

@eugene-eeo good point. fixed

[yoshuawuyts]

Another cool pattern I found myself needing the other day was:

Eventual values

Return a stream from a function synchronously, emit data from it
asynchronously.

const notify = require('pull-notify')
const pull = require('pull-stream')
const xhr = require('xhr')

pull(request('foobar.com'), pull.log())

function request (url) {
  const xhr$ = notify()

  xhr(url, (err, res, body) {
    if (err) return xhr$.abort(err)
    xhr$(body)
    xhr$.end()
  })

  return xhr$.listen()
}

---

Probably not doing it 100% right here - while this could be expanded to be a through stream, the package used allows for multiplexing; wonder if I'm doing it right. Think it'd be a neat little pattern to throw at users

[ahdinosaur]

hey @yoshuawuyts, i wrote pull-async for exactly that use case (returning a single value or error), for returning more than one value see pull-defer.

pull-async example:

function request (url) {
  return pullAsync(cb => {
    xhr(url, (err, res, body) => {
      if (err) return cb(err)
      cb(body)
    })
  })
}

pull-defer example:

function request (url) {
  var deferred = pullDefer.source()
  xhr(url, (err, res, body) => {
    if (err) deferred.abort(err)
    else deferred.resolve(pull.values([body]))
  })
  return deferred
}

[dominictarr]

@yoshuawuyts that way ruins back pressure (not that it matters with XHR, though)
in general, you want to use pull-defer for this. hmm, although, this is making me realize that pull-defer is not obvious enough (if I have to tell you two about it)