# Node Streams & Generators
- Stream? Interface to transfer streaming data - e.g. I/O, files, http request
- **Types:**
- *Writable*,
- *Readable*,
- *Duplex* (Websockets),
- *Transform*
- Mainly callback support, Promises since `15.0.0`
- For `Buffer`, `strings`, otherwise object mode
- `null` has a special purpose.

## Buffering
- Inner buffer size is defined by `highWaterMark` option in constructor in byes or in number of objects when working in `object mode`.
- Inner queue store data processing until inner buffer is full or other data are processed
- `Readable` reads data until you get them with `read`
- For `Writable` method `write` is returning `true` when its enough space in buffer, `false` if it is not.
- [Backpressure](https://gitlab.ack.ee/Backend/backend-hub/-/blob/master/guides/nodejs-streams.md#backpressure)

## Events
- error
- close
- data - when readable stream is flushing its buffer
- finish - when `stream.end()` is called and buffer was flushed
- **drain** - called when `Writable` stream can continue after returning `false` for `write` function
- **pipe** / **unpipe** - called when piping (/unpiping) `Readable` and `Writable` streams. Working with buffering and [backpressure](https://gitlab.ack.ee/Backend/backend-hub/-/blob/master/guides/nodejs-streams.md#backpressure)

## Async iterators / generators
### What it is?
- **Iterator** somethig you can "go through" (e.g. collection aka array, stream)
- **Generator** Iterator where item for each iteration is generated before iteration starts = lazy evaluation.

### Why generators?
- When you are generating/reading huge amout of data or you need to work in batches (e.g. line by line) you want to save memory (so you really dont want to save it in array).
- Construct from other programming languages (OOP)
- Some people find it's syntax more readable

### Node iterators & generators
Iterators are just object implementing [iterator protocol](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#the_iterator_protocol).
Summary: They need `next()` function returning object:

{
    done: boolean // Finished?
    value?: any // Omitted when done = true
}

Generators are just "iterators with inner state". Special syntax - **star function** and **yield** (or **yield\***) keyword.

### Examples

function* intervalGenerator(fromDate: Date, toDate: Date, step: number) {
    let stepDate = new Date(fromDate);
    let nextStepDate: Date;

    while (stepDate < toDate) {
        nextStepDate = new Date(stepDate);
        nextStepDate.setDate(stepDate.getDate() + step);

        if (nextStepDate > toDate) {
            nextStepDate = new Date(toDate);
        }

        yield {
            from: stepDate,
            to: nextStepDate,
        };
        stepDate = nextStepDate;
    }
}

const fromD = new Date('2021-12-01');
const toD = new Date('2021-12-24');
const daysToXmas = intervalGenerator(fromD, toD, 1);

daysToXmas.next(); // => from: 2021-12-01, to: 2021-12-02
daysToXmas.next(); // => from: 2021-12-02, to: 2021-12-03

for (const day of daysToXmas) {
    console.log(day); // => from: 2021-12-03, to: 2021-12-04;
                      // => from: 2021-12-04, to: 2021-12-05, 
                      // => ...
}

You can always use object style

const daysGeneratorObj = (fromDate: Date, toDate: Date, step: number) => ({
    [Symbol.iterator]: () => {
        let stepDate = new Date(fromDate);
        return {
            next: () => {
                let nextStepDate = new Date(stepDate);
                nextStepDate.setDate(stepDate.getDate() + step);

                if (nextStepDate > toDate) {
                    nextStepDate = new Date(toDate);
                }
                const value = {
                    from: new Date(stepDate),
                    to: nextStepDate,
                };

                stepDate = nextStepDate;
                const isDone = nextStepDate === stepDate;

                return { value: isDone ? value : undefined, done: isDone };
            },
        };
    },
});

const fromD = new Date('2021-12-01');
const toD = new Date('2021-12-24');
const daysToXmas = daysGeneratorObj(fromD, toD, 1);

for (const day of daysToXmas) {
    console.log(day);
}

Or [you can just use nested functions](https://gitlab.ack.ee/Backend/flash-sport-ls-adapter/-/blob/development/src/app/services/infoService.ts#L94) and work without generator interface.

### Async generators
Same as generators, but they can use promises inside of itself, so they can be consumed by async loops.

For consuming async stream you can use `for await` loop

## Streams and generators
Streams are very similar to generators. Async generator are almost the same as stream, it just don't handles buffering. You can use own readable streams instead of generators:

import { Readable } from 'stream';

const dateReader = (fromDate: Date, toDate: Date, step: number) => {
    let stepDate = new Date(fromDate);
    return new Readable({
        objectMode: true,
        read() {
            let nextStepDate = new Date(stepDate);
            nextStepDate.setDate(stepDate.getDate() + step);

            if (nextStepDate > toDate) {
                nextStepDate = new Date(toDate);
            }
            this.push({
                from: new Date(stepDate),
                to: nextStepDate,
            });

            stepDate = nextStepDate;
            if (stepDate >= toDate) {
                this.push(null);
            }
        },
    });
};

const fromD = new Date('2021-12-01');
const toD = new Date('2021-12-24');
const daysToXmas = dateReader(fromD, toD, 1);

daysToXmas.on('data', day => {
    console.log(day);
});

// OR

daysToXmas.pipe(
    new Writable({
        objectMode: true,
        write: (interval, _, done) => {
            console.log(interval);
            done();
        },
    })
);

- You can create readable streams from generators

function* daysGenerator(fromDate: Date, toDate: Date, step: number) {
    // ... see above
}

const daysToXmas = Readable.from(daysGenerator(fromD, toD, 1));

// Use same as Readable stream above 8-)

- Readable streams have "inner async iterator", so you can read them as async iterators:

const daysToXmas = new Readable({ ///... })

(async function() {
  for await (const day of daysToXmas) {
    console.log(day);
  }
})();

- You can also [pipe async iterator to stream](https://nodejs.org/api/stream.html#stream_piping_to_writable_streams_from_async_iterators)

# Further reading
- [Official documentation: streams ](https://nodejs.org/api/stream.html)
- [Official documentation: async generators and streams](https://nodejs.org/api/stream.html#stream_additional_notes)
- [Hub guide by Jirka Šmolík](https://gitlab.ack.ee/Backend/backend-hub/-/blob/master/guides/nodejs-streams.md)
- [GCloud storage node streams doc](https://googleapis.dev/nodejs/storage/latest/File.html)
- [About backpressure](https://nodejs.org/en/docs/guides/backpressuring-in-streams/)
- ["Easy to read" artcile with async iterator](https://nodesource.com/blog/understanding-streams-in-nodejs/)
- [Streams vs Generators in simple examples](https://blog.logrocket.com/comparing-the-stream-api-and-async-generators-in-node-js-v10/)
- [Nice "crawler" example on async generator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#iterating_over_async_generators)