# Shenlearn V3;

##### General Questions
1. The source codes of files are all in `shenlearn/packages/storage-server/src/file/`
2. What is Yarn Lock?
A: https://classic.yarnpkg.com/en/docs/yarn-lock/#:~:text=In%20order%20to%20get%20consistent,the%20root%20of%20your%20project.
3. Why do some dependencies start with '@' ?
A1: https://stackoverflow.com/questions/49151868/why-do-some-npm-packages-start-with
A2: https://docs.npmjs.com/about-scopes
4. Dependencies vs DevDependencies ?
A:
5. What is frame work and why do we need it ?
6. What are generics, and how to use them ?
7. How are `Stream` and `Buffer` related in `Storage-server` ?
8. What is a CLI, and what is a yarn CLI ?
9. What is `fileName.d.ts` ?
A1: https://stackoverflow.com/questions/21247278/about-d-ts-in-typescript
A2: https://www.typescriptlang.org/docs/handbook/declaration-files/templates/global-d-ts.html
10. What is `JQuery` ?
A: https://www.w3schools.com/jquery/jquery_intro.asp
11. What is the use of the `bin` folder
12. What is dotenv;
A: https://www.npmjs.com/package/dotenv (Like literally everything you will everyneed is in this link);
13. Why do you have to declare a seperate module in `@types/postgres.d.ts` when you've already have postgres installed as a dependency ?
A:

14. When initializing the migration table why do we not have createdAt, deletedAt, and updatedAt in the table anymore ?
A: This is my personal obeservational guess but it seems like you input those variables when you
make a database operation; But still that doesn;t explain why this Item doesn't exist on the
migration table. Wouldn't there be no column for this input ?

15. What is npm registry ?
A: https://docs.npmjs.com/using-npm/registry.html

16. Why do you want to use parametrized Context ?
A: Cuz you want to customize items in ctx; i.e: ctx.database;
SubQ: So where exactly can you input this customized object into CTX?
A: It is defined in `Context/context.ts`;

17. What the fuck is this {join} form 'path' thing ?
18. What is a relation in postgresql ?
A:
#### app.ts

#### Migration Folder
##### 000-initial.ts;
##### Migration.ts;

Contians{
    Class Migration{
        name: string
        up: (sql: postgres.Sql<never>) => Promise<void>
    }
    
    const initalize = async (sql: postgres.Sql<never>): Promise<void> =>{...}
    
    const executedMigrations = async (sql: postgres.Sql<never>): Promise<string[]> =>{...}
    
    const definedMigrations = async (): Promise<Migration[]> =>{...}
    
    export const up = async (sql: postgres.Sql<never>): Promise<string[]> => {...}
}

Q1: what the fuck is `::REGCLASS` ?
A: https://www.postgresql.org/docs/8.1/datatype-oid.html (This is somewhat a generic answer, doesn't tell you everything)
A2: https://stackoverflow.com/questions/13289107/what-does-regclass-signify-in-\postgresql#:~:text=Explanation%3A,the%20oid%20of%20that%20relation%22. (aight this one is better)
Q2: what the fuck is OID(Operation Identifyers) used for in Postgres?
A1: https://stackoverflow.com/questions/5625585/sql-postgres-oids-what-are-they-and-why-are-they-useful
Q3: For all these sql statement why dont you even need a bracket to put your input string?
I.E:

    await sql`SELECT 'migrations'::REGCLASS;`

A: Checked it on npm/postgress. This is the default way of usage.

Like you just put a string after after your sql object.
Q4: What are the 3 most important things about OID ?

Q5: When are the migration functions called? (After which step) And why?
A: So The `migration.up` function is called in the Database `index` document;
And the reason is you want to first initialize the DB, and then make sure that the
connection is successfully established before you begin your migration;

Q6: What does `postgres([url],[options])` do? How does it work from a source code perspective ?
A: So according to npm postgre doc this function instatiates a `sql` instance.
SubQuestion: So what does the `params` of this fucntion do when instantiating a new `sql` instance?
A: It instatiates a `Sql` instance where the url of this connection is given by `process.env.postgres.com`

Q7: What is `SERIAL PRIMARY KEY`
A: This is a pretty good shit: https://kb.objectrocket.com/postgresql/postgres-serial-primary-key-1383#:~:text=A%20primary%20key%20is%20a,key%20column%20of%20a%20table.

Q-Initialize: What does this function do?

const initalize = async (sql: postgres.Sql<never>): Promise<void> => {
  try {
    await sql`SELECT 'migrations'::REGCLASS;`
  } catch (err) {
    await sql`
      CREATE TABLE IF NOT EXISTS migrations (
        id SERIAL PRIMARY KEY,
        name TEXT NOT NULL,
        created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
      );
    `
  }
}

A: In a very vague term this creates the `migration` table if it doesn't already exists.

Q-Migration-Class: What does this Class do?

Q-excecutedMigration: What does this function do?

const executedMigrations = async (sql: postgres.Sql<never>): Promise<string[]> => {
  const rows = await sql<{ name: string }>`SELECT name FROM migrations ORDER BY id ASC`
  return rows.map((row) => row.name)
}

A: It seems pretty straigh forward. Rows are the names of different tables in the `migration` table.
SubQ1: But when was the names of the migration tables ever specified?
A: So it seems like the name was delcare in file `000-initial.ts` in the line

    sql`
    CREATE TABLE files 

SubQ1.1: So this leads me to think, what if we want to create another table without doing roll back? Do we add multiple tables in the `000-initial.ts`. But my guess would be I just start adding more and more files like `001-TableName.ts` and there I just add a new table.

Q-definedMigrations: What does this function do?

const definedMigrations = async (): Promise<Migration[]> => {

  const files = readdirSync(__dirname)
    .filter((name) => name.match(/^[0-9]{3}\-/))
    .sort()
  const migrations = await Promise.all(
    files.map(async (file) => Object.assign(await import(join(__dirname, file)), { name: file }) as Migration),
  )
  return migrations
}

A: First, `files` is going to store the name of all the files in `migrations` folder.
then you would filter the names according to the predfined format that YC gave in the folder.
Then you would sort, the migrations in order, because the order matters. i.e the first table is
000, then second will be 001, and then so on.

Then `migrations` will catch the promises of all the operations we are going to conduct on `files`.
For each file, we are going to `join` them with with the current directory path. In other words, the result of `join(__dirname, file)` is going to be the `path` with which you can access files of this folder.

Then you `await` the `import` of file in the path, which of type `postgres.Sql<never>`, which makes sense because all the postgres SQL commands are of type `postgres.Sql<never>` and since the file that you are importing is full of SQL commands, it makes sense that it is of this type.

Now before the finale, Take a look at this file. Look at the bgining of it, you should see a

Interface Migration{
        name: string
        up: (sql: postgres.Sql<never>) => Promise<void>
    }

Now as we can see, that in `Migration interface` the key `up` is a function that takes in a parameter of percicely the type `postgres.Sql<never>` which completly matches the only function in `000-initial` or anyother future migration files.

Continue'd after this subQ

------------
SubQ: what does the upfunction takes in a `SQL command`, and when does the insertion of the command actually take place?
What I'm refering to is

up: (sql: postgres.Sql<never>) => Promise<void>

There seems to be no input inserted into this function call in `migrations/index.ts`; Then why do we still need declare this `up` function as this type above. Is it purely because we want need to match the format it is in `000-inital` ? If then why do we have the upfunction as this format in the `000-initial.ts` ?

A: (Investigate other usages of this the `up(sql)` function ) Aha, found it, it is being called in `up function of index file`

------------
Continued'd

Then we use `Object.assign(import, {name: file} as Migration)`
The purpose of this function is to first assign the name of the imported migration file which is of the type `(sql: postgres.sql)` to the object, and the `as Migration` command basically casts this combined object into `Interafce Migration`.

------------
SubQ1: What does this snipet of code do ?

    const files = readdirSync(__dirname)
        .filter((name) => name.match(/^[0-9]{3}\-/))
        .sort()

A: We first use the readdirSync() method to read synchronously read file from __dirname;

SubQ1.1: So first need to do WTF is readdirSync();

A: So it seems like to be a function in "fs" module in `Node.js` but I still dont know what it does. so the fs.readdirSync() method is used to synchronously read the contents of a given directory. The method returns an array with all the file names or objects in the directory.The options argument can be used to change the format in which the files are returned from the method.
For more detials: https://www.geeksforgeeks.org/node-js-fs-readdirsync-method/#:~:text=The%20fs.,.readdirSync(%20path%2C%20options%20)

------------

SubQ1.1.1: So what is the "fs" module and why do we need it ?

A: It stands for `File System`. We use to when we want to read files from places. For documentations refer to: https://nodejs.org/api/fs.html

------------

SubQ1.1.2: What does `.filter(name)` do ?

A: My guess would be since `.filter(name)` is chained after `readdirSync(__dirname)` like this `readdirSync(__dirname).filter(name)`. Maybe that`.filter(name)` is not a function of the `fs` module, rather it is a JS builtin fucntion. And to verify that I will now check the type of return value of `.readdirSync()`. Aa bingo, from: https://www.geeksforgeeks.org/node-js-fs-readdirsync-method/#:~:text=The%20fs.,.readdirSync(%20path%2C%20options%20). It returns an array of String, Buffer or fs.Dirent objects that contain the files in the directory. So now I just have to look up `.filter()` function of js. So by filtering is just to filter out the files in the folder that matches the format you want;

------------

SubQ1.1.2.1: What does `match(/^[0-9]{3}\-/)` function do?
A: It is a function of the `String` prototype. This function takes a `regex` pattern as input parameter and returns every part of the string that matches the `regex` pattern you've inputed. For more detailed reference: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/match
And the reason that you want to ^[0-9]{3} is to match the first 3 `char` of the begining of the string that starts with 3numbers. And as we can see, the only other file in `migration` folder starts with `000`.

------------

SubQ1.1.3: What does `.sort()` do?
A: The sort just line them up by 000, 001, and so on. See the first one is 000.

------------

SubQ1.1.3.1: But why do you need to sort tho ? Does the order matter ?

------------

SubQ1.2: So when is the value of `__dirname` declared ?

A: Actually, my intellectual educated guess is that `readdirSync(__dirname)` with out specifying `__dirname` would just result in reading all the files in the current folder, and returning the file names as String in an array.

------------

SubQ2: What does this snippet of code do ?

  const migrations = await Promise.all(
    files.map(async (file) => Object.assign(
    await import(join(__dirname, file)), { name: file }) as Migration),
  )

A: files is Type String[], where elements are all names of the files in `__dirname`. Ok from there, `Join(__dirname, file)` gives the path to each file in the `migration` folder. Then `Object.asssign(await import modeule, {name: file}` gives a key `name` to the imported module, with the `name` of that file which is file from files[], makes sense right. The reason we use `Promise.all()` because we want to wait for all the promise in `files.map()` to resolve.

---------------------------------

SubQ2.1 what does `Promise.all()` do, what is it, and why do we use it here?

*Reference*: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

*What is It / what does it do*: The Promise.all() method takes an iterable of promises as an input, and returns a single Promise that resolves to an array of the results of the input promises. This returned promise will resolve when all of the input's promises have resolved, or if the input iterable contains no promises. It rejects immediately upon any of the input promises rejecting or non-promises throwing an error, and will reject with this first rejection message / error.

For example reference to this snipet of code down below here:

const promise1 = Promise.resolve(3);
const promise2 = 42;
const promise3 = new Promise((resolve, reject) => {
  setTimeout(resolve, 100, 'foo');
});

Promise.all([promise1, promise2, promise3]).then((values) => {
  console.log(values);
});
// expected output: Array [3, 42, "foo"]

*Why do we use Promise.all, are any of the input values in it Promises*:
___________Please fill in the blank______________

---------------------------------
SubQ2.2:

What is `Object.assign()` ?
A: The Object.assign() method copies all enumerable own properties from one or more source objects to a target object. It returns the target object.

What is does is shown below:

const target = { a: 1, b: 2 };
const source = { b: 4, c: 5 };

const returnedTarget = Object.assign(target, source);

console.log(target);
// expected output: Object { a: 1, b: 4, c: 5 }

console.log(returnedTarget);
// expected output: Object { a: 1, b: 4, c: 5 }

---------------------------------

SubQ2.3:

What is `path` module in `Node.JS` ?
A: The Path module provides a way of working with directories and file paths.

what is `join()` function and what does it do ?
A: It joins all strings that passed into this function as parameter;

---------------------------------

SubQ2.4:

What does `.map()` function do ?
A: This function does a certain operation on all elements of an array, and stores the result into another array.

---------------------------------

SubQ2.5:

Properway of `import`:
A: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#:~:text=To%20dynamically%20import%20a%20module,js')%20.

---------------------------------

Q-up: What does this function do?

export const up = async (sql: postgres.Sql<never>): Promise<string[]> => {
  await initalize(sql)
  return await sql.begin(async (sql) => {
    const executed = await executedMigrations(sql)
    const migrations = await definedMigrations()

    if (
      JSON.stringify(executed) !==
      JSON.stringify(migrations.slice(0, executed.length).map((migration) => migration.name))
    ) {
      throw new Error('migration seems broken. please check if migrations file and exists is same')
    }

    const toExecute = migrations.slice(executed.length)

    for (const migration of toExecute) {
      await migration.up(sql)
      await sql`INSERT INTO migrations (name) VALUES (${migration.name})`
    }

    return toExecute.map((migration) => migration.name)
  })
}

A:
* First you call the `initialize(sql)`. the param `sql` should just be the link to your db from config.DB.someshit. The `initialize(sql)` function `selects` the `migration` table, and if `migration` table does not exists yet, it creates a one for you.
* Then first you call the sql.begin() function to `commit` if promise is Resolve, and call `rollback` if failed.
* In side the beign call back function you first check the executed files are equal to the first `executed.length` files in your `migration` folder. (Note that `migrations` is already sorted in the order of 000-Initial, 001-TableName1, 002-TableName2 and so on...).
* Then your toExecute will be `migrations.slice(executed.length)`, this basically says, It starts from index `migration.length` to the end of `migrations` array.
* Then here comes the complex part. For each migration of toExecute you do `migration.up(sql)` where sql is the link to your pg, the`config.DB.someshit`. Now if you remeber, the `Interface Migration` which is the `type` of your migration has 2 keys `name` and function called `up`. This `up` is `migration-file` that your name leads to. IMPORTANT: The reason you pass the `sql` param into this `up()` fucntion is because with in your migration file, the sql object must know which `db` you are referring to. So remember in `database.index` you've had `this.sql = postgres(url)` in the constructor? That was when this `postgres.sql` object was instanciated. The `sql` param that you've passed into your `up()` fucntion is the same `url` which is from `app.ts` when this line was called`const database = new Database(config.postgres.url)`. Now given both the `sql` params are the same, the code knows which database to act upon.
* Then in the `migration.up(sql)` function, you would have a `postgres.Sql` command. I.e: using `000-initial` as an example it will execute the code below:

//Path migrations/000-initial.ts

import postgres from 'postgres'

export const up = async (sql: postgres.Sql<never>): Promise<void> => {
  await sql`CREATE EXTENSION IF NOT EXISTS "uuid-ossp"`
  await sql`
    CREATE TABLE files (
      id         UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
      private    BOOLEAN NOT NULL DEFAULT TRUE,
      filename   TEXT NOT NULL,
      metadata   JSONB NOT NULL DEFAULT '{}',
      url        TEXT NOT NULL,
      upload_at  TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
      links      JSONB NOT NULL DEFAULT '{}'
    )
  `
}

#### ***** Important *****
See the lines after the codes of format `await sql"*Command*"` ?
Had you not passed in the `config.postgres.url` as `sql` into this fucntion, your code wouldn't know what database to act up on.

IT IS ALSO VERY IMPORATANT TO REALIZE THAT THIS IS THE STEP WHERE THE MIGRATION TAKES PLACE
#### ***** Important *****

* Then `await sql INSERT INTO migrations (name) VALUES (${migration.name})` this line inserts what over operation name you have commited onto the table in the `migrations` table of your DB. So here you need to realize that the `migration.name` doesnt strictly has to be the name of a new table right? It can aslo be something like `007-add_new_field--tableName--ass`. Right ?

* At last: You finally can put all the `name` of all the `toExecute` into an `Array<String>` and return it.

---------------------------------
SUBQ: What does this thing below do ?

for (const migration of toExecute) {
  await migration.up(sql)
  await sql`INSERT INTO migrations (name) VALUES (${migration.name})`
}

---------------------------------
SubQ: What does `sql.begin()` do ?
A: Calling begin with a function will return a Promise which resolves with the returned value from the function. The function provides a single argument which is sql with a context of the newly created transaction. BEGIN is automatically called, and if the Promise fails ROLLBACK will be called. If it succeeds COMMIT will be called.
Reference: https://www.npmjs.com/package/postgres (keyword: Transaction)

---------------------------------
SubQ: Think about what this does

 JSON.stringify(executed) !==
 JSON.stringify(migrations.slice(0, executed.length).map((migration) 
 => migration.name))

A: So first, `executed` is of type `string[]` and, what's stored here are the names of the already migrated tables or all the migration operations you've done in the past.
On the hands `migrations` contains all the migrations files that are stored currently in the `migrations` folder.
So what you do with the second part of the if-statment is that you only slice the part that has been already executed, note that the orders are also the same since you have sorted it. So the first `executed.length` elements should be the same. And if not then there is something wrong.

---------------------------------

SubQ: Why the fuck do you `migrations.slice(0, executed.length)` ? Isn't that shit just the same as `migrations` itself ?
A:  Yeah its not the same cuz, its not `migrations.slice(0,migrations.length)`. 我刚才脑子`穿希`了。

---------------------------------

SubQ: But why you want to make sure that `m.length` is equal to `e.length` ? In other words why is there something wrong if they aren't the same ?
A: So think about it this way. If you wanna make a new operation on to the DB, then you would need to create a new migration file right? then `m.length` is not gonna be the same as `e.length` right ? So that's why you slice from `0` to `e.length`

---------------------------------

#### DataBase
##### index
Q1. When do you call database.middleware() in app.ts ? Why is it in that particular order?
A:
Q2. Where is the constructor database(url) called in app.ts? Why is it in that particular order?

---------------------------------
###### middleware()；

public middleware = (): ((ctx: KoaContext, next: () => Promise<void>) => Promise<void>) => {
    return async (ctx, next): Promise<void> => {
      await this.ready
      ctx.database = this
      await next()
    }
  }

middleware = (): ((ctx: KoaContext, next: () => Promise<void>)
=> Promise<void>) => {...}

* First middleware is a function that takes no parameter.
* The return type of this function is a function that is of format after the `:`

##### another very important snipet of code in this section is the below

    return async (ctx, next): Promise<void> => {
      await this.ready
      ctx.database = this
      await next()
    }

Q: There are 4 questions you wanted to consider in this snippet above;
1. why do you wanted to await `this.ready` ? and also what exactly does `this.ready` mean from the code perspective ?

2. where did the `this` came from ?
A: The `this` instance was intantiated in `app.ts` on the line

    const database = new Database(config.postgres.url)

3. where did the `ctx` and `next` come from ?
A: To answer this question it is only safe to first investigate how does `koa` pass on its `ctx` and `next`. So what I heard from `Eleby Line Max` is that the first line of `app.use()` from `app.ts` which contains the first `ctx`. And from there every `app.use()`.

4. Also, how did this code even know that `ctx` was refering to `KoaContext` and ? Like you just wrote `(ctx, next)` which out any type signifier. I'm guessing maybe that it is a chain of operation where `koa` was involved, so it automatically know that `ctx` was referring to `koaContext`.

A: So one possible explanation is that since the `middleware` function promised to return a fucntion with input: (`ctx: KoaContex` and `next: ()=> Promise<void>`) and what you are returning is percicly what the `Promise` was, so the typescript behaves accordingly.

#### This is a very nice reference for `koa`: https://medium.com/netscape/mastering-koa-middleware-f0af6d327a69