changed a year ago  
Published Linked with GitHub

Node.js Collaboration Summit, Day 2

9:30-10:30 NodeJS Asynchronous Scheduling

  • Issue: https://github.com/openjs-foundation/summit/issues/392
  • Facilitator: @ronag
  • Matteo: framing conversation, #51156 - this reference example throws an error (if wrapped in a queueMicrotask it doesn't crash)
  • Matteo: this causes issues for end users & for us
  • Robert: this is weird, inconsistant behaivour
  • Robert: added a PR for process.deferTick to try & resolve this problem
  • Robert: undici users can encounter this if body errors the tick before
  • Matteo: nextTick is used often in Node codebase, but it is not good with promises
  • Joyee: nextTick is too fast, add something to event emitter to get around to avoid crashes?
  • Matteo: the user needs the chance to attach a listener but the crash is intentional
  • Robert: ready is called before a handler is attached so it deadlocks, nothing happens
  • Matteo: this is a widespread pattern in our codebase and need an error in nextTick
  • Matteo: this is fine 99% of the time except when a string(?) is returned by a Promise
  • Robert: we could do special handling in event emitter (checking queues, etc)
  • Joyee: can we use AsyncContext to fix?
  • Matteo: calling emit as a sync method, but there is no one on the other side
  • Joyee: is the issue on the provider (eg undici)'s side?
  • Matteo: we found this in undici but it could be in any user code
  • Joyee: could the listener be part of user code and emitter be part of your(?) code
  • Matteo: yes, a library that returns a string after a promise
  • Matteo: the event emitter starts recieving events before it is handled(?) by the user, even though it is sync
  • Matteo: I have no good answer, lets brainstorm
  • Matteo: it would not be like this if we designed node today
  • Ruy: coming from the PoV of a user, why not having two separate APIs (one sync, one async)? Then it could be fixed by documentation
  • Matteo: asking users not to mix is optimistic
  • Joyee: does new code use nextTick today or do they use queueMicrotask now instead?
  • Matteo: yes, the problem also exist on queueMicrotask
  • Matteo: we will run microtasks of promises on next tick, but I removed it and it broke everybody (talking about https://github.com/nodejs/node/issues/51156#issuecomment-1864656761)
  • Marco: can we slowly kill off nextTick(?)
  • Matteo: internal node isn't affected by this
  • Marco: We could keep nextTick as an internal thing
  • Robert: this is a problem in libraries, an async api returning an event emitter
  • Robert: common knowledge is nextTick runs before queueMicrotask, but that is not true. this is not documented by us, online resources is ~wrong.
  • Robert: usually true but not always, depends on execution context
  • Robert: nextTick will slow down calls from native code to JS due to async hooks. anything in the nextTick queue will slow down native->JS
  • Paolo: (?)
  • Matteo: limited amount of microticks before end user code reaches the code
  • Joyee: I think there are some wip web apis for scheduling low priority tasks, could we use those instead of inventing our own process.deferTick
  • Dan: yes, it's the w3c scheduler api (https://github.com/WICG/scheduling-apis)
  • Joyee: requestIdleCallback is similar
  • Dan: that runs when event queue is empty(?)
  • Dan: nvm, this api solves a different problem (running last)(?)
  • Dan: we could invent a secondary event queue after the others but before some?
  • Joyee: use setImmediate like requestIdle, kind of like it? similar concept and cost
  • Matteo: setImmediate is significantly later, we should probably document this
  • Matteo: it adds significant overhead
  • Robert: we could go for setImmediate, but it is an ugly solution, is there nothing better?
  • Robert: not about running last, it is about running everything queued before yourself before yourself (and not run anything queued after yourself before yourself)
  • Dan: it has been requested for the web platform but not granted, as the only solution is another queue after microtask and before event loop
  • Dan: the problem is recursive which is why web platform hasn't tried to implement something like it
  • Matteo: nextTick and queueMicrotask doesn't work, setImmediate works but is slow
  • Paolo: (?)
  • Matteo: we could add a defer event emitter which only pushes events after a setImmediate loop or something like it
  • Joyee: we're talking about setImmediate being slow, is it overhead or timing?
  • Matteo: both. timing means data being processed is being kept alive longer which adds gc and other overhead
  • Matteo: eg a path to serve a file or some async processing, it will resolve in 1ms more and use more resources. less things can be managed by the process at once
  • Matteo: this works forever, consider not(?) a hot path?
  • Paolo: you say that we could try and detect an async context, what could we do to check what tick an event emitter was created and (?). that should solve the problem?
  • Paolo: (?) on tick 1, (?)
  • Paolo: start listening to events the same time the event emitter is created
  • Matteo: that is the pattern we recommend(?)
  • Paolo: not sure why
  • Paolo: if we do this under the hood, like using setImmediate, would it be noticable?
  • Matteo: this is experienced by 3rd parties mostly (module authors), sitting on top of node
  • Paolo: if we do tick detection, will the end user notice or not?
  • Joyee: I think I've seen failures triggered by commonly used (?) failing
  • Matteo: it is a race condition so can happen whenever
  • Joyee: can we use asynccontext/async hooks?
  • Matteo: that broke everybody
  • Joyee: when you add the event emitter, do that in the before hook. emit in the after hook?
  • Joyee: place this where user callbacks are called.
  • Joyee: can we move (?)
  • Matteo: I tried
  • Joyee: (?)
  • Joyee: haven't looked much but since we are brainstorming
  • Joyee: add event emitter before emitBefore in processTicksAndRejections (?)
  • Matteo: it won't work. the handler for the event handler(?) is added in microtask, which is inside the promise change for problem cases
  • Paolo: if we defer (?) after will it fix?
  • Matteo: no, we break everyone
  • Matteo: another option is we document the problem and tell people to emit events with setImmediate for event emitters returned in a promise/async function
  • Matteo: it is an option but this should probably be a last resort
  • Robert: we could keep it as is and people implementing event emitters don't throw on caught exception/error, just don't error or something
  • Robert: when you add a handler emit
  • Matteo: maybe under an option or something
  • Wes: can we allow module authors to have an option by shipping lint rules?
  • Joyee: a warning is easier but many codebases (user/library) could make it tricky
  • Matteo: I am not necessarily convinced
  • Paolo: can we detect if not in the same tick?
  • Matteo: the problem is identical, yes and no. handler exception can be handled later on, same problem
  • Paolo: if we can detect at runtime, can we log a warning. we already do this if adding more than 10 event listeners for one event(?)
  • Matteo: we already give a warning by throwing an error
  • Paolo: yes but we are missing a reason to give the end user
  • Matteo: I don't think we can catch it. someone is emitting an error/event at a specific time. the end user wanted to add something to the end of the event. but the execution is not going back to them.
  • Wes: I think a lighter touch will move the needle more, if we can not recommend and ship a linter rule in airbnb/standard/etc, it can move the needle a lot by educating people. people will look up the lint warning and see documentation saying not to do this pattern.
  • Luke: I doubt lint detection will be helpful and we needed a full parser (eslint/etc) to be useful
  • Paolo: I think lint detection is impossible
  • Wes: let's not try to get it right, just say mixing event emitters and promises is bad. the lint rule is avoid this generally. add a docs page saying avoid it, or use a microtask(?)
  • Antoine: The thing is you might be mixing without knowing, because e.g. you're calling from your async function some library code that uses event emitters (or vice-versa)
  • Luke: the ts eslint plugin has full-ish flow control analysis, it could be possible to detect it with this ability? how many top packages use ts?
  • Matteo: I don't think we can do it. demos
  • Matteo: this will get you an exeception, real use case
  • Paolo: the very first moment can trigger an unhandled exeception
  • Matteo: this is the problem. how are we supposed to detect this?
  • Paolo: since only the error event is a problem. can we defer the error event to the next tick like nothing happened?
  • Joyee: add a property to error object to make it emit later?
  • Paolo: that was my other proposal but (?)
  • Matteo: we could document this as generally problematic. we could add a function like emitDeferredEvent(eventListener, name, value)? demos
  • Robert: this will break if already a listener internally
  • Matteo: yes but different problem. this will solve the user problem
  • Paolo: who would use this new function?
  • Matteo: we could document(?) and use this everywhere in our codebase
  • Joyee: this solution doesn't look too bad
  • Paolo: eventEmitter.emitDeferred? might be a good idea. if we add new methods for other reasons they would notice(?)
  • Paolo: we have this new method for this use case, might this accidentally encourage people to use this unneeded?
  • Matteo: don't know, but this is a solution
  • Marco: how often does this issue occur?
  • Qard: can we buffer events until the first listener is attached and flush the buffered events?
  • Matteo: that won't work, it could be nested in the microtask queue. you return an event listener with a listener method, it could be 2/3 levels deep in the queue.
  • Wes: why does that not work?
  • Qard: What if you create an event emitter that buffers events and then give it to the user with a thenable that stops the buffering when then is called?
  • Paolo: at some point you have a queue where nothing will happen until the nextTick(?)
  • Paolo: we can add something like that (emitDeferred?) and call it a day
  • Robert: it is hard to tell how often this problem is as it is encountered irregularly and not easy to know this is the cause
  • Paolo: could we document this problem and not give a solution instead?
  • Matteo: probably a bad idea, what about our own code?
  • Matteo: fs.createReadableStream, do we use this new deferred event method?
  • Matteo: this is a good question
  • Paolo: if we change our code to use it, it will be easier to wait for other ticks internally
  • Paolo: can the event emitter wait until the next tick to throw any error?
  • Paolo: if we document it as a user problem, our codebase needs no change(?)
  • Matteo: if we don't change our code, the end user will have a bug
  • Paolo: but we will have to change every event emitter in our codebase. defacto changing the event emitter is harder than just changing how we process the queue
  • Matteo: no because we do not want to change how user code runs
  • Matteo: it is hard that is why we scheduled a session
  • Paolo: I would rather break user code, sorry(?)
  • Matteo: all solutions have trade offs, no single good solution
  • Paolo: we add this method and runtime detection to detect the problem. defer the throwing of the uncaught exception to the next tick. check if next tick we have a listener. warn that you added a listener at the wrong time.
  • Paolo: add the method publicly too?
  • Matteo: either you change the scheduling - not good breaks everyone. or add a new api for deferring in a slightly better way, but change all of node core. or document it and yolo.
  • Matteo: to start we can document not to do this pattern
  • Matteo: we could also add the new defer method

10:30-11:00 Improving native memory management & diagnostics

intro

  • Joyee: propose this session to discuss native memory
  • Joyee: dynamic import memory problem, many users get a bug
  • Joyee: happened to find a fix/workaround, hoping to spread the knowledge around so more people can try to help with bugs like this
  • Joyee: discuss oilpan migration which can help avoid these types of bugs
  • Joyee: brainstorm improving memory diagnostics

current system

  • Joyee: internally many js land objects are created as wrappers around internal c++ objects
  • Joyee: most have wrap suffix in classes
  • Joyee: almost all are migrated, most inherit from baseobject (baseobject.h/baseobject.cc)
  • Joyee: each node thread (main/worker) has a node environment (src/env.cc)
  • Joyee: each environment has a principal realm and shadow realms (soon). but we are mostly discussing the principal realm
  • Joyee: the principal realm is a v8 execution context, most of the time there is the single principal realm context
  • Joyee: base objects belong to the realm they are created in, trapped in a cleanup queue
  • Joyee: to book keep, we add cleanup code adding to a set. when the realm goes down, it iterates through the queue and invokes destructors of those remaining (base)objects
  • Joyee: eg when a worker thread goes down or main thread when node is about to exit
  • Joyee: many base classes inherit baseobject, like handlewrap: ref-counted c++ objects, usually by libuv callbacks
  • Joyee: when ref count is 0, node notifies v8 to delete the js wrapper
  • Joyee: eg wrapper for tcp connections (tcpwrap)
  • Joyee: filehandle(/etc) is kept alive by js not libuv. memory management for these depends on garbage collector
  • Joyee: v8 notifies node which destructs the object
  • Joyee: bindingdata is only cleaned when the realm goes down
  • Joyee: eg socket class creates a tcpwrap, a reference is added back to the socket
  • Joyee: using a heap snapshot, you can see the gc edges in the devtools
  • Joyee: the js wrapper points to the tcpwrap, the back reference is a cyclic edge
  • Joyee: when the wrapper is kept alive by js, we use a MakeWeak method
  • Joyee: the make weak object makes the handle to the js object weak, so the c++ object no longer keeps the js object alive
  • Joyee: in the weak callback when the js object dies, we delete the c++ side
  • Joyee: they are otherwise kept alive in the cleanup queue
  • Joyee: differentiate what is being kept alive by viewing c++ code and seeing methods used
  • Joyee: many baseobjects have additional js references, some using global handles and weak callbacks. we suggest not doing that
  • Joyee: do no critical cleanup in SetWeak callbacks as it has no guarentee of being called. but it works so we rely on it
  • Joyee: v8 does not always understand our callbacks, it is easy to run into memory issues like previously discussed bug
  • Joyee: some bugs only repro in a repl due to the repl exclusively using contextifyscript
  • Joyee: contextifyscript calls makeweak to make it weak, but it doesn't setup enough in the js land to keep it alive so it gcs too early and node crashes
  • Joyee: create heap snapshot, by pushing objects into an array gc won't call incorrectly so we can analyse the snapshot without a crash
  • Joyee: accidentally created a memory leak by keeping it alive in js land
  • Joyee: we have no good way to tell v8 to clean global handles up
  • Joyee: v8 can't see our c++ side so will mess up gc
  • Joyee: the fix is a new upstream internal field pointing to a script
  • Joyee: we don't use global handles so the actual reference is an internal field which v8 can see and understand, so it gcs correctly

oilpan

  • Joyee: use v8's c++ gc library, oilpan, as it can create c++<->js references which v8 knows how to track

  • Joyee: there are patches which bootstrap this already

  • Joyee: starting migration plan to make use-after-frees less common

  • Joyee: oilpan removes the divide of v8 heap vs node heap

  • Joyee: v8 understands previous case study fine now, no need for annotation of v8 objects for it to understand

  • Joyee: accidental performance bump for crypto.createHash()

  • Joyee: entire hashing operation is up to 10% faster, unexpected

  • Joyee: some blockers due to external memory tracking in heap snapshots and small regressions in some apis which allocate large buffers. probably due to previously underreporting, investigating

  • Joyee: wip upstream v8 patch to support tracing v8::Data

  • Robert: will oilpan end up in n-api too?

  • Joyee: if we do not use it internally, it is difficult to see if it actually works

  • Joyee: requires a trace-based c++ gc library on user end to hook into it

  • Joyee: n-api is supposed to be engine-agnostic and not all js engines may provide a gc api like oilpan

  • Joyee: better use it first internally before any pattern possibly exposed to n-api to make sure they work together

  • Joyee: we already have a helper in node.h to create a reference which works with node objects creating a reference to a js wrapper from a c++ thing, because it needs a specific layout to work in the same heap

  • Wes: does anyone else know this?

  • Joyee: @legendecas (Chengzhong) as well, and addaleax

11:00-11:15 Coffee break

11:15-12:15 Node.js Release group meeting

  • Issue: https://github.com/openjs-foundation/summit/issues/389
  • Facilitator: @ruyadorno
  • Ruy: introducing session & agenda, are looking to get more people involved in release WG
  • Ruy: meetings typically dive into scheduling release
  • Marco: we cut security releases yesterday
  • Ruy: node 21 won't have any more regular releases (outside of security release)
  • Rut: likely next release is for node 22 (Marco is owning)
  • Marco: April 23rd, node 22 is scheduled to release
  • Ruy: node 20 release line needs volunteers for a release in April (around the same time as the node 22)
  • Richard: looking for the right reviewer for v8 PR which makes a bug fix as well as mac-specific build issue (Apple likely changed something in xcode)
  • Richard: have been discussing changes in
  • Richard: we are supporting node 18 until next year (April 2025) - as node 16 had an extended EOL schedule
  • Richard: the policy is changed to have one active maintence line because it was challenging to have 3 active/maintened release lines
  • Ruy: want to discuss difficulties around making releases happen/scheduling
  • Richard: we make regular calls for volunteers for releases & have periods of time where we have regular cadences
  • Richard: current releases are roughly out every 2 weeks
  • Richard: active maintence releases are roughly monthly
  • Richard: maintence releases are best-effort/when possible
  • Richard: there has been issues maintaining this schedule
  • Richard: there is an expectation that once something lands in main/current it will land in LTS imminently (although there's a 2 week waiting policy)
  • Richard: people seem to have less time these days to contribute
  • Richard: there's frustration with the CI & chasing down problems in the build/systems
  • Richard: the time commitment is a bit unkown based on debugging these problems vs. reviewing if the changes are what is breaking builds/CI
  • Richard: looking for people to volunteer more to help keep up with the cadence
  • Richard: the longer in between releases the harder it gets to port changes
  • Richard: last release he did had over ~400 commits to land in LTS
  • Richard: there's good documentation & scripts in place to help alleviate the process of pulling in these changes
  • Richard: but this still requires manual review/processing & human thought
  • Wes: reviewed the release repo & wasn't immediately aware of what someone would need to know or do to participate
  • Wes: might be good to add some bullet points to make it easy to understand who would be right to get involved
  • Richard: we should do that
  • Richard: don't want to limit the release WG to releases
  • Richard: releasers also help others to train up/pair on releases, can just prepare a release (a PR to release branch) and a releaser does the actual release
  • Richard: we want to encourage participation without the requirement of doing releases
  • Richard: other issues: pull requests aren't labelled correctly, something was labelled incorrectly, at scale the automation doesn't catch human error in labelling
  • Joyee: we should apply labels no matter what (ex. the bot should add the CI label & semver evaluation label)
  • Joyee: you should be able to skip any PR which doesn't have that label (ie. this would help skipping unevaluated PRs)
  • Ruy: there is a lot of unwritten rules & expectations on the group
  • Ruy: a releser is a very important role in the project & we want/need people we can rely on (ex. tenored collaborators, professional engineers)
  • Wes: "there is no specific … qualifications to participate" is written into the working group's documentaton
  • Ruy: but there should be a higher standard for these people/roles
  • Richard: there is something written down about this (ie. a policy we have, although it's a bit hidden)
  • Richard: you don't have to be technical to help with releases, but we do require a high degree of trust with individuals who participate
  • Luke: 1. on testing, how is CITGM involved in or associated with the release WG?
  • Luke: 2. is it possible for this to be less flakey?
  • Richard: modules in CITGM must be able to be tested with something like npm test
  • Richard: afaik, as long as they can remember, CITGM has never been green
  • Richard: CITGM won't tell you if your release is good, but it may tell you that your release is bad
  • Richard: helps to flag certain breakages in key ecosystem packages
  • Richard: generally, CITGM fails often - so you want to identify if CITGM is failing in a new way as a releaser
  • Richard: all the modules in CITGM have different test tooling & strategies so it's hard to parse their success/failures
  • Richard: there was a recent purge of modules that consistantly were failing
  • Richard: consistantly failing & skipping checks should be an indicator to remove that module
  • Richard: CITGM is a sniff test, generally it tracks popular packages & often/try to consume a variety of node APIs
  • Richard: the test suties can take awhile to run, so pruning is helpful in theis regard as well
  • Richard: we'd be open to changing CITGM but it has been valuable
  • Luke: I'm a release nerd 🤓 & appreciate what CITGM is/does
  • Luke: is there any tooling or patterns to look at old vs. new runs & detect errors
  • Richard: there isn't tooling today in CITGM to tell the difference between the failures in different runs
  • Ruy: wants to expand on the recent policy change…
  • Ruy: ~1yr ago had someone jump into the calls & self-nominated to become a releaser
  • Ruy: the group believes it's important for participants to be involved with the project first & codified that self-nomination wasn't a valid way to become part of the WG
  • Joyee: noticed that some of the platforms are flakey on specific platforms, wondering if we can configure CITGM to skip specific envs/archs for that test suit
  • Richard: CITGM does have this capability today
  • Richard: the suite of modules is a moving target because those packages are always updating themselves as well
  • Richard: there has been some work to remove flakiness
  • Jakob: can we ask modules to use a specific test runner?
  • Richard: one of the goals of CITGM was to sniff-test the ecosystem as it is (without being permissive about which tools packages have to use)
  • Ruy: want to keep the bar low for module authors
  • Joyee: some of the modules tested are testing frameworks themselves (ex. jest, mocha)
  • Jakob: what about the output/reporter - could we not ask them to provide that (ie. it should just be a flag)
  • Joyee: npm run test-tap..?
  • Jakob: yes
  • Ruy: we could try things like setting specific env vars for the different test runners and try to consolidate TAP output
  • Wes: a lot of companies are using selective test running platforms, we might be able to ask for an open source plan/license
  • Richard: worth trying, everything requires time though & so it's really a people problem
  • Richard: there's two things here, identify problems & then when you have, what do we do about it
  • Wes: these tools often report a confidence rating (ie. to help navigate flakey tests)
  • Marco: have been working on change that would help track the number of times a test fails & to see if it's flakey (looking for help/collaborators)
  • Ruy: you don't need to join the Release Team but within the Release WG there's also a CITGM Team that people can join to help out with CITGM - please come join/help out
  • Wes: people should care & be interested in getting their tools into CITGM (as there's real value)
  • Ruy: when we tried to prune projects out of CITGM maintainers often came back & did the work to fix their tests
  • Richard: adding projects does add time, so we're mindful about which projects we include
  • Wes: could we ask these projects to make a "CITGM" test(?)
  • Richard: not too bothered if it's called "CITGM"
  • Richard: the tool already does a lookup for the test defined (ie. "test" script defined)
  • Richard: this does require someone on our side to reach out when there's issues still
  • Matteo: something happened last year made a lot of the modules to start to be flakey (likely because we fixed a bug 🤣)
  • Matteo: the change/issue for flakiness was/is not obvious & nobody noticed (spent a week trying to debug themselves, for instance https://github.com/nodejs/node/pull/51290)
  • Matteo: the tldr; we need to try to prevent these kinds of issues from happening again/catching those problems
  • Richard: part of Rafeal's push to purge some modules, was because we want to get CITGM in a place where it's consistently passing
  • Richard: if we get to that state we can then start to track when modules are failing in new ways
  • Richard: even if it's just one platform which is consistently passing
  • Richard: there is failure fatigue
  • Matteo: how broken is it?
  • Marco: everything
  • Ruy: human problem… we have 10-20 problems & you have to compare them
  • Matteo: seems impossible to maintain & get green - some of the platforms are odd
  • Joyee: sometimes you don't even care about the platform
  • Matteo: undici's tests break everytime node changes something (which is likely good, because we're catching something)
  • Matteo: have set up automation to test undici's own test suit against nightly node builds & open issues when there's failures
  • Matteo: would be nice to automate issues opening when there's a failure to the targeted module's repo (would help with maintence/participation)
  • Ruy: want to encourage everyone to contribute back/participation (lots of good ideas)

12:15-13:45 Lunch break

13:45-15:00 Node.js Next-10 meeting

  • Issue: https://github.com/openjs-foundation/summit/issues/401
  • Facilitator: @sheplu
  • Jean gives an introduction about the next-10 initiative
    …
  • Michael: …looking for trusted parties to document…
  • Matteo: Please sync up with the OpenJS Foundation Kylie & Robin on the Speakers Boureau initiative. This should provide funding to approved speakers to promote our projects around the globe.
  • Matteo: Some YouTube influencers are teaching wrong things. I am worried. We should look at the content.
  • Wes: We are identifying the same problem. Even if we do invite only it would be difficult.
  • Michael: we don't have enough bandwidth and people to review all of them. Maybe we should have trust-worthy reviewers.
  • Wes: maybe it does not have to be a blocking thing but just have some people to proof-read which may help with our relationship building too. Like if there are security gotchas they can ask for help to review.
  • Matteo: if we want to promote content through our channels, I think that should be better
  • Michael: trying to find a group that can review the content all the time would be a challenge
  • Matteo: we used to do that with Node.js collections.
  • Matteo: for example, if we want to publish a video on our youtube channel but no body looks at it?
  • Micheal: the idea is to publish them in the ambassador's own channels. We can tweet about them with a light endorsement but don't make it official content.
  • Jean: I agree with what Michael. Problem is like what we said before, if you search for Node.js content the top results are usually written many years ago.
  • Michael: it's also an opportunity to build a group of ambassadors that we can reach out to. It's better than having group of people doing their own thing, some get tweeted, some don't. It does take level of trust.
  • Owen: Are there good resources out there that a do reflect the code quality / conventions that can be used as a reference? Like a styleguide?
  • Wes: there is definitely that kind of content. And probably got promoted before if they are written by Node contributors. But we need more. I think we should spend more time promoting those than content that are more bloggy. The goal of this is to have people to do this. Forming the opinions that matter, not just writing about what existing options are out there. We are trying to promote intro-level content or deep-dive. Not necessarily a style guide
  • Matteo: Do you know that the foundation has official training?
  • (most of the room raised hands)
  • Wes: how many of us are certified
  • (Only Jean had it, because it was free, he got it twice).
  • Matteo: because it makes no sense for us to be certified.
  • Wes: I don't have it because I have no idea what it covers.
  • Matteo: how many people in the session was involved when the training was created
  • Ruy was involved. Joyee was asked, did not have time to do it.
  • Matteo: …it was trapped at the C-levels
  • Matteo: I think we should try not to teach outdated practices in existing materials. The best way to promote something is to put something on our website.
  • Matteo: we should get collaborators to look at them(?)
  • Matteo: there's money flowing around this. There was historically some distrust in this program. That was not very helpful. I looked at it so I cannot be certified by it.
  • Matteo: That can be fit into the ambassador program. If anyone is intersted I can give a code for you to check it out. I don't think it's something that serves the project in the right way at this point.
  • Jean: The last time the ceritifcation was updated was 2019, not sure but I think it was not updated to ESM
  • Wes: It was updated to ESM
  • Jean: A lot of times on the Internet you may find something that's not helpful or outdated.
  • Michael: there are some companies paying people to write the right content. As a project we don't have people like this. I am wondering what we can do to encourage people to write content. One thing is trying to encourage people to write. Another is to make sure they are trust worthy. The ambassador program is more about the former.
  • Wes: we should reach out to the foundation certification thing and make sure it's better promoted. Like promiting it on the landing page. But it's separate from what Micheal said.
  • Wes: another thing is some people get the status from the project itself, and the governance is shrewed towards that. Trying to have a ambasaddor program to have people doing talks and writing content and increase our outreach. And have them contribute back to the project.
  • Wes: does anyone other than Matteo do this kind of content productions?
  • Jakob raised hands
  • Marco: another point on the agenda. The survey. Important to figure out what should go into the survey.
  • Jean: last time most respondants were from Europe. Bit strange. So we added (are adding?) questions about countries before technical questions
  • Michael: (…didn't catch)
  • Raz: how about new features? Like the test-runners? Getting feedback is difficult.
  • Michael: so questions like "are you using the test running? Any feedback"?
  • Marco: (…) before people get bored and just drop (the survey)
  • Michael: there was a bunch of …
  • Raz: the official twitter account can get more opinions from the community. for any major thing that we add for example.
  • Joyee: maybe not a very helpful suggestion: corepack
  • Jakob: for example parseArgs()
  • Wes: regarding corepack, we can take the goals we didn't reach agreement with yesterday and ask them in the survey. Do we have a deadline for the survey
  • Marco: 24, 23 of this month
  • Michael: so we maybe need to prepare it in a week? maybe a bit too fast?
  • Marco: the plan is to have a final decision about the survey at the next next-10 meeting, 17 April
  • Michael: we did wonder about package managers (did ask about version managers last year?) adding this to the survey this year should give more information to help us
  • Wes: rephrase the question as: do you think the node project take the opinion of version managers?
  • Wes: maybe the more contentious issues we were talking about yesterday can use help with input of a broader audience
  • Michael asked about a PR
  • Marco says that there is a PR in the next-10 repo (about the survey: https://github.com/nodejs/next-10/pull/261)
  • Michael: that's the best place to make suggestions about the survey
  • Jean: maybe we should discuss about sponsorship & support. How can we bring more company supporting & sponsoring Node.js
  • Dan: I suspect if the Node project has a way to accept money to give to the engineers there would be
  • Jean: is there a way to give money to Node.js project?
  • Michael: there is a bug bounty project (?)
  • Joyee: The foundation is not the way to sponsor development in core.
  • Wes: how do I sponsor a feature?
  • Matteo: I don't there is a queue of companies trying to sponsor the Node.js foundation or the project
  • Matteo: I do think there is a big lack of "how do I get a bug fixed in Node.js"
  • Wes: I agree that member companies don't have this misconception (that foundation is not the way to sponsor engineering work). We can provide better outreach to let people know about the foundation. People reporting the bug almost never have the access to the checkbook.
  • Joyee: My impression is that people outside the project think that there is a node team and there is a foundation, so the team is supposed to be fundded by the foundation..right? But that's not how it works. I got this impression from fixing bugs I mentioned this morning and things like require(esm). I think if we do better telling people that's not how things work people may be more motivated to sponsor engineering work or know where to look.
  • Matteo: yes some people think node is maintained by Santa. we need to change with the problem right now: how to communicate correctly. (?) platform provide a way to sponsor maintainers
  • Dan: I think it would be helpful for sponsors to have methods to get very specific goals sponsored.
  • Jean: conclusion:
    • we should improve training and documentation.
    • Please jump into the survey PR.
    • Funding. Running out of time. Come to the next 10 meeting to talk about.
  • Robert: if there is a place to see what proposal they can fund it would be very helpful.

15:00-15:15 Coffee break

15:15-16:00 JavaScript modules in TC39 and Node: past, present and future

  • Issue: https://github.com/openjs-foundation/summit/issues/397
  • Facilitator: @littledan & @guybedford
  • Dan & Guy give an intro of ESM.
  • Guy: during the early phase of ESM one of the goals is interop with CJS. At some point it got derailed into supporting user land module loaders to support require(esm). And then it ended up cutting concerns about loaders out and just pushing the CJS interop to user land.
  • Guy: there was a concern about aligning the semantics. The main difference between CJS and ESM is that ESM decoples exeuction from static analysis of module requests.
  • Michael: for the past collab summit what was useful was to find top-of-the mind issues and discuss solutions
  • Dan introduces import defer (https://github.com/tc39/proposal-defer-import-eval) and module {} expression (https://github.com/tc39/proposal-module-expressions)
    • import defer only evaluates if it's accessed. Nicolo is developing benchmarks, help if you are interested
  • Wes: asked about crashing during evaluation
  • Dan: good point
  • Ruy: Asked about mocking
  • Dan: being developed
  • Jakob: no way to legitimately replace a module or unload a module in V8 (?)
  • Dan: let's go through the slides first. Let's not make plans in Node.js that depend on import defer in the short term
  • Joyee: the intent is to replace the way to monkey-patching CJS module. There is no proper way to customize CJS loading, so people go with monkey-patching, over the past 10 years. One idea is some kind of hook to replace the monkey-patching, people can detect if the module hooks exist and avoid monkey-patching, and avoid semver-major change. For example, instanbul and babel depends on a common package called pirates that get 20+M downlaods per week, it we can just update it to use the new hooks in semver patch, all users of them can just upgrade the transitive dependency in semver-patch and it uses the new hooks on newer versions of Node.js and no longer monkey-patch
  • Matteo: the current design has two use cases: 1. monkey patching, 2. doing async work in the hooks. The new hooks are complimentary and not exclusive to each other.
  • Qard: loader is good for typescript transpiling, but for APM there are two different modules for patching module as they are loaded, require-in-the-midle for CJS and import-in-the-middle for ESM. IITM does unspeakable things to attempt to retain the same design as RITM. ESM is immutable so patching directly is not possible. Basically import-in-the-middle hacks around the immutability with a shim module that registers setters for all the exports within module scope where reassigning values is valid. This happens in the loader thread where we can't actually run patch code, we can only modify code text, so we're very limited in what we can do.
  • Dan: import maps
  • Joyee: solution mentioned in the thread: use a synthetic module facade
  • Ruben: module loader is performance sensitive. Monkey patching is significant compatibility burden. Not sure if adding a new API helps
  • Joyee: We can try to focus on pirates first, and migrate the majority of popular packages which are what we care about, and start deprecation.
  • Jacob: We have been trying to figure out what the CJS module does, but it's been very difficult
  • Dan: let's focus on articulating requirements
  • Joyee: It seems to be repeating require(esm), it looks impossible until someone figures out how to make it possible
  • Geoffrey: (?)
  • Joyee: as maintainer we want to know what to avoid, at the moment we don't know if touching Module._* and Module.prototype._* would break prople. Official hooks can help us set up boundaries.
  • Guy: source-phase-import on the way.
  • Geoffrey: we really need some help in the Moduel part of code. If you wondering why the hooks stuff take so long, it is because we need contributors. That bugs there are opened for six months or so, we need some help.
  • Wes: as a user, I only care about replace modules in testing, and wondering why TC39 is not working on this part. feels like some disconnection here
  • Dan: we have a TC39 Module Harmony Working Group and weekly meeting dedicated for that group. Contact @guybedford or nicolo to join the group.
  • Nicolo: ES modules are designed to be synchronous as possible. But there are some assertions in the spec getting in the way of synchronous import defer. V8 maintains the assertion. Need to figure out whether it's actually required or just paranoid.
  • Dan: there are hopes in the loaders/compartment APIs (https://github.com/tc39/proposal-compartments).
  • Nicolo: presenting https://github.com/tc39/proposal-defer-import-eval
  • Dan: we have several issues to think through (e.g. code caching) it would be very useful to have people from Node.js to figure out how to make it performant for Node.js
  • Dan: This reached 2.7 and wants tests would be great to have feedback from Node.js

16:00-17:00 Node.js involvement in standards

  • Issue: https://github.com/openjs-foundation/summit/issues/398
  • Facilitator: @littledan @CanadaHonk
  • Cross-over session with WinterCG: https://github.com/wintercg/admin/issues/64
  • Dan goes through a list of ongoing TC39 proposals
    • AsyncContext
    • Temporal
    • using
    • iterator methods
    • shadowrealm
    • decorators
    • Signals??
    • Type annotations??
    • Records and tuples??
  • CanadaHonk goes through a list of WinterCG proposals
    • Minimum Common API (console, performance, WebAssembly, atob/btoa, crypto, setTimeout, etc)
    • import.meta
    • Runtime keys
    • Fetch adaptations
    • package.json ⇒ LF
    • Future:
      • Ecma TC55/WinterTC
      • Socket API
      • CLI API
      • (CLI.args, CLI.env, …)
  • This is not specification, more like recommendation: https://common-min-api.proposal.wintercg.org/
  • Matteo: there is one thing about the API, (?)
  • Dan: fetch is not going to work the same way as in the web browsers and in the server environments.
  • Robert: what happens if we would like to ship something that can break
  • Dan: Node.js collaborators have been participating in WinterCG to prevent specifying something that is not compabible with Node.js
  • Dan: we are thinking about going to a ECMA TC for new proposals, like the Socket API from Eric and CanadaHonk's CLI API
  • Luca: trying to explain why we are standardizing node incompatible. We don't want WinterCG to work like standardizers v.s implementers. We want to talk to implementers and not be like standardize something that implementors hate.
  • Michael: to date, there is no formal agreement that Node.js is going to be WinterCG compliant. My take is that at least to today, it is important to know if Node should do (it's unclear why it is something important for Node.js contributors to work on)
  • Robert: there's the difference between Node.js users and collaborators.
  • Michael: My understanding is that there are some people who get paid to work on WinterCG. And it feels like people are asking Node.js contributors to work on it before it's too late. It would be useful to bring that investment into Node.js because it's not necessarily the day job of contributors
  • Michael: the idea is that don't show up at the last minute, just make sure that there's ample time
  • Wes: node is a distributed product and not all people are focus on different topic like CLI/parseArgs proposal. It would work better if WinterCG reaches out to people working on each specific area, and get that funded (?)
  • CanadaHonk: for CLI proposal we did reach out to node deno and bun, but I don't think WinterCG can just standardize something that no runtimes has implemented
  • Matteo: Recently we shipped an additional property in the response object and broke the compatibility of other runtimes. If we implement a statndard but we deviate, we'd break others, I'd like to avoid this kind of situations.
  • Dan: I think the goal of WinterCG is making a programmable platform friendly to server and it should end up being unification within node. This is not universally agreed in WinterCG. If some extension is useful for servers it should be standardized. Recently WinterCG specified subset of AsyncLocalStorage that is compatible with node/deno/cloudflare worekrs.
  • Matteo: fetch response, async iterators on the response object (? streams)
  • Matteo: people respresenting node in WinterCG
  • Dan: they are not representing Node.js – wintercg include participants from Node.js. Node.js should have a way to raising positions https://github.com/nodejs/standards-positions set up by @jasnell
  • Robert: it mostly benefits from other runtimes. negative effects include we end up with WinterCG specifies something but we don't want to implement
  • Dan: we have to figure out stable funding part to support Node.js
  • Andreu: It will be good to encourage people to work on WinterCG in distributed (?) to see if they can (?).
  • Dan: Andreu has been working on the details of AsyncContext and CommonMinimumAPI, ultimately I hope WinterCG can make the transition between runtime transparent
  • Luca: I don't work on Node, I work on Deno. I appreciate feedback from Node.js on WinterCG but it can be difficult find the right people. Anything that makes my life easier would be appreciated, like the standard position repo, a mailing list or a slack channel where I can ask for feedback. It is difficult as not everyone got funded to work on Node.js for specification.
  • Dan: there is process like to escalate issue in two weeeks in the standards-positions. the repo can be a good place to reach out to people.
  • Wes: the project already have structure like WG and teams to work with this, not individual users. I imagine the Socket sohould be raised in the Web-Server framework team. I am worried that it is a place for every topic because it can be too noisy for a person to watch a unified repo.
  • Dan: We can triage the issues in the repo and mention the correct team.
  • Wes: sounds great
  • Michael: standards-positions sound like perfect solution but it has not happened yet, just trying to avoid last-minute feedback asking for now.
  • Dan: I personally want to work on this and am happy that James started it.
  • Owen: we worked on web-components across multiple browsers, they created standard-position repos too. happy to share more of that within winter cg.

CycloneDX

  • Dan: knowledge sharing about SBOM format etc.
  • What should be Node's software supply chain security strategy? What do we think about SLSA? npm security.
  • Matteo: security WG has been working on a bunch of security topics. My opinion about SLSA, which is shared by many others in the group is that it's incompatible with our model.
  • Matteo: SLSA treats collaborators (who have commit access to the repo) as an attack vector.
  • Matteo: which is fundamentally opposite to what Node.js does
  • Richard: that requires a change to all our process and is fundamentally incompatible with what we do.
  • Matteo: this is not that Node.js is not secure. We operate in trust, SLSA is a model based on distrust.
  • Michael: It was discussed within the security WG. Not to say that we are against it but it was not raised to everyone. https://github.com/nodejs/security-wg/issues/1255
  • Richard: I don't think there is one dictates what Node.js should do. It is gonna be evolutionary, we cannot easily say to change what we are doing. Node.js is a distributed project. If you ask what Node.js thinks about it, the answer can only be "it's mixed".
  • Wes: …
  • Matteo: It's code before people vs people before code.
  • Wes: I think the response is it's not like Node.js is bad because it doesn't do SLSA. But it's Node.js is good, SLSA is bad. The approach of putting code before people is fundamentally broken. We should fight that. identify trust-worthy people and empower them to do the work.
  • Tobias: it needs a lot of work to implement on all the platform we support and from an infra point of view I don't see it happening anytime soon.
  • Darcy: …
  • Dan: what should we prioritize
  • Darcy: I don't think there's a silver bullet.
  • Ruy: coorporate working power…
  • Michael: need volunteers. permission model, best practices, openssf, automiating security releases, SBOM, audit and improve build processes of Node.js dependencies
  • Darcy: don't want to be blocker, want to encourage people to do what's valuable
  • Matteo: If you are interested in SLSA this is being worked on on the OpenJSF level.
  • Matteo: if Node.js core wanted we could make SLSA work. We can just stop trusting each other. But everyone on npm will always be SLSA incompliant. That decreases the value.
  • Dan: GitHub action thing?
  • Matteo: what about your dependencies
  • Darcy: SLSA is not a model that incldues transitive dependencies.
  • Wes: we've been rolling this out. Just to share some experience. Do you wnat to know about the score of some highly used oss package? 4. There's nothing wrong with this package. But it's giving a really bad score. If we define a scope that makes these packages look bad, users would draw the wrong conclusion. Like if there's a MITM package that just doesn't need any updates. It gets 10 out 100 because of that. It can just autoamtically commits everyday to get a higher score but what's the point?
  • Wes: this is also adding a lot of work in the actions. It's a metrics that add a lot of burden on maintainers. We should think about that when designing the specs. If it requires tons of work from projects to be compatible then the spec is bad and it should be changed.
  • Matteo: fastify passes all the tests. OpenSSF score. fastify got close-to-nothing from SLSA.
  • Wes: if you can not safely automatically publish, that is the main problem. If it requires 2FA…
  • Matteo:
  • Wes:
  • Marco: in security WG
  • Jean: if it's not possible then it's not necessary
  • Darcy: If SBOM cannot be used to reproduce software…not sure if it's accurate
  • Marco: it's used to check in on the supply chain of Node.js. It's more for inventory than for security research.
  • Micheal: it's difficult to get to a perfect answer. It can be 50% more like a a learning process for the project. Maybe SBOM can be useful maybe not. We don't know but in the … group we can figure out
  • Darcy: not sure if it's accurate to describe it as…because we don't trust the tooling because we need the human. Is it like ingredient of your cereal…
  • Marco: we are not going to publicize (SBOM). You don't have to publish 100%. When we had the discussion with the foundation we made it clear that it was something we were exploring and not try to publicize because otherwise people would go to check it which would be counter productive because it is experimental.
  • Darcy: if there's no way to check whether the SBOM is accurate then …if we reframe it, remarket it then I may be more onboard
  • Wes: let's not try to invent new meetings
Select a repo