Node.js collab summit, Nov 8, 2024, Dublin

Documentation

talk about learn section of nodejs.org

  • Stephen starts the session
  • Claudio: talk about website redesign and current state of website
  • Augustin: mention that there nobody really responsible of content. (nothing in codeowner)
  • Jacob: maybe we can have some link on the learn docs to mention all the changes
  • Alexander: we should have a "last update"/"last reviewed" part in the doc to reflect how up-to-date it is
  • Augustin: the link between docs and learn is hard to maintain. because docs is versioned and not learn, so learn can link to docs, but docs can't link to learn because it can become out of date.
  • Brian: talk about translation on learn section (… i didn't follow everithing)
  • Matteo: is in general not in favor of translation because it's can be "unsync" from the orignial content ("en")
  • claudio/brian: answer that we use crowdin. It's allow us to keep translation up to date
  • …
  • Claudio: having more external translations would introduce more misinterpretations out there and it would be difficult for maintainers to keep a tap on them
  • mhdawson: there are two problems, out of date content and lack of ownership in the content.
  • Matteo: who is the persona that we are talking about when we talk about the content? Looking at the table of content in the learn session. We should try to seek volunteers and look at the flow and check whether the basics are introduced in a way that makes sense for begineers(?). There are a lot of deep content like profiling but also missing important content like HTTP. If we are not saying anything about those on our website people are going to look into content on the Internet and then maybe see out of date content.
  • Stephen: do people know about good content creator? Can we reach out?
  • Stephen: we should have a scroll of "things that you should know"
  • Augustin: we should ensure that the learn docs are updated. Also learn section isn't a course section. API docs explain API of node. Learn should guide/example with these api.
  • Claudio: it goes back to not knowing who to ping and how to keep the learn docs updated
  • Alexander: how about pointing to external content from Website
  • Claudio: difficult to verify
  • Joyee: contributors are not even filling release notes for notable changes properly, lets not overestimate people's passion about writing learn content
  • Matteo: we can put links to the learn section in docs. Linux foundation training.
  • …
  • Augustin: we shouldn't add link to paid ressource such as LFX training

Issue which summarises : https://github.com/nodejs/nodejs.org/issues/7197

CJS/ESM interop

https://github.com/openjs-foundation/summit/issues/422

--experimental-default-type

  • Okay to remove?
  • Looks like nobody in the room objects
  • Error message from mixing CJS and ESM syntax can be improved
  • Geoffrey: We can go with marking syntax detection as stable and dropping the default mode switch experiment
  • Benjamin: haven't heard complaints from what we currently enable, so probably safe to remove
  • Joyee: we also need to discuss when to remove it - major or next minor?
  • mhdawson: is there benefit in removing it now?
  • Geoffrey: there is more code complexity in keeping it
  • mhdawson: I am thinking if there's no strong reason to remove it then it's less work for releasers, but if it doesn't harm anything

Syntax detection

  • Okay to mark as stable? (It’s been unflagged since July).

Documenting dual package shipping patterns and updating ecosystem practices: https://github.com/nodejs/package-examples/issues/2

  • What are some projects that might already be hitting require(esm) that we should check for regressions from unflagging
  • What patterns are people using to ship modules
    • Filip: I decided to not support CJS users that run Node.js that don't support require(esm)
    • Filip: There are a bunch of runtimes that require all sorts of conditions and I am happy to drop all of them
    • There will probably still be maintainers that want to support older versions of Node.js
    • It's probably up to the maintainers, but we can provide different guides for different choices
    • Jordan: I can now see a future where I would find it feasible to ship native ESM (only) in all of my packages, as long as there is an easy method for users on older engines to seamlessly adapt/transpile a consistent compile source, babel-register-style.
    • Geoffrey: maybe some install flag to npm to tell npm to transpile to CJS during installation time
    • Matteo: we need better docs, there are still very prominent community members saying you can't import CJS even though it has been there since the beginning.
    • Geoffrey: our docs could be more opinionated, telling people to write ESM and that it can import CommonJS and they should consider CommonJS legacy (but still supported). We would move away from presenting both as equal first-class citizens.
    • Jacob: we have something similar in the loaders example
    • Andrew: google jupyter notebook for testing it
    • Auguest: ask npm init to use type: module by default
    • Geoffrey: it kept getting proposed but npm never actually accepted
    • Wes: package maintainance working group has a npm init pkg thing being worked on: pkgjs/create
    • Alexander: given that we have node –run should we depend on npm for this?
    • Wes: we can have it in yarn etc. as well
    • Geoffrey: we can also have node --init
    • Darcy: that also needs a registry, npm init reach out into the registry
    • Joyee: it has to reach out into a registry for new tools and updated templates
    • Augustin: maybe just don't add other projects and just the boostrap for Node.js. Such as scripts and type keys in package.json. behing –init
    • Alexander: we should reach
    • Wes: we should give the federation to the ecosystem. node –run has different use cases and it was already controversal
    • Darcy: there are also expectations that people have in these tools that cannot/still not be filled by Node.js builtin options, like npm run with workspaces. the most useful part of npm init is to use templates
    • npm init history: https://github.com/nodejs/TSC/issues/1445#issuecomment-1742118965 (there's something in the slack channel last month? too)
    • Darcy: there are a lot of features in init and it's hard to provide them in core
    • Jordan: the only thing that I care about in init is license and version, and licenses bring a ton of SPDX complexity. It should be easy for the ecosystem to develop their own different buckets
    • Augustin: why can we not have esm in internals ?
    • Joyee: several factors: CJS static analysis not working on all existing code especially dynamic exports. Too many dynamic exports and imports. Our internals is not real CJS, it's a mini loader that takes CJS-like input. We need to re-invent a loader that does the same for ESM. Also
  • What else is blocking them from migration to ESM
    • jest: vm modules
    • stephen: observability (for APM), choices is either shipping CJS or diagnostic channels (for database drivers, routing frameworks, caching libraries; not general-purpose libraries or application code)
    • Bryan: other vendors may have different ideas and there generally need to be ways to observe things not in diagnostics channels
    • Wes: there are a bunch of folks bringing up things in bundlers not supported in browsers
    • Matteo: I got reports about CJS output not being able to be bundled and I asked many times for an example that a bundler does not support CJS and never got one. Bundlers started with supporting bundling CJS.
    • Geoffrey: newer build tools like rollup are designed to be ESM first
    • Matteo: vite has a massive doc about how to support CJS and they just need to read the docs. People who complain just don't read the docs.
    • Alexander: most of the times the module system doesn't really matter(?)…
  • What are the practices that are used to migrate
    • Filip: no good tools that I know of
    • Alexander: vscode quick fix
    • Alexander: lint rules for writing ESM that's requirable
    • Jordan: there should be one, but if not, no-restricted-syntax
    • Augustin: LLM, spoiler it's didn't work
    • Bryan: what is the incentive to switch to ESM
    • Geoffrey: easier for the module to be run in browsers, better tree shaking
    • Augustin: easier to learn when you come from browsers.
    • Matteo: if you author JS, and only for Node.js, none. If you write TypeScript and run it on Node.js and it's pain. It's TypeScript in ESM syntax targetting CJS output that's the problem. It's more about the types. It's so complex and I am not even a beginner.
    • Geoffrey: there is benefit in using ESM if you write TypeScript
    • Filip: module resolution, .ts -> .js rewrite. Before require(esm) the only incentive is ERR_REQUIRE_ESM, now it's none
    • Geoffrey: Marco has a PR to TS for flags for reslution that are compatible with Node.js
    • Alexander: we have a template to dual publish TypeScript. It would be good to document the subset that's compatible in ESM/CJS: no top-level await, not dynamic export
  • Matteo: summary: more diagnostics channels/obvservability support, dual publishing for TypeScript, doc etc. for a path/story. Lint rules and auto fix.
  • What kind of content should we put out there, where…
    • Linters
    • API doc
    • Learn docs
    • Influencers, ambassadors, write tutorials, have them seek out expert review, would have a bigger impact
    • Wes: don't be afraid to stand up and tell them they are wrong, DM them, they don't have bad intention, they might just not know
    • Matteo: this doesn't work very well. We can ship the best software but the world doesn't know about it if we don't tell them

Missing pieces of ESM

  • Performance
    • Geoffrey: we have a WIP to remove async handling, it's also failing tests due to unexpected async activity when we try to use the async ESM loader as default
    • Jacob: it's currently impossible for ESM to be faster CJS
    • Joyee: it'll be possible when import defer eval is shpped in v8
    • Andrew: I also noticed in our runtime (just-js) that sync is a lot faster, async is the problem, not ESM
    • Alexander: comes back to documenting the fast path
    • Jacob: don't duplicate paths because jumping back and forth is not great
    • Matteo: can be a list of good first issues to remove async
    • Joyee: we already have the duplicate path for require(esm)
    • Geoffrey: long term we should syncify the ESM loader, ship sync hooks, deprecate CJS monkey patching, merge together the loaders
  • Snapshot & SEA support
  • ?

Synchronous, in-thread & universal module loader cutomization hooks: https://github.com/nodejs/loaders/pull/198

  • Joyee summarizes the new proposal
  • Matteo: the initial motivation to make it async is to support e.g. network loading. I have been experimenting making async operations synchronous, similar to what we do in core. abstracted it into a package everysync: https://www.npmjs.com/package/everysync. We need to document an escape hatch for people using the async hooks, to use this instead
  • Jacob: yes, the plan before removing the async hooks is to write a migration guide including everysync (or similar). The design proposal is already in progress.
  • Joyee: there’s work in V8 to create lighter-weight threads with isolates, that should land soon. these faster threads should make the 'de-async via threads' approach more viable
  • What are the things not supported in module.register() that we can use the new API to fix
  • Is sync resolve + sync load + startGraph enough
  • deasync for certain workloads

File-based preload configurations (noderc): https://github.com/nodejs/loaders/pull/225

Facilitate migration to new features and breaking changes

  • Jacob demostrante the ts correct specifier codemod https://github.com/JakobJingleheimer/correct-ts-specifiers
  • ?: can also do the modification on the module loader level
  • mhdawson: (..some concerns about mismatched node.js versions, hopefully Zoom captured it)
  • Luke: codemod but also for dependencies. We have codemod on bundles. I think at one point jquery did it. List things in your dependencies but not fix them. I would caution about loader.
  • Alexander: Google etc. has something similar that converts dependencies statically. I agree with that approach
  • Jacob: can also do it in a post-install script.
  • Thomas: dependabot
  • Darcy: people may not necesarily want to run some codemod they find on the internet. Improved dependabot integration would be easier to adopt
  • Jacob: we can have blessed ones for the dependabot, marked as verified or official
  • Darcy: it could be a lot of compute, would be good to have a registry for projects that have already migrated, avoid waste of compute to convert projects that have already been migrated
  • Jacob: sounds like something a registry can help with ;)
  • Darcy: we (vlt) are happy to help
  • Joyee: projects can have a local file like codemod_x: version mappings to skip running again on code already migrated
  • Jacob: also not all codemods need to be run, only for the Node.js versions you need to support
  • Geoffrey: this effort would be a great use for a tracking issue in the node repo that we tag with 'good first issue', with a list of codemods that we wish would exist. this would be an especially great first issue for a newcomer as these don’t require learning how to build the node codebase
  • mhdawson: put it into a list
  • Jacob: We can have a tracking issue for breaking changes that do not yet have a codemod
  • Alexander: we have a single Node.js version policy in Bloomberg. Could use some formalization, structured machine represenation of features/things that will be gone in the next major version.
  • Jacob: don't want to discourage the contributions…
  • Matteo: generically it's something we try to follow, we never break something without providing alternatives. Over the years we have some behavior changes e.g. ordering of events that can only provide an alternative in the form of a flag. It can be difficult sometimes.
  • Jacob: edge cases can be hard.
  • Alexander: can have strategically important cases where it should not be modified (?)
  • James: workers have no compatibility breaks so piling up e.g. two url/string implementations. The hardest part is the behavior changes that cannot be codemod-ed. If codemod patch internals it won't work on other runtimes.
  • Jacob: I don't think we would patch internals unless they are doing naughty things?
  • James: there are a lot of code out there doing naughty things

Diagnostics WG

  • Stephen presenting some slides about async context
  • … https://github.com/tc39/proposal-async-context/blob/master/CONTINUATION.md
  • Stephen: it currently doesn't work very well with concurrency.
  • Chengzhong: cache behavior…haven't come to conclusion
  • Stephen: diagnotics docs don't exist in learn section
  • Augustin: it's from guide section … so not new content (maybe need an update ?)
  • Future of import-in-the-middle
  • Bryan: critical package for APM vendors, gives ability to manipulate ESM modules, providing a shim to the exports
  • Bryan: too many edge cases that it cannot support. In particular when the module modifies its exports, instrumentation doesn't work on those. Currently we at datadog just modify the code in the hooks. Full of perf implications but it's the best we have.
  • Matteo: what's the timeline for supporting those things. Is it ever possible to support them? Tying into the discussions about ESM migration. This is probably the biggest blocker. Especially for the database packages. We need to sit down, plan it, document packages that are broken. Send messages via diagnostics channels if possible. Provide a path for APM vendors. Lots of work.
  • Bryan: it's also worth mentioning that diagnostics channels are great but we still need some monkey patching abilities
  • Joyee: useful to list out the kind of pacakges that would be affected (db, sdk..) and what can be done (diagnostics channels, or making exports const) so it would be easier to get help from e18e folks
  • Jordan: compartment proposal may provide hooks for this use case
  • Simon: abort control is possible with diagnostics channels, not possible with tracing channels. shows doc
  • Simon: is monkey patching something that will be allowed
  • Joyee: the ability to patch sources will always be there for transpilers. But relying on it to patch functionalities is really up to the hook authors' discretion. It will always be brittle as the package code is out of our control. The more reliable approach would still come down to having the ECMAScript specification allowing some kind of mutability or refelction of modules.
  • Thomas: trying to get live debugging, currently doing inspector protocol, lots of overhead, trying to collaborate with v8, collaboration with Google on this part..
  • Alexander: can we transactional memory
  • Thomas: many ideas, thread pause shorter..intelligently copy memory for data processing, copy on write…
  • Andrew: is it something that'll be available on all platforms or just Linux
  • Thomas: preferrably all platforms, currently we use go + eBPF at datadog…Opentracing from elastic…
  • Bryan: how much of import-in-the-middle is just implementation details?
  • Stephen: expose the C++ interface of the V8 CPU profiler and serialize them into buffers.

Tooling group session

  • wes: present the pkgjs
  • Replace close button platform with bluesky
  • mhdawson: there's still quite a lot of followers on the close button platform
  • Wes: yeah but the engagement is better on Bluesky. Can start with cross posting too
  • Someone: it's already WIP!
  • Tierney: ICYMI, lots of developers are migrating from close button platform to bluesky
  • Wes: also bluesky is open source
  • Augustin: why not mastodon
  • Jacob: bluesky has automation
  • Augustin: what do the foundation use to post
  • Jacob: hubspot, doesn't support bluesky yet
  • Joyee: why can we not just share it with onepassword
  • James: we need to involve the OpenJS foundation
  • Wes: should be taken to CPC
  • mhdawson: we delegated it to the foundation to post social. we don't need to ask to take it back.
  • Joyee: a lot of times it's us asking them to post on the slack channel and they do it because we don't have the password. And wait for many hours because of timezones. It doesn't make sense.
  • mhdwason: just propose that
  • How to handle Experimental status when adoption is significant (nodejs/TSC#1397)
  • Stephen: even though module.register is experimental, it's used by all APM vendors, cannot really be broken
  • James: having experiments without a timeline in advance is critical in their decision in using it or not
  • Jacob: give warning before major changes happen
  • Stephen: when we ship experiments that means we want people to experiment, the expectation is that it may not always work like this. There should be a tracking issue in the warning message for them to follow.
  • James: setting expectations, having clear documentations about status and timeline of upcoming changes
  • mhdawson: what do we want to optimize, adoption or no adoption
  • Joyee: mentions https://github.com/nodejs/node/issues/55505 - we can have code assigned to experiments and a page for the experiments, similar to what we have for deprecations
  • mhdawson: we should agree the model that we want to take about adoption
  • James: we meant to add codes to experiments, just never ended up adding them
  • James: people actually don't care about whether it's stable - you put it out there, people are going to use it
  • Joyee: more granuarity - can I use it in code I maintain in the forseeable future v.s. can I use it in code that I probably won't maintain forever
  • Joyee: better notification channels for the former
  • James: similar to LTS, need to be able to plan for it
  • Wes: 1 year etc. is fine, in the ecosystems there are dependencies that you cannot plan around with, sometimes there might not be viable options, users can only rip out the tool e.g. Turborepo
  • Wes: we need to be clear about being used in apps v.s. used in libraries
  • mhdawson: 3 months of notice etc. would be useful
  • Wes: we need library authors to also make features relying on experimental features experimental and opt-in for their users
  • Jacob: opt-in..
  • Stephen: timebomb..
  • Tierney: about experiment code, maybe make the experiment opt-in via API
  • James: those who are impacted the most usually don't have power to change it. They may not even see the console.
  • James: another approach, warning depending on node_modules
  • Joyee: we are doing that for require(esm) on v22
  • Tierney: we need to change maintainer behavior..
  • Wes: get people to use lint rules
  • Jordan: every time people say let's change people's behavior, that's usually not possible. You can only carry it out with a carrot, not a stick. If we want people to use A over B we must make A better than B.
  • Wes: not being broken in the next deploy is obviously better
  • James: if it's not in node_modules, require people to use a flag
  • mhdawson: a breaking change is still going to break users. Comes back to whether we want people to use it. We need to be clearer
  • James: that's a problem for all breaking changes, not just experiments
Select a repo