Nushell core team meeting 2022-08-03

Attendees

• Darren
• Reilly
• JT
• Michael
• Stefan

Agenda

• Celebrations event
• Module proposal
• nupac development
• overloading commands by input type

Discussed Topics

Celebration stream

Nushell blasted through the 20k stars!!! (past fish)

Discord has a 25 people limit for video chat (not useful for a big celebration) (could be used for small office hours later on)

JT has streamyard to link to youtube or twitch, take in comments, and small group in the stream

Friday August 5, 20:00 UTC https://everytimezone.com/s/553cdb44

Pipeline refactor (jt)

Goal avoid copyint EngineState

Command::run now would take the engine_state: &'a and PipelineData<'a> constrained to the same lifetime

External has to run in a separate thread (We can not share 'static references to the engine state)
Also par-each for rayon parallelism should still work. (Efforts by the community to get more stuff parallel in general)

Possible workaround: scoped threads

We don't want to Arc<Mutex<EngineState>> our way to a global interpreter lock (GIL) to have the same challenges as Python with multithreading/

This design challenge is probably a blocker for the current parallelization PR (https://github.com/nushell/nushell/pull/6213)

nu core crate

Trying this idea was hard due to it affecting all commands

Can we separate out more stuff to do quicker testing

nupac

A package manager for nu written in nu

https://github.com/skelly37/nupac

skelly and friends are working on it

Current gripe: package metadata as a comment in the scripts (PR on nu_scripts would introduce it to every script right now)

Potential alternative: nuon file like the package.json/Cargo.tomls of the world
Could also map to plugins written in different languages than nu

We should make sure we get a solution that can be picked up by the community, we don't want to end up in the situation that we deem it unsuitable for the community after they did all the work.

Package should be an opt-in thing and not necessary to use/get started with nu(shell)

But official sources would be very helpful to serve the often mentioned needs (completions, prompt integrations)

Backing data should probably be an easy to work with github repo

Side note: probably among the biggest open source project in nu on github (good for regression tests and performance analysis, feedback about language constructs)

Jakubs module proposal

https://hackmd.io/@nucore/r1zilyw6q

Problems to solve

• How to import environment variables from parser definitions in modules in a clean way
• Potential namespacing issues (resolve collisions in a predictable way)
• Currently impossible:
  • overlay add in a module is forbidden (as it dynamically changes environment and definitions)

Changes

Separate environment from parser definitions (-> makes parsing more consistent?)

module syntax would be a subset of a script (replaces two somewhat different parsing modes we currently need)

environment {} block defines part to run when sourcing
This can not interfere directly with the

Field report: IOX (Michael)

https://github.com/influxdata/influxdb_iox

• iox currently not yet cratified
• nushell as a querying/viewing tool

Primarily a time series analytics/db environment
Techstack: Arrow, parquet as persistance file format (allows chunking)

Michael tries to combine core nushell commands with the iox commands to

Binary nushell build embedded in own build of iox

https://github.com/stormasm/influxdb_iox/tree/stormasm/nushell/iox_nu

Would be another avenue to use nu as a domain specific language

Q: instead of further cratification of our commands could we use targeted feature flags to create the core subset of nu-command

Required cleanup: Command overloading based on pipeline output type

Our documentation is now a superfund site…

Help messages have to be polished.
Website is borked

Surface the types to the user in an understandable way

The common behavior should be predictably same, and the options documented

Seems to only work at parse time?