Nushell core team meeting 2022-11-16

Attendees

• Darren
• JT
• Jakub
• Reilly
• Michael
• Stefan
• Dan
• WindSoilder
• FilipAnderson

Agenda

• Timing for GitHub Nushell support
• Plans for parser changes
• Trimming down commands / messaging our intent as we do
• More thoughts on plugins
• Input/output signature syntax
• Final <15min PRs

Discussed Topics

Observations before the scheduled agenda

table --expand is going great (and with the display_hook you can expand it based on terminal size)

This release packaging was quick

GitHub nu(shell) integration

nu extension still refers to the objective-C lisp .nu
We are starting to reach critical mass to reach out to github/linguist for .nu as nushell

Needs 200 repos
https://github.com/github/linguist/blob/master/CONTRIBUTING.md#adding-a-language

Plans for parser changes

JT's thinking of landing small incremental changes instead of a big bang
- will support bashisms like >, &&, ||. some changes to the pipeline data structure needed (can no longer be a simple `Vec<Command>``)

a | b
Pipeline {items: &[Command]}

a > file.dat
a || b
a && b
// Needs a more flexible structure
enum Pipeline {
  Redirect(...)
  Or(..)
  And(..)
  ..
}

Experiment now archived

Trimming down commands

• Areas to trim:
  • build-string
    • trivially replaced > gone
  • shells command (n, p)
    • hogging valuable chars
  • dataframes commands (big chonker)
    • Balance value (biggish data runtime performance, conference wowfactor) and cost (build times, docs complication, dev expertise needed)
    • currently no direct replacement available
  • indexing commands

Darren's command spreadsheet: https://docs.google.com/spreadsheets/d/1w8FWy7FXhkiIQ7EJN0wF8SdDWImGwVwxywEg85LJtpU/edit#gid=0

Please put in your hottakes: commands that can be useably replaced, renaming to consistency (also inspect flags)

Dan: we can still make breaking changes and can really reflect on the state of the standard library
Stefan: should strike the right balance between combining a million tiny commands and using 1 big bloated command with 100 flags
JT: latent fear that we loose momentum but we should feel comfortable to make the breaking changes! we are in a reflective period to inspect naming/organization. No we have momentum for the editorial process and reviewing -> let's do it
Help get users down the happy paths -> help with common patterns
Darren: removing commands is hard when we lose functionality, tough decisions to make. is it OK to lose functionality like shells?

JT:
from ... :parse read convert load(s)
to ... :serialize write format dump(s)

read/write file vs pipe
Dan:
to and into are confusing
The pair for file format parsing is core to nushell functionality + casting/interconversion

JT:
Steady stream of breaking changes vs. ripping the bandaid in a batch
Consensus: At least make the topical changes together

Maybe rename the from/to pair to something and make to the casting command (most languages seem to have To... methods/functions)

Plugin update

JT: been thinking about options for plugins: new plugin system (WASM? DLLs?) vs augmenting the existing plugin system incrementally. Leaning toward incremental
Jakub: agrees w/ incremental
Stefan: agrees w/incremental from risk and community perspectives
Darren: Gonna need a big jump in functionality to do things like dataframes in plugins
Michael: Need to get Fernando's input and consult him on the future of dataframes
JT: we should remove complicated parts that don't have enough maintenance energy behind them. maybe we put out a survey asking how many people are actually using dataframes

-> Action Item: community survey about dataframes (Michael, JT)

Input output type signatures

Jakub's exploration

def foo(in: int) -> string { 
   ls | get $in | get name 
}

def ls(path?: Path) -> Stream<{ name: Path, size: Filesize, .. }>
   ls 
}

JT's notes:

def ls(path?: Path) -> Stream<{ name: Path, size: Filesize, .. }> {
   ls 
}

ls | get Foo

ls | get n<tab> (gives you `name`)

# Stream => Stream<Any>
def ls(path?: Path) -> Stream {
    # ...
}

type LSEntry = { name: Path, size: Filesize, .. }

# Stream => Stream<Any>
def ls(path?: Path) -> Stream<LSEntry> {
    # ...
}

# Scripts to Programs

gradual progression in the type stringency (we currently have opaque types, moving to wards structural types, + named parts in the structure. + type aliases/ "structs")

(one-liner to) Scripts to programs

Q: WindSoilder: How stringent should we be for the input type.

Can we infer based on the implementation or do we assume Any if not given.

$in is kind of overused as a name
(additional reserved name input as a potential solution to clarify)

PR's

Open for discussion

Triage if necessary, fill in small details if you are sheparding the PR

• 6983 Grouped config commands better (closes #6911) (@webbedspace)
  • breaking-change (darren - i have reservations. it's probably not a breaking change because old configs still work, but the new "structured" config format works too. which means the config rust code is implemented twice in order to support old and new formats. the implication is we'll eventually remove the non-structured format which then will be a massive breaking change since it changes the format of the config.nu file.)
  • Will cause future user churn/ apparently still supports old configs
• 6994 Allow iteration blocks to have an optional extra index parameter (alternative to -n flags) (@webbedspace)
  • we touched on this one in the last meeting but didn't get to a decision due to merge conflict
  • pre-meeting vote - land (darren)
• 7028 Move the config of complete rules to other place (@Decodetalkers) - breaking-change
  • Currently marked as draft
• 7066 Support interpolation in backtick strings (@webbedspace)
  • pre-meeting vote - land (darren) i don't really see any harm in having it
• 7079 add signature information when get help on one command (@WindSoilder)
  • pre-meeting vote - land (darren, stefan) - we can continue to iterate if we want
• 7082 feat: Use Raw text to save if pipeline data is ExternalStream or String (@Decodetalkers)
  • breaking-change
  • pre-meeting vote - land (darren)
• 7087 Check for WRAP_PROMPT_XXX in environment (@PerBothner)
  • I'm on it (Jakub)
• 7107 Use try_exist to handle the permission in cd (@Decodetalkers)
  • try_exist is a +nightly feature so they wrote their own nu_try_exist until it is stable
• 7119 Fix for escaping backslashes in interpolated strings (fixes #6737) (@gavinfoley)
  • We had to revert the predecessor, they figured out a fix, just wondering if we have enough tests.
  • pre meeting Stefan & Darren: sentiment landing
• 7122 Restore original do -i behavior and add flags to break down shell vs program errors (@SUPERCILEX)
• 7124 new command url parse (#6854) and url subcommands tests (@raccmonteiro)
  • review: Dan
  • pre-meeting vote: land (darren)
• 7136 Make first and last always return 1 element, let take return from end of list (@rgwood)
  • breaking-change
  • Currently marked as draft
  • pre-meeting vote: land (darren)

Looking for reviewers

• 6806 Protocol: debug_assert!() Span to reflect a valid slice (@dbuch)
  • CI green
  • Several areas are affected: we should make sure the fixes are not hiding larger problems
• 7132 chore: chrono_update (@Decodetalkers)
  • Currently marked as draft
  • Would like to get this one in before the next release (darren)

In process, no need for the full team

Pull out of here if you have concerns

• 6898 prevent error when lines reads bad bytes (@merelymyself)
  • We discussed that this approach was slightly going in the wrong direction, merelymyself recently had less time to work on nushell.
• 6910 Make seq return a ListStream where possible. (@merelymyself)
  • pretty close, (Stefan: could pick it up if necessary)
• 6984 [MVP][WIP] less like pager (@zhiburt)
  • cool, most bikeshedding seems to have succeeded, polishing, (crossterm 0.25)
• 6990 make ls work like dir on windows (@richardmarklund)
  • breaking-change
  • CI currently not green
  • discuss if we don't want to go that route
• 7010 Make json require string and pass around metadata (@Kixunil)
  • breaking-change
  • Good fix of weird coercions to strings that give wrong parses after from ... commands
  • They expanded to more than just from json
  • Currently merge conflict that needs resolution (+code review)
• 7056 Fix needs_quotes() in to nuon (closes #6989) (@webbedspace)
  • Generally approved, Blocked by Stefan's review comments (Will try to get around to addressing them)

Cold:

Only discuss if closing vs waiting is contentious

• 6798 Skip parsing named flags when not specified (@Yesterday17)
  • Currently marked as draft
• 7002 return Error if get meet nothing and without "i" (@Decodetalkers) - breaking-change
  • unclear status and PR description
• 7051 allow tables in ++ operator (@dmatos2012)
  • Currently marked as draft
  • Actual behavior dependent on the append implementation. weird conversion when mixing tables with other lists