Nushell core team meeting 2022-11-23

Attendees

  • Reilly
  • Darren
  • JT
  • Michael
  • Jakub
  • Stefan
  • filipanderson245
  • labestiapty
  • WindSoilder

Agenda

  • Elephant size
    • We have picked the correct size
  • Highlighting unknown externals in red
  • Disable strict flag validation for external commands
    • Frequent source of complaints (hard work to get a complete issue list)
    • if extern not fully up to date (or capable to express the allowed grammar) nu rejects commands. (even though it only should be helping with completions/help notes)
  • Removing exec
  • Parser Refactoring
    • moving away from type-directed parsing
    • out>, err>, out+err>
  • && vs ;
  • Redirection differences with Bash
  • How to not save entries to history?
  • More chatting about plugins (including update on wasm)
  • break, continue, and return

Discussed Topics

Caching external commands for syntax highlighting

  • simple, Nu-centric idea by Jakub: https://github.com/nushell/nushell/issues/4134#issuecomment-1322193649
  • JT: previous experiments on every enter press were slow
  • Complication through for example WSL2 file system boundary slowness (NTF..Slow in general)
  • cache invalidation fun
  • Jakubs suggestion in user space
    • Benefit wouldn't run for scripts or contexts that users don't need it
    • implementation of parts in nu might have additional performance cost
  • JT
    • Could we have a multipass highlighter
    • Would require asynchronous updates in reedline

Disable strict flag validation for externals

Don't (yet) have a tracking issue for all the problems in this area
JT+Stefan: we should make the command go through no matter what, maybe with warnings

Jakub: we might want this for custom commands too, for custom commands that are wrappers around externals. Otherwise you gotta use aliases (which are quite limited)
Stefan: need something like rest () for flags but with enforced order (including non flag parts) as that matters on externals.
More thought needed in this area

Fall-through signatures idea (structured approach) (Added by Jakub)

def foo [x, -y: int, --bar, ...rest, ..-flags] {
    # $rest      captures all unknown positionals
    # $flags     captures all unknown everything else (unlocks fall-through signature)
    # $signature captures all signature as a structured data (preserves ordering)
    
    $rest
    $flags
    $signature
}

# $rest is [ 20 ]
# $flags is [ --baz -z ]

> foo 10 20 -y 30 --baz -z
 0   20
 
 0   --baz
 1   -z
 
 #   parameter    name    value   known
────────────────────────────────────────
 1   positional   x          10   true
 2   positional              20   false
 3   option       -y         30   true
 4   flag         --baz           false
 5   flag         -z              false
 
 > foo 10 20 -y 30 --baz -z | into call
 foo 10 20 -y 30 --baz -z 

Jakub: externs are the highest priority
JT: need someone to step up and update the signatures, nobody's finishing them
Darren: can't really finish cargo completions, there's an infinite number if you consider plugins etc.
Agreement that we should pass through unrecognized bits for known externals
JT: should we just use carapace for external completions?
Jakub: yeah probably
Some concern because we can't really bundle carapace, doesn't help with the default experience
Let's start with the fallthrough

Removing exec? Bug source

Welcome to the posix world, problems with ssh etc.

Darren if we can't exactly mirror all the bash flags and behavior this will cause problems.

(JT: would we have to treat flags as strings if they are unknown.)

Blocker for using nushell in ssh / non-priviliged contexts on a server etc.

Conclusion We need to go to the drawing board in shelling out to externals and running exec etc. Treat more stuff as string if necessary

Parser refactoring

Still using type-directed parsing, JT's been augmenting the existing parser

new redirection syntax

Tech demo by JT

New syntax to keep math the same
Behavior still different from how regular shell's work when it comes to the combinations

Stdin/Stdout/Stderr handling is still buggy in several places! (also in do etc.)

Observation we can't be 100% bash compatible (dump straight from StackOverflow) without breaking core nushell functionality (free expressions for math/comparison etc.)

new keywords (while, and missing break, continue, return)

need some hand rolled parse logic

||, &&, ;

We currently treat ; like bash &&

General hunch: ; should still continue to default to break on error

&& would go well with one-liners on the internet
but \\ is really arcane for error handling

Consensus: we are different enough from bash that its not worth it

Goodnight sweet prince bashisms

Darren: if we don't get to a try style error handling pull the simpler bash style error handling back from the drawer to keep the users able to handle their errors properly.

How to avoid saving to history?

In zsh etc you prepend a space to the command
We could do the same
Stefan: needs a simple reedline change. Any volunteers?
Let's write up an issue

Plugins update

JT chatted with wasmtime person, trying to figure out whether wasm will be right for Nu in the future. Doesn't see that happening for at least a year (maybe longer).
We are probably back to JSON-RPC.
Rust ABI stability (or lack thereof) makes generics hard. wasm has similar problems.
repr(C) works but you lose so much of Rust's functionality

Is the current plugin system good enough for v1.0? Probably
Need a clear owner if we're going to deliver a radically revamped plugin system
Conclusion for now: keep plugins as-is, maybe someone will come along to drive plugins 2.0

Misc

Darren: try the table -e display output hook
Darren: try out Maxim's scroll TUI https://github.com/nushell/nushell/pull/6984

PR's

Open for discussion

Triage if necessary, fill in small details if you are sheparding the PR

  • 7028 Move the config of complete rules to other place (@Decodetalkers)
    • breaking-change
    • Currently marked as draft
  • 7066 Support interpolation in backtick strings (@webbedspace)
    • pre-meeting vote - land (darren) i don't really see any harm in having it
  • 7087 Check for WRAP_PROMPT_XXX in environment (@PerBothner)
    • I'm on it (Jakub)
  • 7107 Use try_exist to handle the permission in cd (@Decodetalkers)
    • try_exist is a +nightly feature so they wrote their own nu_try_exist until it is stable
    • Reilly: don't understand what this does. Can someone who does update the title/description? will also need comment editing/rewording
  • 7136 Make first and last always return 1 element, let take return from end of list (@rgwood)
    • breaking-change
    • Currently marked as draft
    • pre-meeting vote: land (darren)
  • 7209 uniq -i does not convert to lowercase (#7192) (@raccmonteiro)
    • breaking-change

Looking for reviewers

  • 6806 Protocol: debug_assert!() Span to reflect a valid slice (@dbuch)
    • CI green
    • Several areas are affected: we should make sure the fixes are not hiding larger problems
  • 7132 chore: chrono_update (@Decodetalkers)
    • Currently marked as draft
    • Would like to get this one in before the next release (darren)
  • 7158 Add extended attributes (xattr) column to ls -l on supported platforms (@remmycat)
    • Question: How should this be tested properly
  • 7204 Fix ignoring errors capturing stdin (@SUPERCILEX)
    • CI fail
    • hints at the larger issue around pipeline stalls/redirection etc.

In process, no need for the full team

Pull out of here if you have concerns

  • 6984 [MVP][WIP] less like pager (@zhiburt)
    • cool, most bikeshedding seems to have succeeded, polishing, (crossterm 0.25)
  • 6990 make ls work like dir on windows (@richardmarklund)
    • breaking-change
    • CI currently not green
    • discuss if we don't want to go that route
  • 7010 Make json require string and pass around metadata (@Kixunil)
    • breaking-change
    • Good fix of weird coercions to strings that give wrong parses after from ... commands
    • They expanded to more than just from json
    • Currently merge conflict that needs resolution (+code review)
  • 7056 Fix needs_quotes() in to nuon (closes #6989) (@webbedspace)
    • Generally approved, Blocked by Stefan's review comments (Will try to get around to addressing them)
  • 7119 Fix for escaping backslashes in interpolated strings (fixes #6737) (@gavinfoley)
    • We had to revert the predecessor, they figured out a fix, just wondering if we have enough tests.
    • pre meeting Stefan & Darren: sentiment landing
    • blocked on merge conflict
  • 7141 color_config now accepts closures as color values (@webbedspace)
    • Currently marked as draft
    • Question: making sure output remains quick enough
  • 7200 Add make completion out of box (@fwfurtado)
    • out of scope -> move to nu_scripts

Cold:

Only discuss if closing vs waiting is contentious

  • 6798 Skip parsing named flags when not specified (@Yesterday17)
    • Currently marked as draft
  • 6898 prevent error when lines reads bad bytes (@merelymyself)
    • We discussed that this approach was slightly going in the wrong direction, merelymyself recently had less time to work on nushell.
  • 6910 Make seq return a ListStream where possible. (@merelymyself)
    • pretty close, (Stefan: could pick it up if necessary)
  • 7002 return Error if get meet nothing and without "i" (@Decodetalkers)
    • breaking-change
    • unclear status and PR description
  • 7051 allow tables in ++ operator (@dmatos2012)
    • Currently marked as draft
    • Actual behavior dependent on the append implementation. weird conversion when mixing tables with other lists
Select a repo