# Nushell core team meeting 2022-11-23

## Attendees

- Reilly
- Darren
- JT
- Michael
- Jakub
- Stefan
- filipanderson245
- labestiapty
- WindSoilder

## Agenda

- Elephant size
- We have picked the correct size
- Highlighting unknown externals in red
- simple, Nu-centric idea by Jakub: https://github.com/nushell/nushell/issues/4134#issuecomment-1322193649
- Disable strict flag validation for external commands
- Frequent source of complaints (hard work to get a complete issue list)
- if `extern` not fully up to date (or capable to express the allowed grammar) nu rejects commands. (even though it only should be helping with completions/help notes)
- Removing `exec`
- Parser Refactoring
- moving away from type-directed parsing
- `out>`, `err>`, `out+err>`
- `&&` vs `;`
- Redirection differences with Bash
- How to not save entries to history?
- More chatting about plugins (including update on wasm)
- `break`, `continue`, and `return`

# Discussed Topics

## Caching external commands for syntax highlighting

- simple, Nu-centric idea by Jakub: https://github.com/nushell/nushell/issues/4134#issuecomment-1322193649
- JT: previous experiments on every enter press were slow
- Complication through for example WSL2 file system boundary slowness (NTF..Slow in general)
- cache invalidation fun
- Jakubs suggestion in user space
- Benefit wouldn't run for scripts or contexts that users don't need it
- implementation of parts in nu might have additional performance cost
- JT
- Could we have a multipass highlighter
- Would require asynchronous updates in reedline

## Disable strict flag validation for externals

Don't (yet) have a tracking issue for all the problems in this area
JT+Stefan: we should make the command go through no matter what, maybe with warnings

Jakub: we might want this for custom commands too, for custom commands that are wrappers around externals. Otherwise you gotta use aliases (which are quite limited)
Stefan: need something like rest (...) for flags but with enforced order (including non flag parts) as that matters on externals.
More thought needed in this area

Fall-through signatures idea (structured approach) (Added by Jakub)

def foo [x, -y: int, --bar, ...rest, ..-flags] {
    # $rest      captures all unknown positionals
    # $flags     captures all unknown everything else (unlocks fall-through signature)
    # $signature captures all signature as a structured data (preserves ordering)
    
    $rest
    $flags
    $signature
}

# $rest is [ 20 ]
# $flags is [ --baz -z ]

> foo 10 20 -y 30 --baz -z
 0   20
 
 0   --baz
 1   -z
 
 #   parameter    name    value   known
────────────────────────────────────────
 1   positional   x          10   true
 2   positional              20   false
 3   option       -y         30   true
 4   flag         --baz           false
 5   flag         -z              false
 
 > foo 10 20 -y 30 --baz -z | into call
 foo 10 20 -y 30 --baz -z 

Jakub: externs are the highest priority
JT: need someone to step up and update the signatures, nobody's finishing them
Darren: can't really finish cargo completions, there's an infinite number if you consider plugins etc.
Agreement that we should pass through unrecognized bits for known externals
JT: should we just use carapace for external completions?
Jakub: yeah probably
Some concern because we can't really bundle carapace, doesn't help with the default experience
Let's start with the fallthrough

## Removing `exec`? Bug source

Welcome to the posix world, problems with ssh etc.

Darren if we can't exactly mirror all the bash flags and behavior this will cause problems.

(JT: would we have to treat flags as strings if they are unknown.)

Blocker for using nushell in ssh / non-priviliged contexts on a server etc.

**Conclusion** We need to go to the drawing board in shelling out to externals and running `exec` etc. Treat more stuff as string if necessary

## Parser refactoring
Still using type-directed parsing, JT's been augmenting the existing parser

### new redirection syntax

Tech demo by JT

New syntax to keep math the same
Behavior still different from how regular shell's work when it comes to the combinations

Stdin/Stdout/Stderr handling is still buggy in several places! (also in `do` etc.)

Observation we can't be 100% bash compatible (dump straight from StackOverflow) without breaking core nushell functionality (free expressions for math/comparison etc.)

### new keywords (`while`, and missing `break`, `continue`, `return`)

need some hand rolled parse logic

### `||`, `&&`, `;`

We currently treat `;` like `bash` `&&`

General hunch: `;` should still continue to default to break on error

`&&` would go well with one-liners on the internet
but `\\` is really arcane for error handling

Consensus: we are different enough from bash that its not worth it

*Goodnight sweet prince bashisms*

Darren: if we don't get to a `try` style error handling pull the simpler bash style error handling back from the drawer to keep the users able to handle their errors properly.

## How to avoid saving to history?

In zsh etc you prepend a space to the command
We could do the same
Stefan: needs a ~~simple~~ reedline change. Any volunteers?
Let's write up an issue

## Plugins update
JT chatted with wasmtime person, trying to figure out whether wasm will be right for Nu in the future. Doesn't see that happening for at least a year (maybe longer).
We are probably back to JSON-RPC.
Rust ABI stability (or lack thereof) makes generics hard. wasm has similar problems.
`repr(C)` works but you lose so much of Rust's functionality...

Is the current plugin system good enough for v1.0? Probably
Need a clear owner if we're going to deliver a radically revamped plugin system
Conclusion for now: keep plugins as-is, maybe someone will come along to drive plugins 2.0

## Misc
Darren: try the `table -e` display output hook
Darren: try out Maxim's `scroll` TUI https://github.com/nushell/nushell/pull/6984

# PR's

## Open for discussion
Triage if necessary, fill in small details if you are sheparding the PR
- [7028](https://github.com/nushell/nushell/pull/7028) Move the config of complete rules to other place (@Decodetalkers)
- breaking-change
- Currently marked as draft
- [7066](https://github.com/nushell/nushell/pull/7066) Support interpolation in backtick strings (@webbedspace)
- pre-meeting vote - land (darren) i don't really see any harm in having it
- [7087](https://github.com/nushell/nushell/pull/7087) Check for WRAP_PROMPT_XXX in environment (@PerBothner)
- I'm on it (Jakub)
- [7107](https://github.com/nushell/nushell/pull/7107) Use try_exist to handle the permission in cd (@Decodetalkers)
- try_exist is a +nightly feature so they wrote their own nu_try_exist until it is stable
- Reilly: don't understand what this does. Can someone who does update the title/description? will also need comment editing/rewording
- [7136](https://github.com/nushell/nushell/pull/7136) Make `first` and `last` always return 1 element, let `take` return from end of list (@rgwood)
- breaking-change
- Currently marked as draft
- pre-meeting vote: land (darren)
- [7209](https://github.com/nushell/nushell/pull/7209) `uniq -i` does not convert to lowercase (#7192) (@raccmonteiro)
- breaking-change

## Looking for reviewers
- [6806](https://github.com/nushell/nushell/pull/6806) Protocol: debug_assert!() Span to reflect a valid slice (@dbuch)
- CI green
- Several areas are affected: we should make sure the fixes are not hiding larger problems
- [7132](https://github.com/nushell/nushell/pull/7132) chore: chrono_update (@Decodetalkers)
- Currently marked as draft
- Would like to get this one in before the next release (darren)
- [7158](https://github.com/nushell/nushell/pull/7158) Add extended attributes (xattr) column to `ls -l` on supported platforms (@remmycat)
- Question: How should this be tested properly
- [7204](https://github.com/nushell/nushell/pull/7204) Fix ignoring errors capturing stdin (@SUPERCILEX)
- CI fail
- hints at the larger issue around pipeline stalls/redirection etc.
## In process, no need for the full team
Pull out of here if you have concerns
- [6984](https://github.com/nushell/nushell/pull/6984) [MVP][WIP] `less` like pager (@zhiburt)
- cool, most bikeshedding seems to have succeeded, polishing, (crossterm 0.25)
- [6990](https://github.com/nushell/nushell/pull/6990) make ls work like dir on windows (@richardmarklund)
- breaking-change
- CI currently not green
- discuss if we don't want to go that route
- [7010](https://github.com/nushell/nushell/pull/7010) Make json require string and pass around metadata (@Kixunil)
- breaking-change
- Good fix of weird coercions to strings that give wrong parses after `from ...` commands
- They expanded to more than just `from json`
- Currently merge conflict that needs resolution (+code review)
- [7056](https://github.com/nushell/nushell/pull/7056) Fix needs_quotes() in `to nuon` (closes #6989) (@webbedspace)
- Generally approved, Blocked by Stefan's review comments (Will try to get around to addressing them)
- [7119](https://github.com/nushell/nushell/pull/7119) Fix for escaping backslashes in interpolated strings (fixes #6737) (@gavinfoley)
- We had to revert the predecessor, they figured out a fix, just wondering if we have enough tests.
- pre meeting Stefan & Darren: sentiment landing
- blocked on merge conflict
- [7141](https://github.com/nushell/nushell/pull/7141) color_config now accepts closures as color values (@webbedspace)
- Currently marked as draft
- Question: making sure output remains quick enough
- [7200](https://github.com/nushell/nushell/pull/7200) Add make completion out of box (@fwfurtado)
- out of scope -> move to nu_scripts
## Cold:
Only discuss if closing vs waiting is contentious
- [6798](https://github.com/nushell/nushell/pull/6798) Skip parsing named flags when not specified (@Yesterday17)
- Currently marked as draft
- [6898](https://github.com/nushell/nushell/pull/6898) prevent error when `lines` reads bad bytes (@merelymyself)
- We discussed that this approach was slightly going in the wrong direction, merelymyself recently had less time to work on nushell.
- [6910](https://github.com/nushell/nushell/pull/6910) Make `seq` return a `ListStream` where possible. (@merelymyself)
- pretty close, (Stefan: could pick it up if necessary)
- [7002](https://github.com/nushell/nushell/pull/7002) return Error if get meet nothing and without "i" (@Decodetalkers)
- breaking-change
- unclear status and PR description
- [7051](https://github.com/nushell/nushell/pull/7051) allow tables in ++ operator (@dmatos2012)
- Currently marked as draft
- Actual behavior dependent on the `append` implementation. weird conversion when mixing tables with other lists