Nushell core team meeting 2022-07-27

Attendees

Andres
Fernando
JT
Jakub
Reilly
Michael
Stefan

Agenda

• Completions
• Lazy dataframes

Discussed topics

New variable naming convention

Arrived after 0.66.
Small missed bits should be fixed now
Parsing a bit gnarly -> polish with parser refactor

Hotfix required

ls panic on non existing path -> fixed by nibon7

Cherry pick for version support branch to not introduce the naming convention change yet?

Goal: releases should feel more stable than using bleeding edge main

Decision Small new branch with cherry picked fix (+ other fix from darren) (Thanks for taking up the challenge JT)

Next rewrites for the 0.80 stretch goal

We have to polish some warts in the engine/data model

Staged improvements instead of the big engine-x at once

Some tech debt will be taken up by incremental but can be payed of by follow up refactors but should be better experience for the users

But we want to do some experimentations to find good solutions
(e.g. the despanning is gnarly)

Detour: spans

IDE support should be able to reuse the definitions and just update spans when files are updated (id for the files in the span required but could be reused to some extent when updating files)

Possible opt: span ID

Repl pseudo file is a room for optimization?

GC for definitions

Reference counting required for ide support anyways
also interactive IDE use will demand overwriting/dropping definitions

File sourcing

• refer to previous meeting

Jakub added relative import support

Still required module import inside modules for reexport

Completions

Extern definition can not capture the full semantics (we have some other potential uses)
Solving it will require some hard work

There are tools out there

e.g. carapace: https://github.com/rsteube/carapace
How to communicate

Can we parse the declarative parts of other completions syntaxes from zsh/fish etc.
(there will alwayse be some parts that needs to be dynamically evaluated e.g. git commit ids)

Also man page parsing/ clap/argparse libary exporters

Inspiration time:

Julia

(we talked about pluto last week)

They try to be strong in the data science field with their math and scientific computing focus

The complexities there might be too much for us to make nushell a data science language first

1.0 and 0.80 goals

Packaging/packages should exist and iron out the kinks before 1.0 arrives
Currently no noticeable progress on package manager development

we should consolidate the different documents we have around that

State of lazy dataframes

We use dataframes as a great way to speed up certain operations but we are definitely not feature complete in that space

little identity crisis of nushell dataframes (R/python have a bunch of algorithms specifically, Julia as well some algorithms)

Other data representations for scientific computing/ machine learning exist (e.g. "Tensors", different matrix formats), we are probably not the best language to address them

BTW: plugins can now define types

Thanks to mathspy
https://github.com/nushell/nushell/pull/6070

Missing in the release note!

Hype for 20k !!!