Nushell core team meeting 2022-07-13

Attendance

Andres
Darren
Fernando
Jakub
JT
Michael
Stefan

Agenda

• Quoted externals and globs
  • PR #6014 disables globs for externals
• Tabled integration status
• Stderr/err
• Bashisms
• Data exploration tools

Topics

Recently: hooks and overlays have been changed by work from Jakub

tabled landed (@zhiburt, @fdncred)

Couple of bugs have been reported
(Still problems with people reporting not on the recent main branches)

Darren has been labelling PRs/Issues

AWESOME: Maxim (@zhiburt) is really quick in fixing bugs on the tabled side while having less ugly hacks!
Basic look and feel is very comparable to previous nu-table

Still somewhat slower than old nu-table -> profile and improve in the future

Output performance very terminal emulator/platform dependent

NEW: config settings for truncating instead of wrapping

PR #6014 globbing and its interactions with externals

Sometimes external commands want the expanded version of * etc. and sometimes the raw *

IN bash * vs "*"

but we need the " ' for regular string literals for crossplatform handling of escape characters

Draft decision: accepting the bash style with little adaptation

Should transfer the expectations as best as possible

Bare word: will expand glob for external
Backticks: will also expand glob
(Single and double)Quotes: prevent glob expansion

Data handling: query planning/data backends

JT had chat with Andy Grove and learned a bunch about query planning/lazy data handling:

We want to be able to tail a log, peak the head of a csv file, but also be performant on the full dataset

For the full case a lot of work has been done by other tools (polars, datafusion etc. with arrow parquet etc.)

How to deal with streaming sources like a running process or background task writing to a file/socket?

Goal: can we handle it without getting too fancy. For special cases the work has already been done. can we delegate were it is advantageous instead of reinventing the wheel.

For regular data schema is a massive enabler
But nushell will also deal with data without a well defined schema

We want to hit 1.0 and in the future might optimize particular usecases later

Key aspect: lazyness in nushell how to balance its optimization on a spectrum of usecases

Dataflow graph relevant to sql/lazy dataframes but could also separately be
room for "compiler" optimization in general nushell

Hook update (jakub)

overlays have been introduced as a way to change the environment and set of available modules/commands (do stuff like virtualenv, conda, nvm)

Now hooks (e.g. change current dir/ env, match on $env.PWD) can be used with either a block {} or string (basically eval can also create defs directly)

TODO: good explicit demo @kubouch

Current rough edges:

• overlay removal restores all set environment variables but could get you stuck in a particular directory (special case to fix currently)

• autoenv behavior needs polish as it is currently clunky

Current ideas

hook add statement as nice sugar

if we want config add to work the same for the config (like git config) we might want to move the config parts over to nuon instead of script to autochange it

Bashisms

Ways to redirect files/std… on bash/posix
https://askubuntu.com/questions/420981/how-do-i-save-terminal-output-to-a-file/731237#731237

simple ones:

cmd > file.txt

ugly ones:

errorings 2>&1 whatever

We do not want to open the slippery slope.

Also confusion should not be accepted with too many symbols
Being different and verbose should not be done just for verbosities sake.

Current compromise for acceptable bashisms:

just file write and file append
> and >>

more bashisms

also consider &&, ;, || for short hand control flow

(a && b) || c

ensures b only runs after a succeeds and cleanup with c if they fail

In the meantime emit parse errors.

Redirection bikeshed

We need to find a good solution for standard out/err/in anyways and is not simple yet

JTs suggestion for redirection the "SHELLFISH"

>stderr>
>stdout+stderr>
git status >stderr> save myfile.txt
git status >stderr> | save myfile.txt

git status >stderr> find "err" > log.txt

Darren's alternative suggestion let them be commands and still require nushell pipes around the special redirection commands