The Module Environment Overhaul

This design document aims to tie together many different moving parts related to how environment is handled in modules.

Related Work

• https://github.com/nushell/nushell/pull/6162 allowed to import modules from within other modules
• https://github.com/nushell/nushell/pull/6150 makes commands like use use file-relative path inside scripts
• Planned refactor of source to allow dynamic paths and only keep the environment (not custom commands etc.). Seems agreed upon within the core team.

Motivation

Assuming the following module:

# spam.nu
export env FOO { "foo" }

export def foo { $env.FOO }

# foo.nu
use spam.nu FOO

Currently, we face the following problems:

1. Environment variable imports work only inside a script. When calling use spam.nu FOO in a module (foo.nu in the above example), the FOO would never be brought in because modules are never evaluated. It is therefore possible to bring in only aliases and custom commands inside modules with use in its current form. (This issue was introduced with #6162.)
2. Fragile environment import namespacing. Calling use spam.nu; spam foo would throw an error because the foo command depends on $env.FOO while we got $env.'spam FOO'. Therefore, the module requires to be imported in a certain way (e.g., use spam.nu *) on the user side which is error-prone. This error is also not checked by the parser since environment variables are tracked at runtime by the engine.
3. Script vs. module dichotomy. Currently, we have two parsing modes with different rules: scripts and modules. For example, attempting to use export in a script throws an error. This can be demonstrated by open spam.nu | nu-highlight – it shows all red even though it's a valid Nushell code. The nu-check command deals with this problem by relying on some trial and error and heuristics to figure out whether a file is a script or a module – not ideal.
4. Parser tracking environment variables in modules. The parser currently tracks which environment variables are exported from a module. This feels a bit strange because environment variables are a runtime construct in the engine. While not being a problem per se, it causes some complications in the hide command and we do not really utilize the information anywhere, except the granular env. var. use and hide from modules. Intuitively, it feels wrong.

All of these issues are related to environment variables and modules. We need to find a solution that would fix all the issues at once and it might consist of multiple moving parts, therefore better to write it down.

How To Get There

It should be possible to split the problem into a several bite-sized chunks.

1. (Done) Simplify environment exports

Instead of having export env for each environment variable, there would be an export-env { ... } command that would evaluate the block and preserve its environment when called in a script.
In a module, the command will be tolerated by the parser but wouldn't do anything in the parser.

To bring an environment from it, you'd need to call source-env or overlay use to evaluate the module (see step 3.).

2. (Done) source-env (previously source) preserves only the environment

Note: Since the dynamic paths for source-env proved to be unfeasible, we moved the export-env evaluation to use and removed source-env.

source-env would evaluate the whole file but preserve only the environment.
Parser definitions would need to be brought in with use or using overlays from now on.

3. (Mostly Done) Script vs. module syntax unification

After this change, modules would be a subset of scripts. That is, a valid module code is also a valid script code. However, some commands would not be allowed in modules (e.g., overlay commands). Note!, I'm talking about top-level module commands. It is still possible to use everything inside, e.g., the export def foo [] { ... } command block.

There are three cases (AFAIK) when module code is not a valid script code:

• export ... stuff: We would treat export XXX in a script the same as calling XXX, except:
• main command: If used in module, it would define a command named after the module.
• overlay use will need to evaluate the module (equivalent of source-env).
• allow export-env in a module

After these three cases, I believe there is nothing more in modules that wouldn't pass as a script. If source-env is called on a module, the export-env { ... } would simply be evaluated as any other command which would work as if you imported the environment variables from the module.

4. Cleanup

• remove export env
• remove export env eval from overlay use
• remove env var functionality from hide
• remove env var functionality from use
• remove source

5. Solve expected complications

See the end of this document.

6. Solve unexpected complications

The End Result

Note source-env is superceded by use

Here is an example sketch of how the end result could look like.

# helper.nu
export-env { let-env BAZ = "baz" }

export def get-bar [] { 'bar' }

# spam.nu
use helper.nu get-bar

export-env {
    print 'loading spam!'
    source-env helper.nu 

    load-env {
        FOO: "foo"
        BAR: (get-bar)
        SPAM_NAME: "spam"
    } 
}

export def foo [] { 'foo' }

export def main [] { $'running ($env.SPAM_NAME)!' }

# REPL session 1 -- unified environment per module
> source-env spam.nu
loading spam!

> $env | get FOO BAR BAZ
0  foo
1  bar
2  baz

> foo
# command not found!

> use spam.nu foo

> foo
foo

# REPL session 2 -- module and script as one
> use spam.nu

> spam foo
foo

> spam            # runs main unsuccessfully 
# error! $env.SPAM_NAME not found!

> source-env spam.nu
loading spam!

> spam            # runs main
running spam!

> nu spam.nu      # runs main
loading spam!
running spam!

> source-env spam.nu  # does not run main
loading spam!

Discussion

Two commands instead of one

Note: Not true anymore, use will import environment

Importing everything from a module would require two commands instead of one:

source-env spam.nu 
use spam.nu *

instead of just

use spam.nu *

One implication of requiring two commands is that you can still have environment error unchecked byt the parser, for example:

# spam.nu
export-env {
    let-env FOO = "foo" 
}

export def foo { $env.FOO }

If you don't source-env spam.nu prior to use spam.nu, the foo command would fail because there is no FOO environment variable brought in.

You could use overlay use spam.nu, though, to keep it in one command. The ovelrays in general would be a way how to "tie together" the environment and parser definitions of a module. We could think of ways to expand the overlays functionality to alleviare this pitfall, e.g., overlay melt spam.nu would take the spam.nu overlay but merge all its definitions into the existing overlay without bringing up the spam overlay at all.

No more individual environment imports

Another disadvantage is that we lose the granularity of environment variable imports. It would no longer be possible to selectively import environment variables from a module one-by-one. However, I think the purpose of environment variables is to define some environment together. Being able to chip off some individual environment variables does not seem that valuable to me. Furthermore, the new export-env { ... } method is very flexible. You could add all sorts of logic to decide how the environment gets created, making it potentially more powerful than the current method. It seems more logical to me to treat module's environment as one piece instead of each env var individually.

hide-env should still be able to hide individual environment variables. That can still be useful occasionally if you need to hot-fix something.

Expected Complications

[Done] File-relative paths in the engine

source-env would need to support file-relative paths which would need to be tracked in the engine somehow. Currently, we have file-relative paths only in the parser, the engine still uses cwd everywhere. Maybe we should make the engine to trace the current file as well?

main edge cases

Allowing main to define a command with the module's name has some edge cases:

# spam.nu

export def foo [] { 'foo' }

export def main [] { 'running spam!' }

# REPL

> use spam.nu       # defines the spam command

> use spam.nu foo   # does not define the spam command?

In the above case, the spam command is not defined. We could make any use of a module make to always define the top-level command, if there is a corresponding main command inside the module.

Or automatically any main or self imports into the spam command:

> use spam.nu main       # defines the spam command

> use spam [ foo main ]  # defines foo and spam commands

or

> use spam.nu self          # defines the spam command

> use spam.nu [ foo self ]  # defines foo and spam commands

Also, I think

# spam.nu
def main [] { 'running spam!' }

> use spam.nu

should not define the spam command. You could still nu spam.nu which would run the main, though.

Why export-env { ... }?

Instead of the export-env command, we could define def-env export-env [] { ... } and instead of source-env spam.nu, we'd call the export-env command to activate the environment.
This would work, however, if you called use spam.nu * and use foo.nu *, then the export-env commands would overlap. I imagine it would be quite confusing. I think the extra instrumentation required for the export-env { ... } would remove this footgun.

Conclusion

With all these changes, environment handling in modules would hopefully become simpler, less error-prone, or at least not broken. The changes involving source-env would also bring in the long-awaited feature of being able to source dynamic paths while keeping the core of the command–-defining an environment–-still there. I hope you liked reading this text and if you read it all, you earned 100 Nu points.