# The Module Environment Overhaul

This design document aims to tie together many different moving parts related to how environment is handled in modules.

## Related Work

* https://github.com/nushell/nushell/pull/6162 allowed to import modules from within other modules
* https://github.com/nushell/nushell/pull/6150 makes commands like `use` use file-relative path inside scripts
* Planned refactor of `source` to allow dynamic paths and only keep the environment (not custom commands etc.). Seems agreed upon within the core team.

## Motivation

Assuming the following module:

# spam.nu
export env FOO { "foo" }

export def foo { $env.FOO }

# foo.nu
use spam.nu FOO

Currently, we face the following problems:

1. **Environment variable imports work only inside a script.** When calling `use spam.nu FOO` in a module (`foo.nu` in the above example), the `FOO` would never be brought in because modules are never evaluated. It is therefore possible to bring in only aliases and custom commands inside modules with `use` in its current form. (This issue was introduced with #6162.)
2. **Fragile environment import namespacing.** Calling `use spam.nu; spam foo` would throw an error because the `foo` command depends on `$env.FOO` while we got `$env.'spam FOO'`. Therefore, the module requires to be imported in a certain way (e.g., `use spam.nu *`) on the user side which is error-prone. This error is also not checked by the parser since environment variables are tracked at runtime by the engine.
3. **Script vs. module dichotomy.** Currently, we have two parsing modes with different rules: scripts and modules. For example, attempting to use `export` in a script throws an error. This can be demonstrated by `open spam.nu | nu-highlight` -- it shows all red even though it's a valid Nushell code. The `nu-check` command deals with this problem by relying on some trial and error and heuristics to figure out whether a file is a script or a module -- not ideal.
4. **Parser tracking environment variables in modules.** The parser currently tracks which environment variables are exported from a module. This feels a bit strange because environment variables are a runtime construct in the engine. While not being a problem per se, it causes some complications in the `hide` command and we do not really utilize the information anywhere, except the granular env. var. `use` and `hide` from modules. Intuitively, it feels wrong.

All of these issues are related to environment variables and modules. We need to find a solution that would fix all the issues at once and it might consist of multiple moving parts, therefore better to write it down.

## How To Get There

It should be possible to split the problem into a several bite-sized chunks.

### 1. (Done) Simplify environment exports

Instead of having `export env` for each environment variable, there would be an `export-env { ... }` command that would evaluate the block and preserve its environment when called in a script.
In a module, the command will be tolerated by the parser but wouldn't do anything in the parser.

To bring an environment from it, you'd need to call `source-env` or `overlay use` to evaluate the module (see step 3.).

### 2. (Done) `source-env` (previously `source`) preserves only the environment

**Note: Since the dynamic paths for `source-env` proved to be unfeasible, we moved the `export-env` evaluation to `use` and removed `source-env`.**

`source-env` would evaluate the whole file but preserve only the environment.
Parser definitions would need to be brought in with `use` or using overlays from now on.

### 3. (Mostly Done) Script vs. module syntax unification

After this change, modules would be a _subset_ of scripts. That is, a valid module code is also a valid script code. However, some commands would not be allowed in modules (e.g., `overlay` commands). _Note!, I'm talking about top-level module commands. It is still possible to use everything inside, e.g., the `export def foo [] { ... }` command block._

There are three cases (AFAIK) when module code is not a valid script code:
- [x] `export ...` stuff: We would treat `export XXX` in a script the same as calling `XXX`, except:
- [ ] `main` command: If used in module, it would define a command named after the module.
- [x] `overlay use` will need to evaluate the module (equivalent of `source-env`).
- [x] allow `export-env` in a module

After these three cases, I believe there is nothing more in modules that wouldn't pass as a script. If `source-env` is called on a module, the `export-env { ... }` would simply be evaluated as any other command which would work as if you imported the environment variables from the module.

### 4. Cleanup

- [x] remove `export env`
- [x] remove `export env` eval from `overlay use`
- [x] remove env var functionality from `hide`
- [x] remove env var functionality from `use`
- [ ] remove `source`

### 5. Solve expected complications

See the end of this document.

### 6. Solve unexpected complications

:eyes:

## The End Result

**Note `source-env` is superceded by `use`**

Here is an example sketch of how the end result could look like.

# helper.nu
export-env { let-env BAZ = "baz" }

export def get-bar [] { 'bar' }

# spam.nu
use helper.nu get-bar

export-env {
    print 'loading spam!'
    source-env helper.nu 

    load-env {
        FOO: "foo"
        BAR: (get-bar)
        SPAM_NAME: "spam"
    } 
}

export def foo [] { 'foo' }

export def main [] { $'running ($env.SPAM_NAME)!' }

# REPL session 1 -- unified environment per module
> source-env spam.nu
loading spam!

> $env | get FOO BAR BAZ
0  foo
1  bar
2  baz

> foo
# command not found!

> use spam.nu foo

> foo
foo

# REPL session 2 -- module and script as one
> use spam.nu

> spam foo
foo

> spam            # runs main unsuccessfully 
# error! $env.SPAM_NAME not found!

> source-env spam.nu
loading spam!

> spam            # runs main
running spam!

> nu spam.nu      # runs main
loading spam!
running spam!

> source-env spam.nu  # does not run main
loading spam!

## Discussion

### Two commands instead of one

**Note: Not true anymore, `use` will import environment**

Importing everything from a module would require two commands instead of one:

source-env spam.nu 
use spam.nu *

instead of just

use spam.nu *

One implication of requiring two commands is that you can still have environment error unchecked byt the parser, for example:

# spam.nu
export-env {
    let-env FOO = "foo" 
}

export def foo { $env.FOO }

If you don't `source-env spam.nu` prior to `use spam.nu`, the `foo` command would fail because there is no FOO environment variable brought in.

You could use `overlay use spam.nu`, though, to keep it in one command. The ovelrays in general would be a way how to "tie together" the environment and parser definitions of a module. We could think of ways to expand the overlays functionality to alleviare this pitfall, e.g., `overlay melt spam.nu` would take the `spam.nu` overlay but merge all its definitions into the existing overlay without bringing up the `spam` overlay at all.

### No more individual environment imports

Another disadvantage is that we lose the granularity of environment variable imports. It would no longer be possible to selectively import environment variables from a module one-by-one. However, I think the purpose of environment variables is to define some environment together. Being able to chip off some individual environment variables does not seem that valuable to me. Furthermore, the new `export-env { ... }` method is very flexible. You could add all sorts of logic to decide how the environment gets created, making it potentially more powerful than the current method. It seems more logical to me to treat module's environment as one piece instead of each env var individually.

`hide-env` should still be able to hide individual environment variables. That can still be useful occasionally if you need to hot-fix something.

## Expected Complications

### [Done] File-relative paths in the engine

`source-env` would need to support file-relative paths which would need to be tracked in the engine somehow. Currently, we have file-relative paths only in the parser, the engine still uses cwd everywhere. Maybe we should make the engine to trace the current file as well?

### `main` edge cases

Allowing `main` to define a command with the module's name has some edge cases:

# spam.nu

export def foo [] { 'foo' }

export def main [] { 'running spam!' }

# REPL

> use spam.nu       # defines the spam command

> use spam.nu foo   # does not define the spam command?

In the above case, the `spam` command is not defined. We could make any `use` of a module make to _always_ define the top-level command, if there is a corresponding `main` command inside the module.

Or automatically any `main` or `self` imports into the `spam` command:

> use spam.nu main       # defines the spam command

> use spam [ foo main ]  # defines foo and spam commands

or

> use spam.nu self          # defines the spam command

> use spam.nu [ foo self ]  # defines foo and spam commands

Also, I think

# spam.nu
def main [] { 'running spam!' }

> use spam.nu

should not define the `spam` command. You could still `nu spam.nu` which would run the `main`, though.

### Why `export-env { ... }`?

Instead of the `export-env` command, we could define `def-env export-env [] { ... }` and instead of `source-env spam.nu`, we'd call the `export-env` command to activate the environment.
This would work, however, if you called `use spam.nu *` and `use foo.nu *`, then the `export-env` commands would overlap. I imagine it would be quite confusing. I think the extra instrumentation required for the `export-env { ... }` would remove this footgun.

## Conclusion

With all these changes, environment handling in modules would hopefully become simpler, less error-prone, or at least not broken. The changes involving `source-env` would also bring in the long-awaited feature of being able to source dynamic paths while keeping the core of the command---defining an environment---still there. I hope you liked reading this text and if you read it all, you earned 100 Nu points. :elephant: