The Monorepo Import Problem

Import generation in Ivy

Ivy requires the insertion of new imports into the user's code. This is a result of NgModule expansion, where the individual directives used by a component are inserted into the directiveDefs property of an ngComponentDef. The component never imported these directives, but instead was imported by an NgModule.

For example, consider the following simple program:

// app.component.ts:
import {Component} from '@angular/core';

@Component({
  selector: 'app-cmp',
  template: '<div *ngIf="expr">expr is true</div>',
})
export class AppCmp { expr = true; }


// app.module.ts:
import {CommonModule} from '@angular/common';
import {NgModule} from '@angular/core';

@NgModule({
  declarations: [AppCmp],
  imports: [CommonModule],  
})

The user never explicitly writes an import to NgIf, either in app.component.ts or in app.ngmodule.ts. However, the compiler analyzes this program and concludes:

AppCmp belongs to AppModule.
AppModule has a compilation scope (a set of directives visible in the compilation of components in a module) which contains AppCmp, NgIf, and NgFor.
AppCmp uses the NgIf directive from its scope, as it matches the *ngIf="expr" syntax in its template.
AppCmp's ngComponentDef.directiveDefs should list NgIf.
An import must be generated for NgIf in app.component.ts in order to do this.

Determination of the module specifier

How should NgIf be imported from app.component.ts? There are a few choices:

via a relative import:

import {NgIf} from './node_modules/@angular/common/src/directives/ng_if';

via an absolute, deep import:

import {NgIf} from '@angular/common/src/directives/ng_if';

via an absolute, entry-point import:

import {NgIf} from '@angular/common';

Option 1 is obviously incorrect - relative imports into node_modules are a bad practice. Option 2 is also incorrect - deep imports like this are broken when running a bundler such as Webpack, as they point to a specific code format (e.g. ES2015) while the bundler might prefer to operate on different code format published in the same package. Option 1 also suffers from this problem.

Option 3 is the right answer, but it has problems of its own. It's purely an Angular convention that NgIf is re-exported from the entry point @angular/common. This convention is specified in the Angular Package Format (APF).

Today, if the NgModule was imported via an absolute import (in this case @angular/common), the compiler assumes that the package follows the rules of the APF, and that all the directives and components in the NgModule can be imported from the same entrypoint.

If the NgModule was imported via a relative import, then it's within the user's program and not node_modules, and a relative import (option 1) is correct.

Monorepo libraries

The above logic works well if the user's project has a simple structure, and is separated into user code and APF-formatted libraries. However, this assumption does not always hold.

For example, in more complex CLI projects (or those using NX Workspaces), the user's code may be split into multiple independent compilations. From here on, they will be referred to as "workspace libraries". There may also be application compilations which depend on the workspace compilations.

A major problem occurs when considering how the compiler should import from workspace libraries, both within the application bundle and within other workspace libraries.

Nature of workspace libraries

At the heart of this issue is how the library and application code will be linked together at runtime. There are two possibilities:

they will be immediately fed to a bundler (like webpack) and built into an application bundle that contains both the application code and the library code.
the libraries will be published on NPM, and used in the imports of another application.

Either or both of these cases can be true.

If case 2 is true, it is essential that the library be published in the APF, and that imports to it go through the entrypoint. Otherwise they end up running afoul of the problems described in Determination of the module specifier above.

However, complying with the APF is a burden, and so an application which is simply attempting to break its codebase into multiple compilations will likely not want to pay the cost of using the APF for each sub-compilation. This is okay, since the sub-compilation output will never be published on NPM, only immediately fed to a bundler.

Nature of imports

The issue arises because these different cases necessitate the use of different imports by consumers of the above libraries.

Consider the following import:

// Component `Foo` used from 'foolib'.
import {FooModule} from 'some/libs/foo';

If some/libs/foo is intended for distribution on NPM, then Foo should be imported from 'some/libs/foo' directly. If some/libs/foo is instead a sub-compilation which will be immediately bundled into an application, then perhaps some/libs/foo doesn't actually export Foo and it needs to be imported from some/libs/foo_component.

Without knowing how some/libs/foo is intended to be consumed (either on NPM or not), it's impossible for the compiler to know how to generate these imports.

Complications of path mapping

In the sub-compilation case where the workspace library will never be distributed on NPM, a deep import format should be used.

Correctly selecting a deep import path can be problematic. It's entirely possible that the target workspace library lives outside of the current compilation's root directories, but that all or parts of it are path-mapped in.

Path mappings are designed to convert in one direction, from an abstract path such as some/libs/foo to a filesystem-based module path. However, starting from one abstract path some/libs/foo and knowing how to reach some other nearby file is not trivial.

Examples

The Angular monorepo itself is a great example of a repository with multiple interdependent compilation units (@angular/core, @angular/common, etc), where each compilation unit is a library in APF format which is intended to be published on NPM.

When @angular/common imports from @angular/core, it should use entrypoint imports and not deep imports, as deep imports are disallowed both in and into NPM published artifacts.

Proposed solution

Since the fundamental differentiator between the two kinds of workspace libraries is the intent of their usage, this is what the compiler must capture.

Problem with `.d.ts` metadata

A simple solution would be for the compiler to record somehow in a library's .d.ts files that the library is not in APF and deep imports should be used for its classes.

However, this runs into a significant roadblock: often when invoking downstream compilations, the upstream library is depended upon via a direct pathmapping to the original .ts source files, not by including the .d.ts files produced by earlier compilation. The outputs of the multiple compilations don't actually meet until the bundling step. As a result, any metadata written by the compiler for a library is unavailable to make import decisions in later compilations involving a dependency on that library.

Configuration

This leaves configuration of downstream compilations as the only viable option.

A new Angular compiler option privateLibraryPaths will be introduced, with a type of string[]. It will accept the same format as paths in the TypeScript compiler, with the intention that some (or all) mapped paths are also listed as private libraries. A convenience value of 'all' will take the paths as all being private, without the need to repeat them.

Pete Bacon Darwin

2019/07/12 11:12:04

Should "workspace compilations" be "workspace libraries" here? (Edited)

2019/07/12 11:17:33

The comment in this code snippet seems to be out of line with the code. (Edited)

Guest Benson2019/07/15 19:18:24

Instead of `all`, would it be possible to support wildcards like `*`, where specifying only `*` would match all `paths` mappings? (Edited)

Joost Koehoorn

2019/07/15 19:19:03

That guest reply was me! :-) (Edited)

Guest Nichols2019/07/24 04:30:27

Sounds to me that this similar to TS Projecr References. (Edited)

Alan Agius

2019/07/24 04:32:54

Sry for the spam above. This sounds like Ts project references. And reason why not use that in the first place? (Edited)

2019/07/24 05:56:28

Nvm, ts project references use relative paths. (Edited)

2019/07/24 05:57:10

How would this work for intellisense in IDE's, since one would not have path mappings nor a pre build library? (Edited)

2019/07/24 06:01:27

Yeah indeed this is problematic, in ng-packagr we go around this, internally we add path mapping to `dts` files for intra dependent compilation. So that we don't include `.ts` sourcefiles into downstream compilations https://github.com/ng-packagr/ng-packagr/blob/f581c7f2a1800864b05566a4ea347ac4d28580f5/src/lib/ts/tsconfig.ts#L115 So that for example when having: `@foo/bar -> @foo/bar/testing` So for example when having: ``` path: { "@foo/bar/*": [ "src/foo/bar/*" ] } ``` (Edited)

2019/07/24 06:01:31

During the compilation it will become ``` path: { "@foo/bar/*": [ "src/foo/bar/*.d.ts", "src/foo/bar/*" ] } ``` (Edited)

Alex Rickabaugh

2019/07/24 06:37:07

Alan, great question! This doesn't actually affect intellisense in the IDE at all. The IDE understands the project perfectly, because it has the user's path mappings and can decipher the imports the user wrote. (Edited)

2019/07/24 06:38:14

This is purely a problem for the Angular compiler, because it has to generate a new, never-before-seen import to a class (directive) the user never mentioned, and it doesn't know whether the user wants to use entrypoint paths (if the library is APF/published on NPM) or deep paths (if the library is "private"). (Edited)