# $localize - tagged template design

Design a localization system which can be used for:
- `Goal-A`: Angular i18n or any other application/library.
*(Not directly tied to Angular projects)*
- `Goal-B`: Can be used without a compile step.
*(Easy to adopt, important for quick turnaround in dev mode)*
- `Goal-C`: Supports runtime translation evaluation.
- `Goal-D`: Supports compile time translation inlining.
- `Goal-E`: Supports message extraction for generating translation files

:::info
"translation evaluation" is the process of rendering a localized message with translated text at runtime.
:::

:::info
"translation inlining" is the process of replacing the original localization marker and message with a translated message, leaving no trace of the original i18n localization marker.
:::

### Additional design docs

* [Global import migration](https://hackmd.io/I6obNkZmQNKhw7KlgcgYHg)
* [Message id support](https://hackmd.io/33M5Wb-JT7-0fneA0JuHPA)
* [Legacy message id handling](https://hackmd.io/EQF4_-atSXK4XWg8eAha2g)

### Prior Work
- [Design: Angular translation service](https://docs.google.com/document/d/1h_y3mJ6kULNEoM2CymCgiqt5WMaLNm0cIkrhcLY30TY)
- [skolmer/es2015-i18n-tag: ES2015 template literal tag for i18n and l10n (translation and internationalization)](https://github.com/skolmer/es2015-i18n-tag)

## Summary of Proposal

$localize `:greeting:Hello ${name}:title:`;

`$localize` is a global [tagged-template handler](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#Tagged_templates) function that works by marking messages that need to be translated.

- The `$localize` function is not tied in any way to Angular and can be used independently of the Angular framework. (See `Goal-A`).
- The `$localize` function will work without any further processing. (See `Goal-B`) This is important for quick turnaround in development mode.
- The `$localize` function is global because we want to make sure that it can be easily identified in the output code even after passing through minification and obfuscation tools (such as web pack). This will make it easy achieve `Goal-D` and easily identify and inline messages when compile-time inlining.
- An extractor tool can be run over javascript files to collect all of the strings for translation. `Goal-E`

### Translations

The main point of tagging messages in code for translation is the ability to replace them with translated messages later.

Consider the following tagged message:

alert($localize`Hello World`);

#### Runtime evaluation

If we load a set of French translations like this:

loadTranslations({
  // Assume that 123456789 is the message id
  123456789: 'Bonjour le monde',
});

The alert would display `Bonjour le monde`.

The above example is referred to as `runtime inlining` and it supports `Goal-C`.

#### Compile-time inlining

An alternative is to process the bundled source code with a command line tool which identifies the `$localize` tagged strings and converts the code in-place.

Therefore the inlined code would look like:

alert('Bonjour le monde');

Notice:
- That `$localize` is completely removed as part of the compile time in lining.
- That the compile-time inlining can run on any input, but it is specifically designed to be able to take production output which is minimized and chunked into multiple files, such as from web-pack or rollup with terser.

## Detailed Design

### Source format

In its simplest form the translation string would be:

$localize `Hello ${name}`;

However when extracting localization strings it is often necessary to have description and attribute names. For this reason a more complete form would be:

$localize `:meaning|description:@@custom_id:Hello ${name}:placeholder_name:`;

The additional meta-information useful for translation is stored in "meta-blocks". This is done so that metadata is not removed from production build by the minifier. The implication of this is that the runtime evaluation and compile-time inlining processes must remove these metadata blocks.

### Pass-through behavior

It is important for the code to run without any additional processing. For this reason `localize` is implemented with a minimal runtime which needs to be present in any application containing `$localize` tagged messages.

This minimal implementation is attached to the global object by adding the following "side-effecty" import:

import '@angular/localize/init';

### Runtime translation

The minimal pass-through implementation checks for the existence of a method called `$localize.translate()`. If this exists then it will call it to get a translated version of the message.

This is the basis of runtime translation support.

$localize.translate(
    messageParts: TemplateStringsArray,
    expressions: readonly any[]):
      [TemplateStringsArray, readonly any[]];

:::info
A message to be translated consists of `messageParts`, which contain the static strings parts of a message, and `expressions` which are the values to be substituted in between each message part.

Both are passed to the `$localize.translate()` function because a translation may re-order the `expressions` for a particular locale.
:::

There is a basic runtime translation implementation provided. It is configured by importing and calling `loadTranslations()`.

import {loadTranslations} from '@angular/localize';

loadTranslations(localizationMap: {
  [messageId: string]: string
}): void;

Calling this function will ensure that the `$localize.translate()` function is initialized and configured to use the given translations.

### Command line tools

In addition to runtime the localization library provides a set of CLI tools which can perform extraction and infusion of the translated strings back into the source.

#### Translation message extraction

:::info
Extraction does not need to be implemented for v9.

For v9 the translation files can be generated by the Angular CLI (e.g. `ng xi18n`), which uses the current ViewEngine compiler to extract messages directly from templates.
:::

Extraction is performed with `localize-extract` binary. It is responsible for scanning the source files of the project and looking for the `$localize` tagged messages as described in [Source Format](#Source-format) section.

#### Compile-time translation inlining

Translation inlining is performed by the `localize-translate` binary. The command is responsible for searching the file for the `$localize` tagged messages and replacing them with the translated messages. This process completely removes the need for and `localize` function implementation.

alert($localize`Hello ${name}`);

will translate into

alert('Bonjour ' + name + '');

Notice that `$localize` has been completely removed in the process.

## Angular integration

Given an angular template such as this:

<h1 i18n>Hello {{name}}!</h1>

The Angular compiler generates code which includes this snippet.

var $I18N_7$;
if (ngI18nClosureMode) {
  // ... Google specific message localization
} else {
  $I18N_7$ = $localize`Hello ${"\uFFFD0\uFFFD"}:INTERPOLATION:!`;
}

## Implementation goals

There are a number of pieces to be implemented (or taken into consideration but deferred till after v9) as part of this `$localize` design. Some will be implemented by the Angular Framework team and some by the Angular tooling team. Below are the implementation goals for the Framework team.

### v9.0.0

| Feature | Description |
| -------- | -------- |
| **Passthru infusion** | Provide basic `$localize` implementation which only returns the string so that dev-mode does not need any additional processing. |
| **Compile-time infusion** | Implement a Babel plugin that can map localized message to a specified translation during compilation. This will be managed by the CLI. |
| **Update ngtsc** | Modify ngtsc to generate `$localize` tagged template expressions rather than calls to `ɵɵi18nLocalize()` |

### Post v9.0.0

| Feature | Description |
| -------- | -------- |
| **Message extraction** | Don't implement extraction. Just use existing template XLIFF we already have. This means that we will only be able to translate templates (not strings in source code). This is fine since v8 Works the same way. (VERIFY THIS WORKS) |
| **Run-time infusion** | Implement a version of `$localize` that supports loading translations at run-time and maps localized messages to the relevant translation. |

## Implementation details

The features are implemented in a reusable manner to allow them to be put together in different ways to support a variety of use-cases. For example, as stand-alone tools, or integration with build pipe-lines other than Babel.

### Folder layout

angular/angular/packages
  localize
    init/index.ts
    src/
      localize/
      tools/
      utils/
      translate.ts
    test
      ...

The folders are described below:

* `init/index.ts`: attaches the `$localize` function to the global object.
* `src/localize/`: the implementation of the `$localize()` function used in `init`.
* `src/tools`: contains the compile-time inliner and extraction tools
* `src/utils/`: contains reusable functions that are shared between the other parts of the library. Some may be exported publicly for 3rd party use.
* `src/translate.ts`: contains the `loadTranslations()` and `$localize.translate()` functions.

The `localize` folder is a top level Angular package (distributed as an npm package). `localize/init` is also an entry-point into the package.

## Outstanding questions

#### Code sharing

Some utilities are common to more than one Angular package. For example, `utf8Encode()` is used in `i18n` and `compiler`. How should we avoid code duplication of this code? I.E. how to share code (via Bazel depdendencies or npm package dependencies?)

:::info
Code that needs to be shared within `@angular/localize` is exposed as a secondary entry-point at `@angular/localize/utils`. This is the cleanest way to share code since the `run_time` entry-point is in Angular package format but the `compile_time` entry-point is a node.js package format.
:::

:::warning
Currently the `compile_time` entry-point relies upon code inside `@angular/compiler` to parse the translation files. So that package must be a peer dependency (or optional?). There is an open question about what to do about this.
:::

## Alternative proposals

### Deployment proposal (July 24)

The following proposal should provide a way to avoid having to include an import in application code. *It is subject to confirming some technical details for webpack.*

1) Always include a minimal viable (not just passthru) implementation of the global `$localize` function in `@angular/core` as standard.
> [name=Miško Hevery]
> - Including it in `@angular/core` may be too late, since it is possible for application call to call `$localize` before `@angular/core` is loaded.
> - I think in `ngDevMode` we should create `$localize` which will throw an error if someone access it with a message to add `$localize` shim. (This may be missed for abover reason.)
> - This also implies that `@angular/core` has side-effect, and and I am not sure what weird implications that would have down the line.
> - This also breaks the runtime i18n which requires that we need to load the messages before the application `.js` is loaded (because runtime i18n implies that we don't want to have multiple `.js` files)

2) During the production builds (in webpack), check for the existence of `$localize` calls in the entire program (i.e. all the bundles). If none are found then remove the global function from the program.

3) During compile-time infusion, remove the global function from the program.

This approach gives us the following benefits:

* There is no need to add an import for the global `$localize` in user code - ever! - not even in `polyfill.ts`.
* In development mode, there is no extra processing required to support i18n as the `$localize` function is always available whether or not you are using i18n.
* If there is no i18n being used (i.e. not `%localize` tagged strings) then the global function is stripped out in production mode - a form of tree shaking.
* If not explicitly providing translations, despite using `i18n` tags, even production builds will work as expected.
* If using compile-time infusion, the global function will be stripped out as part of the infusion process.
* Full run-time infusion is just the same as the pass-thru case, except that translations are provided through a loading mechanism.
* Application developers can use `$localize` to programmatically tag strings straight away with no further effort in the framework.

There is only one `$localize` function for all runtime scenarios (pass-thru and run-time infusion):

declare const $localize: Localize;
declare interface Localize {
  (messageParts: TemplateStringsArray, ...expressions: any[]): string;
  /**
   * A map of message/id to translated message. Used in runtime infusion to hold the translations.
   * */
  map: {[key: string]: string[]};
  /**
   * A function that will return a string identifier for the given messageParts
   */
  hash(messageParts: string[]|TemplateStringsArray): string;
}

This interface provides a `map` property and a `hash` method. The `map` holds the currently loaded translations. If there are none then `$localize` falls back to the pass-thru mode. This is a minimal implementation to support all pass-thru and run-time infusion modes.

Implementing runtime infusion in an application, is simply a case of loading up translations. The stategies for doing so are still TBD.

### extra script in a global fn proposal (July 24)

<html>
  ...
  <body>
      <script src="runtime.ca5745c8d8fbdbaf630b.js" type="module"</script>
      <script src="polyfills.60185d0cfe1893b52a44.js" type="module"></script>
      <script src="i18n-fr.893b52a4460185d0cfe1.js" type="module"></script>
      <script src="main.deadf89a6d615608cfde.js" type="module"></script>

  </body>
</html>

(function(global: any) {
  const $localize: Localize = function localize(messageParts: TemplateStringsArray): string {
    const id = $localize.hash(messageParts);
    const translation = $localize.map[id] || messageParts;
    let message: string = translation[0];
    for (let i = 1; i < translation.length; i++) {
      message += arguments[i] + translation[i];
    }
    return message;
  };
  $localize.map = {};
  $localize.hash = messageParts => messageParts.join('');
  return global['$localize'] = $localize;
})(typeof window !== 'undefined' && window || typeof self !== 'undefined' && self || global);

### takeways
- use side-effecty import to load `$localize`
- use side-effecty import to load `$localize.load` for runtime localization of strings.
- placeholders are passed as expressions
- for new users it is reasonable that we ask them to install a new package
- installation should be as simple as `ng add @angular/localize` and include modification of the polyfills.ts
- and possibly also configuration of the infusion step in the build pipeline (angular.json) - this might be a separate schematic that would require `ng generate` invocation rather than a default one invoked by `ng add`.
- we need to ensure that we give a compiler error instructing the user to `ng add @angular/localize` if the shim is not loaded
- `ngtsc` can do this by detecting if $localize type is defined in the global scope
- we should also give a runtime error if the shim is not loaded
- `@angular/core` in `ngDevMode` adds `$localize` to global scope (if not already defined), and this implementation will throw error if someone tries to use it instructing them ho `ng add @angular/localize`.
- for existing projects we need to detect if the project uses i18n and if it does invoke `ng add @angular/localize`
- we can detect i18n usage by scanning metadata of application ~~(and possibly node_modules)~~ and look for `/\wi18n\w/` expression.
- we might need to modify angular.json to teach users about how to use the new localization pipeline
- for existing projects that are upgrading we should try to convert the existing angular.json with i18n build config to the new pipeline (or ensure that the old pipeline invocation is still compatible).

### Non-global proposal

Building on an [idea by Trotyl in GitHub](https://github.com/angular/angular/pull/31609#discussion_r311094455)...

The idea is that we can avoid having to use a global in the source code by providing a two-step compilation process for compile-time inlining.

:::info

**Conclusion**

We have decided not to adopt this proposal and to stick with the plain global function design.

**Publishing libraries**

The introduction of two steps would cause problems with publishing libraries (e.g. to npm) that support i18n.

Since the library would be bundled, the first step would need to be run before publication - leaving the global marker in place - which would then prevent the run-time local import approach from working. The only way around this would be to fall back on the global approach, which is what we are proposing already.

**Complicated build pipe-line**
Further, this approach requires each application project's build pipeline to be aware of the two steps. While we could build a TS/webpack based approach, we are unlikely to provide other build tool plugins and also it would require the application developer to understand this tooling to configure their build appropriately even if they were available.
:::

#### An example

Source code before compilation:

import {$localize} from '@angular/localize';
alert($localize`Hello World`);

Source code after TS compilation [step-1] (i.e. in ES2015 bundle):

alert(ɵlocalize`Hello World`);

Source code after i18n inlining [step-2] (i.e in locale specific bundle):

alert(`Bonjour le monde`);

#### Run-time/pass-through localization

To use localization in your app:

* Add a normal `import {$localize} from '@angular/localize';` to source code that requires i18n (i.e. user code and in our generated templates).
* This import actually refers to the run-time localization code.
* If you do nothing else then you just need to add a dependency on `@angular/localize` to your package.json and you get run-time inlining.
* The template compiler would add local imports of `$localize` as necessary where `i18n` tags have been found in the template

#### Compile-time inlining

To do compile-time inlining:

* Turn on a flag in ngc (similar to `enableIvy`, say `inlineI18n`).
* A TS transform will convert all the locally imported `$localize` identifiers into a global identifier (i.e. it would go through and rename all `$localize` identifiers with some global like `ɵlocalize`) and also remove all the `import {$localize} from '@angular/localize';` statements).
* this global `ɵlocalize` has no implementation but acts as a marker that finds its way through the compilation process untouched by minifiers etc into the bundles.
* Run the actual compile-time inlining on the bundles containing the global, just like we planned already.

#### Benefits

This will result in the following benefits:

* There is no need to add anything to `polyfill.ts` or other files outside of CLI setups.
* If you don’t use i18n then nothing has to change in your app or build setup.
* backwards compatibility for v8 projects
* if you use `i18n` tags in your templates, then you just need add `@angular/localize` to your package.json dependencies and you have run-time/pass-through localization with no more work.
* minimal change to v8 projects
* if you want to use compile-time inlining then you just turn on the `inlineI18n` flag and ngtsc/CLI work together to generate the bundles with no trace of the original `$localize` code.
* no extra deployment cost (i.e. in polyfills.ts) for projects that use compile-time inlining.
* It is possible to support i18n in multipe Angular apps on a single browser page using run-time inlining.
* Each app will get its own copy of the runtime `$localize` code.

#### External users

For non-Angular users of `@angular/localize`:

* The `@angular/localize` is a required dependency - but no other Angular packages.
* This package will contain the run-time localization plus the utilities for doing compile-time inlining.
* Run-time inlining works out of the box.
* Just add `import {$localize} from '@angular/localize';` and build as normal.
* Compile-time inlining requires the two-step compilation process
* First step:
* if compiling with TypeScript then add the `ts.Transform` used in ngtsc to the compiler configuration (published as part of `@angular/localize`).
* if not using TS (e.g. compiling with Babel) then add a Babel plugin to the compiler configuration (possibly published by Angular team).
* Second step:
* the second step is the Angular agnostic inlining tool (i.e. a wrapper around a Babel plugin) so this can be reused as-is.
* If there is no compiler (e.g. plain JavaScript), then steps 1 and 2 can effectively be joined together to provide compile-time inlining in a tool that runs directly on the source JS.