# Localization Changes

# Goals
In order of importance:
1. Maintaining translation files is too hard. Translation files shouldn't require any effort to start using for normal content.
- There is no process to generate them from existing content
- There is no process to update them with new content
- Migrating to using them to support localization is a large endeavor

2. Contributing translations or making a translation mod is hard:
- Extracting translations from closed source mods will let translators translate mods.
- Testing translations needs to be easier, since custom translation keys sometimes require guesswork for translators (context of where the key is used is not immediately apparent)

3. Translation keys are verbose and inconsistent, they need to be more consistent and nested below the content they correspond to rather than nested below other concepts:
- Current key patterns follow vanilla patterns, which separates translations inherantly bound together:
* `ModItem` currently has `DisplayName` set to `Mods.ModName.ItemName.ClassName` and `Tooltip` set to `Mods.ModName.ItemTooltip.ClassName`
* Town NPC have translations all over: `Mods.ModName.TownNPCMood.NPCName`, `Mods.ModName.Bestiary.NPCName`, `Mods.ModName.Dialogue.NPCName`

4. Too much indentation. Despite hjson being simple syntax, hjson files can look ugly and disorganized with multiline strings and inconsistent tabs and spaces.

5. Simplify backend code:
- Remove `ModTranslation` class.
- All content now only know how to retrieve their associated translation key, they do not store the translation values themselves.
- Use `LocalizedText` instead, use vanilla approach to loading translations (only load and keep the current language in memory)
- Automatically handle `LocalizedText` key construction and default value generation.

# Proposed Implementation
Each item corresponds to the numbered goal above:

---

tldr: Code based translations are no-longer supported (`ModTranslation.SetDefault`), all localized text must have a key in an English `.hjson` file. All localization files will be automatically populated.

---

1. After building a mod, all localization files for all languages supported by the mod will be automatically populated with all values after mod loading finishes:
- tModLoader will track a list of recently built mod names. After mod loading, if a mod contained in that list matches a recently built mod, the localization file updating process will occur for that mod. The name is then removed from that list.
- Each localization file will be loaded into memory collecting known keys. This mapping is then augmented with all translation keys from all bits of content. For example, each loaded `ILoadable` will be queried for properties with the type `LocalizedText`. This will capture automatically localized entries such as how a `ModItem` called `ExampleSword` becomes "Example Sword" during loading.
- The automatic localization will behave as it has before. For example tooltips will be empty strings if not found and `ModItem` display names will be the classname with spaces added between capital letters.
- This also means that modders can define their own `LocalizedText` properties in their own classes and use them directly without extra effort initializing them.
- Once all known keys are collected, each set of localization files is updated with missing keys.
- Every localization file will follow the layout of the default/English file.
- Only languages with an existing .hjson file will be updated. Modders can start supporting a new language by creating an hjson file for that languages and rebuilding the mod. For example, creating `ru-RU.hjson` and rebuilding will results in `ru-RU.hjson` being populated.
- Missing keys will be added to the first file found that contains the corresponding "Prefix", or to the default file if not found. ("Prefix" meaning everything before the final `.`)
- Non-English hjson files will be populated with commented out entries, signifying that they are yet to be translated and will fall back to English.
- When writing out the files, the layout and ordering will follow the default/English layout. All entries missing from the original file will be added to the end of their respective "Prefix", all new "Prefix" will be added to the end of the file.
- A modder can reorganize the English localization files and rebuild the mod. This will cause all the other localization files to update to the new organization.
- For example, adding a new `ModItem` to the game will add that item to the "Mods.ModName.Item" prefix in the default localization file. If the modder had a specific localization file with the "Mods.ModName.Item" prefix in it, it would end up there instead.
- Entries in the hjson files will not be populated by dynamic values assigned via `Override`, but rather their original dummy value from their entry in the hjson file. `Override` is meant to support in-code translations at load time.
- Comments: Comments from English .hjson files are preserved attached to the entry they are immediately above. They will end up in the non-English files attached to the same entry.
- Discussion: Should we attempt to preserve comments in non-English localization files? How should that be done? Some use comments on hjson files to credit the translator.
- Spacing: 1 line between each prefix
- This approach will require a mod build and reload to start localizing, but there is no suitable alternate for this due to dynamic content.
- Existing mods will need to be rebuilt and published to allow localization mod makers to easily make localization mods, but we expect 1.4.4 to require this of most mods anyway, so this is not an issue.
- If no changes are detected, the file won't be touched to keep file modified date data.
- Mods with dynamically loaded content should ensure to load the mod with all content enabled to populate the .hjson files. Alternatively, we could update the hjson files every load rather than just after being built.

2. Various features to facilitate translating mods
- Wiki guide to show all this
- Full mod example. (Just adding Mod.OtherMod keys should work, right?)
- Example will use sortAfter to ensure it overrides the existing values, if any.
- Button to extract hjson files for current language
- The extracted hjson files will be complete and suitable for localizeing. This will work even if the mod didn't originally support this language.
- Extraction will not be preventable by original mod maker. The file layout will mirror the file layout in the original mod sources folder exactly.
- The intention is that the hjson files should be drag and drop replacments that the original mod maker can directly use, if they want to accept localization contributions
- Live translation updating
- Modders Toolkit feature to watch a file for changes and to apply them in game.
- Won't work for already retrieved things, like UIText.
- Some way to toggle displaying original keys in their in-game usages. Maybe Modders Toolkit can MonoMod edit `LocalizedText.Value` to return "Value ({Key})" so translators can see the exact key in usage? Only useful for custom keys used in UI or other places, probably needs more discussion.

3. We will reorganize with this pattern: `Mods.ModName.Category.ContentName.DataName`. There are 2 options, having the `ILoadable` classname be the `Category` or the `ILoadable` class containing a `Category` property.
- Each `ILoadable` will define a `Category`, and all translations contained within a `ModType` will be contained within that `Category`. `ContentName` is the internal name of the piece of content. For example, `ModItem` will set `Category` to "Item". So `ExampleItem` and `ExampleSword` will appear as:

        Item: {
            ExampleItem: {
                Name: Example Item
                Tooltip: Just an example.
            }
            ExampleSword: {
                Name: Example Swooord
                Tooltip: Very sharp, watch out.
            }
        }

- Alternatively, instead of `ModType` setting a `Category`, the `Category` is derived directly from the `ModType` classname, so instead of "Item" it would be "ModItem".
- Other custom keys can still be whatever they want and will still load and be preserved in the localization files.
- Another potential feature is customizing the level of nesting in the hjson files. For example, we could have translation automatically not use object syntax when there are 2 or less entries in the prefix:

        Item: {
            ExampleItem.Name: Example Item
            ExampleItem.Tooltip: Just an example.
            
            ExampleSword.Name: Example Swooord
            ExampleSword.Tooltip: Very sharp, watch out.
            
            ExampleGun: {
                Name: Example Gun v2
                Tooltip: Talks to you 
                # OutOfAmmoMessage in this case is a LocalizedText defined in the ExampleGun class: public LocalizedText OutOfAmmoMessage => GetLocalizedText(nameof(OutOfAmmoMessage));
                OutOfAmmoMessage: Feed me more delicious bullets please
            }
        }
        
        Common: {
            Message1: Hellow
            Message2: Hi
            Message3: Welcome
            Message4: Bye
        }

4. We plan to somehow allow the `Mods.ModName` to be ommited from hjson files, allowing 2 tabs of indentation to be recovered. There are 2 suggested options:
- "Mods.ModName" will still be the key, due to potential collisions with existing Terraria translation keys. Most likely the filename being "en-US-Mods-ExampleMod.hjson" or some other pattern will facilitate this. If more compartmentalization is desired, "en-US-Mods-ExampleMod-Item.hjson", for example, could contain all ModItem translations without needing to specify that or any indentation.
- For example: en-US.hjson

            Mods: {
                ExampleMod: {
                    Item: {
                        ExampleSword: {
                            Name: Example Swooord
                            Tooltip: Very sharp, watch out.
                        }
                    }
                }
            }

- For example: en-US-Mods-ExampleMod.hjson

            Item: {
                ExampleSword: {
                    Name: Example Swooord
                    Tooltip: Very sharp, watch out.
                }
            }

- For example: en-US-Mods-ExampleMod-Item.hjson

            ExampleSword: {
                Name: Example Swooord
                Tooltip: Very sharp, watch out.
            }

- Another possibility is that we could blacklist the vanilla keys for for mod names to get rid of "Mods." prefix altogether. These keys include various values that could potentially make useful mod names: "BestiaryInfo", "WorldGeneration", "CreativePowers", "Announcement", "Currency", etc. In all there are 219 names that would need to be blacklisted, and later Terraria updates could add more, which might invalidate an existing mod name.
- Feedback or a poll will help us decide this.

5. Backend code
- Remove `ModTranslation` completely
- Rather than load all language files into memory (into `ModTranslation` objects) and select which to use when refreshing languages, we will follow the vanilla code more closely and load just the selected language.
- All existing `ModTranslation` will be replaced with a `LocalizedText`
- Modders will no longer be able to assign translations in code as shown in [ExampleBlock.cs](https://github.com/tModLoader/tModLoader/blob/1.4/ExampleMod/Content/Items/Placeable/ExampleBlock.cs): ~~`DisplayName.AddTranslation(GameCulture.German, "Beispielblock");`~~
- In `LocalizationLoader.RefreshModLanguage`, first the default/English localization files will be loaded into a Dictionary followed by the current language, if different.
- This is needed to ensure that `Language.GetTextValue` falls back to English rather than the key for a non-localized mod
- Each LocalizedText attached to content will be updated and various `Lang` caches updated as currently done.
- Dynamic localization:
- Modders using dynamic values must have a stub entry in hjson files. Creating new keys not represented in the localization files will not be allowed, but assigning a override value to a key will be allowed.
- Any dynamic translation values will be assigned via `LocalizedText.Override` during `Mod.SetLanguage`, which will be called during `LocalizationLoader.RefreshModLanguage` with the current culture as a parameter.
- New keys will not be allowed to be added from this method.
- For example:

                # hjson file
                SpawnConditionsMessage: Defeat the {0}! 

                public class MyMod : Mod {
                   public override void SetLanguage(GameCulture culture, Dictionary<string, LocalizedText> localizations) {
                       if(ModLoader.TryGetMod("Tremor", out Mod Tremor)) {
                           localizations["Mods.ExampleMod.SpawnConditionsMessage"].Override = string.Format(localizations["Mods.ExampleMod.SpawnConditionsMessage"].Value, Lang.GetNPCNameValue(Tremor.Find<ModNPC>("WallOfBones").Type));
                       } else {
                           localizations["Mods.ExampleMod.SpawnConditionsMessage"].Override = string.Format(localizations["Mods.ExampleMod.SpawnConditionsMessage"].Value, Lang.GetNPCNameValue(NPCID.WallOfFlesh));
                       }
                   }
                }

- We need more examples and current usages of dynamic localizations so that we can plan on implementing this feature correctly.
- Effectively, the text returned will be in this order:
1. Override text:
- "The 1245th sword item in the game"
2. Current language localized text:
- "El Example Sword"
3. English localized text (automatically generated for new content):
- "The Example Sword"
5. Key itself:
- "Mods.ExampleMod.ModItem.ExampleSword.DisplayName"
- This can only happen when attempting to translate a key that was not discovered by the automatic localization generator, eg because it was not present on an `ILoadable` instance at load time
- `LocalizationLoader.translations` removed, all localizations are loaded during `RefreshModLanguage` so there is no need to store modded `LocalizedText`.
- Automate and streamline ~~`ModTranslation`~~ `LocalizedText` key binding and default localization values:
- All `ModTranslation` in `ILoadable`s replaced with `LocalizedText` getter:

            Old:
            public ModTranslation Tooltip { get; internal set; }
            New:
            public LocalizedText Tooltip => GetLocalizedText(nameof(Tooltip))

- `GetLocalizedText(this ILocalizedModType self, string suffix)` is an extension method that will return the LocalizedText instance from `LanguageManager.Instance._localizedTexts` derived from classname, mod name, and `ILocalizedModType` category.
- For `LocalizedText` in non-`ILocalizedModType` instances or other situations, modders can use `Language.GetText` directly:
- `public LocalizedText BossMessage => Language.GetText("Mods.ExampleMod.Common.BossMessage")`
- Derive key category from `ModType` (`Category` property in `ILocalizedModType` or just `nameof(ModType)`)

            class ILocalizedModType : IModType
            {
                public string Category { get; }
            }

- Modders exposing a ModType can add `Category` to it by implementing `ILocalizedModType`, otherwise it will be based off of the `ModType` classname
- Derive key field from `LocalizedText` property name. In this case, "Tooltip" will be part of the key:

            public LocalizedText Tooltip => GetLocalizedText(nameof(Tooltip))

- For example: A `ModItem` class called `ExampleSword` will have "Mods.ExampleMod.ModItem.ExampleSword.DisplayName" as a key since the `ModItem` category is `ModItem` and the `LocalizedText` property is called `DisplayName`.
- Allow ModTypes to define their own default localization values:

            [DefaultLocalizedValue(typeof(DisplayNameGenerator))]
            public virtual LocalizedText DisplayName => GetLocalizedText(nameof(DisplayName));

            [DefaultLocalizedValue("")]
            public virtual LocalizedText Tooltip => GetLocalizedText(nameof(Tooltip));

            [DefaultLocalizedValue("Bestiary text for {DisplayName}")]
            public virtual LocalizedText BestiaryText => GetLocalizedText(nameof(BestiaryText));

- This will control how hjson files are populated, and can simplify code for tModLoader ModTypes and other ModTypes.
- All existing code using `LocalizationLoader.GetOrCreateTranslation` and `DisplayName.IsDefault()` will be automated via this system and applied during `RefreshModLanguage` before localization caches are updated.
- Example: ModItem cleanup:
- Removed from ModItem.Register:

                    DisplayName = LocalizationLoader.GetOrCreateTranslation(Mod, $"ItemName.{Name}");

- Removed from ModItem.AutoStaticDefaults:

                    if (DisplayName.IsDefault())
                        DisplayName.SetDefault(Regex.Replace(Name, "([A-Z])", " $1").Trim());

- Added to ModItem:

                    [DefaultLocalizedValue(typeof(DisplayNameGenerator))]
                    public virtual LocalizedText DisplayName => GetLocalizedText(nameof(DisplayName));
                    
                    public static string DisplayNameGenerator(ModType modType) {
                        return Regex.Replace(Name, "([A-Z])", " $1").Trim();
                    }

# More detailed code examples

1. Nothing for modders to change. The resulting hjson files will look like current examplemod hjson files, just automatically populated and formatted as mentioned. Modders can opt to organize how the the English files are organized and the other languages will automatically update on build and reload.
2. Every mod will have to publish an update for the full translation templates to be available to translation mod makers. We expect this to be required for 1.4.4 anyway.
3. Modders will have to move translations around and fix the keys to fit the new patterns
- For example: All items will need their display name and tooltip together
- Old Approach

            Mods: {
	            ExampleMod: {
                    ItemName: {
                        ExampleItem: Example Item
                        ExampleSword: Example Swooord
                    }

                    ItemTooltip: {
                        ExampleItem: Just an example.
                        ExampleSword: Very sharp, watch out.
                    }
                }
            }

- New Approahc:

            Mods: {
	            ExampleMod: {
                    Item: {
                        ExampleItem: {
                            Name: Example Item
                            Tooltip: Just an example.
                        }
                        ExampleSword: {
                            Name: Example Swooord
                            Tooltip: Very sharp, watch out.
                        }
                    }
                }
            }

4. Modders can optionally use the feature to allow `Mods.ModName` to be ommited from hjson files if they desire to. Renaming `en-US.hjson` to `en-US-Mods-ExampleMod.hjson` and removing the top 2 and last 2 lines as well as 2 tabs of indentation should be useful for most modders.
- Actual filename pattern TBD. Other suggesion: use periods instead `en-US_Mods.ExampleMod.json`
- Newly generated mods and mods without existing .hjson files will automatically follow this convention.
5. Examples pertaining to backend changes are detailed in `Proposed Implementation` above.

# Long term roadmap
These are additional ideas building off other planned improvements. Some of these might not happen.

- A selection of common tooltips mirroring Terraria tooltips that can be customized:
- https://github.com/tModLoader/tModLoader/pull/1126
- https://github.com/tModLoader/tModLoader/issues/540

- Support for Lang.CreateDialogSubstitutionObject
- Currently impossible to mention hotkeys and mod npc names, except in cases where the modder can format the string manually.
- Can't do that in most situations: GameTips
- Translations lack the ability to 'bind' values (such as current damage), instead, the site of the translation must pass in the values and all translations must follow the same format.
- Potential implementation:

    - Need a way to do similar things in-game possibly? Need more feedback
- Localize Chat commands
- New language support
- Allow mods to specify a non-English default language
- Workshop localization support: Translating mod display name, categories, and description
- Add actual resource pack support to the tModLoader workshop. Augment resource pack support to allow overriding modded resources.
- Mod-contained resource packs
    - Language tweaks along with texture tweaks can be contained in a special folder that will be used in the same manner as existing resource packs, allowing them to be selectively applied.

- Allow modder to define key, similar to `ModItem.Texture` property override
    - For example, imagine you have 20 `BoxingGlove` items in your mod all with the tooltip "Great for boxing with!". In the current design, it'll look like this:

Item: {
CopperBoxingGlove: {
Name: Copper Boxing Glove
Tooltip: Great for boxing with!
}
SilverBoxingGlove: {
Name: Silver Boxing Glove
Tooltip: Great for boxing with!
}
IronBoxingGlove: {
Name: Iron Boxing Glove
Tooltip: Great for boxing with!
}
// and so on...
}

    - Currently, you can use the text substitution feature to do this:

Common: {
BoxingGloveTooltip: Great for boxing with!
}

Item: {
CopperBoxingGlove: {
Name: Copper Boxing Glove
Tooltip: {$Mods.ExampleMod.Common.BoxingGloveTooltip}
}
SilverBoxingGlove: {
Name: Silver Boxing Glove
Tooltip: {$Mods.ExampleMod.Common.BoxingGloveTooltip}
}
IronBoxingGlove: {
Name: Iron Boxing Glove
Tooltip: {$Mods.ExampleMod.Common.BoxingGloveTooltip}
}
// and so on...
}

    - This proposal allows the modder to override the path for specific translation keys by overriding LocalizedText:

public class ModItem {
public virtual LocalizedText DisplayName => GetLocalizedText(nameof(DisplayName))

public virtual LocalizedText Tooltip => GetLocalizedText(nameof(Tooltip))
}

public abstract class BoxingGlove : ModItem {
public override LocalizedText Tooltip => Language.GetText("Mods.ExampleMod.Common.BoxingGloveTooltip");
}

Common: {
BoxingGloveTooltip: Great for boxing with!
}

Item: {
CopperBoxingGlove: {
Name: Copper Boxing Glove
}
SilverBoxingGlove: {
Name: Silver Boxing Glove
}
IronBoxingGlove: {
Name: Iron Boxing Glove
}
// and so on...
}

    - The code for updating .hjson files would need to not update the "missing" entries. As it currently is written, it would just need to make sure to query the `TooltipKey` property of the loaded content rather than rely on hardcoded paths.