# Javascript repo agent - written in Go This is a proposal for an FHL project to prototype a repo agent written in golang. This is all very speculative and I have been working to get things off the ground to test viability for a range of scenarios. **NOTE: Open to suggestions on...anything really. Names, approach, whatever. Feel free comment or put things inline. If you make changes inline please mark your alias with the insertion** ## What The repo agent is a golang tool which will be able to perform a number of tasks: - Effectively do the work of lage, to execute cascading tasks in a repo to build, test, lint, etc. - Run in a server mode, listening for file changes and triggering rebuilds as appropriate - Handle extra tasks like microbundle creation (to work with cloudpack) or orchestration tasks - Run as an http server to be able to serve up packages, bundles, files, etc. ## Why(s) ### Potential use cases - Speed up builds beyond what lage can do, particularly when chaining to native commands rather than invoking node commands - Leverage the server to act as a haste map implementation, suitable for injection into jest tests and bundling react-native via metro. This is a known perf bottleneck in metro bundling at least and similar to what FB does internally. - Work as a custom resolver to speed up resolution in bundling - Act as a watch mode for ts->js builds to avoid using ts-jest and ts-node/babel-loader in testing/bundling - Act as an extensible orchestrator for various actions that require understanding of repository layout and package dependencies (without requiring every package to do this on its own) - ??? - Profit ### Why Go(lang) Instead of Rust This one is a bit more arbitrary. I honestly am not particularly knowledgable about rust, but my sense of the tradeoffs is: - Rust is slightly more optimizable, but slightly harder/slower to write - Go has better library support for things like websocket and http servers so we don't have to do everything by hand - Go seems slightly easier to interop with JS stacks. We have the esbuild codebase to look at for ideas, and the author of SWC is reworking TSC in Go (despite SWC being RUST) That's really it. ## Where ### Where is it being built It's currently being built in a branch in the rnx-kit repo. It's convenient to build it in a JS monorepo to be able to easily test the changes in place and leverage some of the existing infrastructure. Longterm it can go wherever is most appropriate. Likely in the common tools repo that we are building. ### Where are we right now I've been tinking with it getting certain bits up and running over the last week as it is hard to parallelize efforts until initial scaffolding is done. - Right now it works as a lage replacement with no caching - It can load and parse workspaces and lage config (from root package.json) and correctly execute various build commands at the repo or specific package scope. - It builds successfully and is about 10-15% faster than lage in brief testing. Sometimes more than that but I haven't spent much time tuning. - I will be trying to get the basic JS <-> Go connection via websocket done this week just for simple commands. - It does work in both Windows and MacOS ## Proposed Architecture While it is currently just a command line executable golang program I will be moving it to an architecture where the golang program boots up and establishes a websocket which various JS clients can connect to. This websocket connection is for IPC and for having multiple client connections. (You can imagine a root execution call creating the server and then a jest.config.js file creating a haste map connection) ### Things to do In no particular order: - [ ] Add matching JS module and front end - [ ] Websocket connection between JS and Golang back-end - [ ] Figure out official build/publishing pipeline for native builds - [ ] Add http server support - [ ] Prototype and try out resolver feasibility - [ ] Prototype and try out haste map implementation - [ ] Prototype and try out JIT JS files in bundling (module resolutions -> https://localhost:XXXX/packages/\<packageName\> -> just in time builds of JS files) - [ ] Colored output strings - [ ] Correct iostream routing and reporting for build commands - [ ] Rebuild based on invalidations/file changes - [ ] Integrate caching layer of some sort - [ ] Integrate distrubted caching layer of some sort - [ ] Whatever else people want to experiment with Ultimately I don't have a particular agenda to try out. This type of thing seemed like a common way of approaching a lot of the things I've seen in the 1Bundler discussions for optimizations and like some of the things David is already doing in Cloudpack.