# Prebuilt filecoin-ffi design

Please leave comments via the rendered view.

### Problem

Right now, filecoin-ffi must be set up as a dependency in a special way, involving replace directives: https://github.com/filecoin-project/filecoin-ffi#go-get

The user must also choose whether to manually download a prebuilt zip from GitHub, or to follow the build process in the README. This entire process requires manual steps from the user, and follows an anti-pattern in Go, as every transitive downstream must carry the same replace directive.

Ideally we would publish a "prebuilt" filecoin-ffi module, so Go could download the prebuilt versions normally. Unfortunately, with the prebuilt object weighing 100+ MiB makes that unfeasible; the git repository would keep growing at a fast pace, meaning that a new `git clone` could download gigabytes after just a few dozen versions.

### Solution

Turns out you don't need to use a VCS repository to publish a Go module. After all, GOPROXY is zip files over HTTPS for each version, so why can't we just close that loop and make `go build` transparently download the equivalent prebuilt zips from github releases?

Turns out there's a feature precisely like that: https://golang.org/ref/mod#serving-from-proxy

Below are the proposed steps to implement the use of said feature.

#### Step 1

We need to be able to serve `<meta>` tags for `?go-get=1` HTTP GET requests, so `go build` knows where to fetch the prebuilt zips from. This means a custom domain, because `github.com/user/repo` doesn't give you that ability, as far as I can tell.

So we could declare sibling prebuilt modules somewhere like `go.fil.org/ffi-prebuilt-linux-amd64`. We wouldn't even need a server - we could just point it to GitHub Pages. That's what I do with my modules; see e.g. https://github.com/mvdan/mvdan.cc/blob/master/gofumpt/index.html.

Such sibling modules would be split by GOOS/GOARCH, because otherwise we'd need to have one single zip containing prebuilt objects for all platforms.

#### Step 2

We need that meta tag to point towards a GOPROXY able to serve the zip files at the right URL paths, like `/@v/v1.2.3.zip`. GitHub serves the release zips on different paths, so we'd have to map them.

The simplest way seems to be to have a lightweight HTTPS server somewhere, such as `proxy-prebuilt.fil.org`. It would take `/ffi-prebuilt/@v/v1.2.3.zip` and redirect that straight to `github.com/filecoin-project/filecoin-ffi/releases/download/v1.2.3.zip`. It would similarly need to handle `.info` and `.mod` requests, but all of that should be fairly straightforward.

My hope is that an off-the-shelf HTTP server could be configured to do the above, but if not, we could also write a small Go HTTP server to do it. We could even make it a standalone project, as this GOPROXY glue could be reused outside of filecoin-ffi.

#### Step 3

Change the current github.com/filecoin-project/filecoin-ffi module to forward to the right prebuilt module. For example:

$ cat prebuilt_linux_amd64.go
// Miners can build with -tags=ffi_source to skip the prebuilt modules.
// +build !ffi_source

package ffi

import prebuilt "go.fil.org/ffi-prebuilt-linux-amd64"

func SomeAPI() SomeResult { return prebuilt.SomeAPI() }

There's many ways that we could design the package structure and code layout to reduce duplication. But if we end up needing to duplicate all APIs to forward them to the prebuilt module, we could always use a bit of code generation. What's important is that running `go build` on `linux/amd64` would only download the prebuilt object file for that platform.

There would also be `// +build ffi_source` files with the current logic to require the built object file to be present. Those would be the "non-forwarding" files that would get released into prebuilt zips.