Building And Releasing CouchDB For Fun And Profit

--- tags: nh, couchdb --- # Building And Releasing CouchDB For Fun And Profit ## Introduction This is a general-purpose guide designed to onboard people into the CouchDB release cycle. Parts of it can also be adapted to troubleshoot building CouchDB on your local device (something that, as an end user, you shouldn't need to do unless we don't support your architecture/OS). _This guide is currently a work in progress._ ## Prerequisites To build CouchDB, you will need: - A bit of free space on the disk you're building CouchDB on. - Erlang/OTP. This can be any version above or equal to OTP 19, but for the sake of convenience with using `couchdb-pkg`, it should be OTP 20 (I use 20.3.8.26, the latest in the 20 release line). I'd recommend [asdf](https://asdf-vm.com/) for managing multiple Erlang installations. - ICU, this is usually available as libicu on Linux, on recent macOS versions, it's pre-installed. - GNU Make - GCC or clang, make sure that your clang installation aliases to `gcc` and `g++` if you're using it. - Python, 3.5 or up. - Node.js, ideally the latest LTS version (16 at the time of writing). - Sphinx (installable via `pip install sphinx`), but more recent Sphinx versions are broken. I've had success with `pip install "sphinx==3.1.2"`. - The Sphinx RTD theme (`pip install sphinx_rtd_theme`). - Spidermonkey. This is Mozilla's JS compiler chain, it's also known as `libmozjs` or just `js`. CouchDB supports several versions of this: 1.8.5, 60, 68, and 86. One of these versions should ideally be available in your package manager, if you're on a M1 Mac, you can use one of [these Homebrew formulas](https://github.com/janl/homebrew-couchdb/tree/master/Formula). For _packaging_ CouchDB, you also need: - A piece of hardware running Ubuntu 20 LTS. Our packaging repo is specifically tuned to build DEBs and RPMs from a Ubuntu machine. It also needs Docker installed, so it can't be Ubuntu running in Docker. I'd recommend a dual-boot with an already existing system. - (For ARM builds) Something that runs ARM, like a Raspberry Pi 4. - Docker - curl, gpg and the rpm tools (on Ubuntu, `apt install curl gpg rpm`) - A bit more disk space, because we're going to spit out a bunch of intermediary files and binaries. - Windows requirements tbd ## Building CouchDB on Linux or macOS CouchDB follows the regular Unix procedue of building programs from source: 1. Get a copy of the source code 2. Run the configure script 3. Execute a make task ### 1. Get a copy of the source code You can get a fresh copy of the CouchDB source code via a Git checkout: ```bash git checkout https://github.com/apache/couchdb.git ``` If you're building a specific version, you can checkout to the specific tag: ```bash git checkout tags/3.2.1 ``` If you're releasing, check out to the major version branch (3.x at the time of writing): ```bash git checkout 3.x ``` ### 2. Run the configure script As is convention for Unix programs, CouchDB contains a Bash script that generates a Makefile tailored for your system. This script is called `configure`, and you can run it like this once you're in your CouchDB source code directory: ```bash ./configure ``` This'll check out CouchDB's dependencies, and initialize the tooling used to build CouchDB. Notably, it doesn't strictly check for build-time dependencies such as Spidermonkey - this will error during the build step instead. By default, the configure script will assume you have Spidermonkey 1.8.5 installed. You can specify a different version by setting a command line flag like this: ```bash ./configure --spidermonkey-version 68 ``` ### 3. Execute a make task CouchDB has a variety of make tasks available. The most important ones are these: - `make all` - builds all components. This is the default when you run make without any arguments - `make dist` - creates a __release tarball__. This is what we're going to need as a basis to build binaries from. - `make distclean` - gets rid of build and distribution artifacts so you can do a clean build If you're just looking to build CouchDB for your own usage, run `make`. If you want to package CouchDB after, run `make dist`. ## Building CouchDB on Windows TBD :) ## Releasing CouchDB ### Overview CouchDB releases are distributed in the following ways: - As a source tarball - Windows x64 executables (complete with installer) - A Mac app in a .zip file - Software packages for the following distributions: - Ubuntu LTS (16, 18, 20) - Debian 9 and 10 - CentOS/RHEL 7 and 8 The CouchDB project does not distribute any packages to Linux/BSD distributions' package archives. This is a task performed by volunteers not in association with the project. Additionally, these packages end up being several years out of date (you can see an overview [here](https://repology.org/project/couchdb/versions)). Mac releases are currently handled by Jan Lehnardt. ### Releasing source tarballs Before we do anything, we need a release tag. This can be an RC version (e.g. 3.3.0-RC1), or a plain release (e.g. 3.3.0). Once that's been created, we checkout to the tag and run `make dist`. This will create a source tarball that we can distribute. _TODO: how to generate SHA hashes for the tarball_ _TODO: how to sign this tarball_ ### Releasing DEBs, RPMs and binaries This is enabled via [`couchdb-pkg`](https://github.com/apache/couchdb-pkg), a repository that provides support files and scripts in order to automatically build DEBs and RPMs for different systems, using Docker. This will need to be run on a machine that supports building DEB files, as building RPM files on a non-RPM machine is easier. Ideally, this would be Ubuntu 20.04. Make sure you have the dependencies listed under "Prerequisites" above. You will want to clone `couchdb-pkg`: ```bash git clone https://github.com/apache/couchdb-pkg.git ``` From here on, your main interaction will be with the `build.sh` script. To display a short info text about what `build.sh` can do: ```bash ./build.sh ``` You will need the tarball you built as the first step of the release somewhere where it's accessible. To build for every platform and arch that we support, run: ```bash ./build.sh couch-all path/to/your/tarball.tar.gz ``` This will take a while. Basically, for every arch/platform, it: - Downloads a docker image that's normally used in the CouchDB CI process - Mounts a volume to the destination package path - Copies over the tarball we provided - The docker container then builds a release based off this tarball In the end, we'll have our generated packages in `pkgs/couch`. The RPM packages for CentOS/RHEL need to have correct file permissions, so we run: ```bash sudo chmod a+w pkgs/couch/centos-*/*.rpm ``` _TODO: how to sign rpms_ _TODO: how to upload packages to artifactory_ ### Releasing Docker images TODO ### Releasing Windows executables #### Setting up Windows To build for Windows, you're going to need some kind of Windows installed somewhere. For ideal use of existing hardware, a laptop or something running Windows is ideal, but running it in a VM (as long as you give it enough system resources) is fine, too. If you're setting up a fresh copy of Windows, make sure it's 64-bit. Any version of Windows above 7 is 64-bit-only, but we recommend Windows 10 (preferrably a N version, as those come without media bloat). You want to make sure to install all available Windows updates. This can take quite a while, be sure to keep checking the Windows update screen, as it can download multiple updates in sequence instead of at the same time. Once all of your updates are done, create a snapshot of your VM (if you're running Windows in a VM). #### Setting up `couchdb-glazier` Windows builds are made easier via [couchdb-glazier](https://github.com/apache/couchdb-glazier), a collection of scripts that aims to automate the setup and build process, much like `couchdb-pkg` does for DEBs and RPMs. Open an Administrator PowerShell, and run the following: ```bash mkdir C:\Relax\ cd C:\Relax\ Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1')) choco feature enable -n allowGlobalConfirmation choco install git ``` At this point, you may need to restart PowerShell for it to recognize `git` as a command. Once that's done: ```bash Set-ExecutionPolicy Bypass -Scope Process -Force git clone https://github.com/apache/couchdb-glazier &.\couchdb-glazier\bin\install_dependencies.ps1 ``` This will take a loooong time. Make sure your disk that Windows is on has around 50GB of free space or more. Your PowerShell may not immediately recognize some of the commands available to you as a result of running that script. If, during the script execution, it complains about `pip` not being available, simply execute it after you've restarted PowerShell: ```bash pip install --upgrade sphinx docutils pygments nose hypothesis sphinx_rtd_theme ``` As a shortcut from now on, in case you reopen PowerShell, you can set the important environment variables using the `shell.ps1` script: ```bash Set-ExecutionPolicy Bypass -Scope Process -Force &.\couchdb-glazier\bin\shell.ps1 ``` #### Building dependencies and CouchDB on Windows In addition to what the installer has installed for you, you'll want to have `pkg-config` installed via choco: ```bash choco install pkg-config ``` Next up, you'll want to build Erlang. This should Just Work(tm) for you if you follow the relevant [section](https://github.com/apache/couchdb-glazier#building-erlang) in the glazier readme. For building SpiderMonkey, save yourself the trouble of cloning the entire `gecko-dev` repository. Instead, just download the relevant sources in a tarball: ```bash C:\mozilla-build\start-shell.bat cd /c/Relax wget.exe https://ftp.mozilla.org/pub/spidermonkey/prereleases/60/pre3/mozjs-60.1.1pre3.tar.bz2 tar xvf mozjs-60.1.1pre3.tar.bz2 cd mozjs-60.1.1pre3/js/src ``` The only difference in the build process is that you will need to run the configure script without the `--with-system-icu` option, as we want to use the bundled ICU. You will also need to set the PKG_CONFIG_PATH and PATH to make the Mozilla build tools aware of your library locations: ```bash # set the PATH to include pkg-config, as well as the pkg-config path itself. do this once you're in the mozilla build tools shell export PATH="/c/ProgramData/chocolatey/bin:$PATH" export PKG_CONFIG_PATH="/c/Relax/vcpkg/installed/x64-windows/lib/pkgconfig" # instead of the configure script used in the readme, use this ../configure --disable-ctypes --disable-ion --disable-jemalloc --enable-optimize --enable-hardening --with-intl-api --build-backends=RecursiveMake --with-visual-studio-version=2017 --disable-debug --enable-gczeal --target=x86_64-pc-mingw32 --host=x86_64-pc-mingw32 --prefix=/c/relax/vcpkg/installed/x64-windows ``` CouchDB's build tools may complain about a missing `mozjs-60.lib` file later, so after you've finished building SpiderMonkey, you want to make sure CouchDB's LIBPATH includes said `.lib` file. You can do this either by setting the LIBPATH yourself (TODO how?) or copying the file to somewhere in the existing LIBPATH, like this: ```bash copy C:\Relax\mozjs-60.1.1pre3\js\src\build_OPT.OBJ\js\src\build\mozjs-60.lib 'C:\Program Files\erl19.3.3.14\lib\erl_interface-3.10.2.2\lib' ``` The rest of the CouchDB build process should now run exactly as described in the relevant [section](https://github.com/apache/couchdb-glazier#building-couchdb-itself). Hooray!

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.