--- tags: freebsd, pkg, poudriere, blog, draft --- # pkg and poudriere - a lightning trip - pkg: tools to create, install, upgrade, or delete tarballs using an included manifest and url-accessible repositories - poudriere: automated tool to create sets of packages from Makefiles using FreeBSD ports tree FreeBSD 10+ provides a new [pkg(8)] tool and a new efficient binary format for storing package info, with similar functionality to `apt` or `yum` in Linux distros. Despite this, building custom packages on FreeBSD is a very simple task and we should use it as much as possible, both to set build -specific options, and to make our own private ports, instead of just relying on upstream packages and changes. poudriere is the tool of choice to do this. It's job is to build each package and all its dependencies within a FreeBSD jail, and to provide annotated build logs and final packages that are directly usable as an external package repository. Used with zfs it leaves no trace on your main system other than a handful of config files in `/usr/local/etc/poudriere*`. ### whats in a `pkg(8)`? - a tarball - a manifest - take a look at `security/vault` ```shell $ mkdir /tmp/vault; tar -C /tmp/vault -xvf /var/cache/pkg/vault-0.10.0-0232b7c2d6.txz x +COMPACT_MANIFEST x +MANIFEST tar: Removing leading '/' from member names x usr/local/bin/vault x usr/local/share/licenses/vault-0.10.0/catalog.mk x usr/local/share/licenses/vault-0.10.0/LICENSE x usr/local/share/licenses/vault-0.10.0/MPL20 x usr/local/etc/rc.d/vault $ jq -C . /tmp/vault/+MANIFEST | less -R ``` ```json { "name": "vault", "origin": "security/vault", "version": "0.10.0", "comment": "Tool for securely accessing secrets", "maintainer": "swills@FreeBSD.org", "www": "https://vaultproject.io/", "abi": "FreeBSD:12:amd64", "arch": "freebsd:12:x86:64", "prefix": "/usr/local", "flatsize": 51555423, "licenselogic": "single", "licenses": [ "MPL20" ], "desc": "Vault is a tool for securely accessing secrets. A secret is anything that you\nwant to tightly control access to, such as API keys, passwords, certificates,\nand more. Vault provides a unified interface to any secret, while providing\ntight access control and recording a detailed audit log.\n\nWWW: https://vaultproject.io/", "categories": [ "security" ], "users": [ "vault" ], "groups": [ "vault" ], "annotations": { "FreeBSD_version": "1200062" }, "files": { "/usr/local/bin/vault": "1$b6fd251e3ac2f24681269d2310594a0bb3909dfaaf50f68894fed2b8e34c8921", "/usr/local/share/licenses/vault-0.10.0/catalog.mk": "1$6c2d3ba563526f9c6a146e17878b9b486709f26bd88d72c9f3dcffcc6ff881d5", "/usr/local/share/licenses/vault-0.10.0/LICENSE": "1$47cd751e6e6d155f6b79f15393444030b1e4dce054da282166eb5e140e4d9a1a", "/usr/local/share/licenses/vault-0.10.0/MPL20": "1$60222c28c1a7f6a92c7df98e5c5f4459e624e6e285e0b9b94467af5f6ab3343d", "/usr/local/etc/rc.d/vault": "1$7c680d40536e2e6e5510f53882b72f4a644991b8f5e87c4147feff26cca77c95" }, "scripts": { "pre-install": "if [ -n \"${PKG_ROOTDIR}\" ] && [ \"${PKG_ROOTDIR}\" != \"/\" ]; then\n PW=\"/usr/sbin/pw -R ${PKG_ROOTDIR}\"\nelse\n PW=/usr/sbin/pw\nfi\necho \"===> Creating groups.\"\nif ! ${PW} groupshow vault >/dev/null 2>&1; then\n echo \"Creating group 'vault' with gid '471'.\"\n ${PW} groupadd vault -g 471\nelse\n echo \"Using existing group 'vault'.\"\nfi\necho \"===> Creating users\"\nif ! ${PW} usershow vault >/dev/null 2>&1; then\n echo \"Creating user 'vault' with uid '471'.\"\n ${PW} useradd vault -u 471 -g 471 -c \"Vault Daemon\" -d /nonexistent -s /usr/sbin/nologin\nelse\n echo \"Using existing user 'vault'.\"\nfi", "post-deinstall": "if [ -n \"${PKG_ROOTDIR}\" ] && [ \"${PKG_ROOTDIR}\" != \"/\" ]; then\n PW=\"/usr/sbin/pw -R ${PKG_ROOTDIR}\"\nelse\n PW=/usr/sbin/pw\nfi\nif ${PW} usershow vault >/dev/null 2>&1; then\n echo \"==> You should manually remove the \\\"vault\\\" user. \"\nfi\nif ${PW} groupshow vault >/dev/null 2>&1; then\n echo \"==> You should manually remove the \\\"vault\\\" group \"\nfi" } } ``` ## mandatory fields - definitive list is in `libpkg/pkg.c:pkg_is_valid` ```ucl comment: desc: maintainer: name: origin: version: prefix: www: ``` ## minimal example - a manifest file with 8 mandatory fields - a possibly empty directory - single command-line invocation ```sh #!/bin/sh pkg create --verbose \ --root-dir /var/empty \ --manifest ./manifest.ucl \ --out-dir /tmp/ ``` ### example minimal manifest ```yaml name: hal origin: example/hal comment: "Open the pod bay doors, HAL." prefix: /usr/local version: 0.0.1 www: https://2001/hal maintainer: dch@skunkwerks.at desc: <<EOD I'm sorry, Dave. I'm afraid I can't do that. EOD ``` ## building manifests ### additional fields injected by `pkg-create` - abi - arch - flatsize = uncompressed (installed) footprint in bytes ```yaml abi: FreeBSD:12:amd64 arch: freebsd:12:x86:64 prefix: /usr/local flatsize: 0 ``` ### contents - files, dirs and links Files, directories and softlinks are not included unless they are specifically referenced in a dict: ```yaml files: /usr/local/bin/foo: sha256sum, /usr/local/bin/bar: { sum: sha256sum, uname: baruser, gname: foogroup, perm: 0644 } } directories: { /usr/local/share/foo-1.0: y, /path/to/directory: { uname: foouser, gname: foogroup, perm: 0755 } } ``` The sha256sum and permissions are calculated by `pkg-create` and do not need to be included in the manifest itself. You can use `""` or `"1$0"` as dummy fields instead of sha256. - more tips in https://gist.github.com/dch/7c0711898ae64d0b54c1e6db24ee8a8c ## a custom build script Here's a real example, mu's packaging script for a custom application ```shell #!/bin/sh -e # elixir puts its _build dir in the root of this repo,alongside build.sh NAME=mu cd $(dirname $0) ROOT=$(pwd -P) BUILD=${ROOT}/_build cd - # borrow rebar from packages test -x ${HOME}/.mix/rebar || mix local.rebar --force rebar /usr/local/bin/rebar test -x ${HOME}/.mix/rebar3 || mix local.rebar --force rebar3 /usr/local/bin/rebar3 # ensure we have latest git tags git fetch --tags || /usr/bin/true # ensure we pick up base system commands by preference export PATH=/usr/local/bin:$PATH # UTF8 while we build export LANG=en_US.UTF-8 export LC_ALL=${LANG} export MIX_ENV=prod # clean up any existing release tarballs rm -rf $(find ${ROOT}/rel ${BUILD} -name *z) mix deps.get mix release --env=prod RELEASE=$(find ${ROOT} -name ${NAME}*tar.gz) test -e "${RELEASE}" || exit 1 # use a random temporary dir for everything else BASE=`mktemp -d /tmp/${NAME}.XXXXXX` || exit 1 STAGING=${BASE}/staging DEST=${STAGING}/usr/local/lib/${NAME} PKG=${BASE}/package MANIFEST=${PKG}/manifest.ucl mkdir -p -m0750 ${PKG} ${DEST} ${STAGING}/usr/local/etc/rc.d # exclude non-root users during tar etc umask 027 # get template manifest and other static files cp -a ${ROOT}/templates/pkg/ ${STAGING}/ cp -a ${ROOT}/templates/manifest.ucl ${MANIFEST} # unpack the mix release itself excluding files with tokens tar xzf ${RELEASE} \ -C ${DEST} \ --exclude sys.config \ --exclude vm.args # prepare FreeBSD package manifest ## grab the version number from mix DATE=$(date -u +%Y%m%d-%H%M) # get a git tag or nearest sha to use as a suffix to app version GITREF=$(git describe --dirty --abbrev=7 --tags --always --first-parent 2>/dev/null || true) ## inject all that into the manifest uclcmd set --file ${MANIFEST} --type string version ${DATE}-${GITREF} # include each file, its hash, and any permissions: # expected result: # /usr/lib/mu/bin/halbot: {sum: 1$abc123, uname: root, gname: www, perm: 0440 } SHA_LIST=$(find ${STAGING} -type f -exec sha256 -r {} + \ | awk '{print " " $2 ": {uname: root, gname: www, sum: 1$" $1"}"}') # include softlinks: # expected result looks like: # /usr/local/lib/symlink: - LINKS_LIST=$(find ${STAGING} -type l \ | awk '{print " " $1 ": -"}') # include mu-specific directories and permissions: # make sure we exclude things like /usr /usr/local/ /etc/ that are # already in place as we don't want to change their permissions # expected result looks like: # /usr/local/lib/mu: {uname: root, gname: www, perm: 0550}` DIR_LIST=$(find ${STAGING} -type d -mindepth 3 -path \*/${NAME}/\* \ | awk '{print " " $1 ": {uname: root, gname: www, perm: 0750}"}') # make runtime directories mkdir -m0770 -p \ ${STAGING}/var/db/${NAME} \ ${STAGING}/var/log/${NAME} \ ${STAGING}/var/run/${NAME} # strip off _build/state prefix and append this UCL snippet to manifest cat <<UCL | sed -E -e s:${STAGING}:: >> ${MANIFEST} files: { $SHA_LIST $LINKS_LIST } directories: { $DIR_LIST /var/log/${NAME}: {uname: root, gname: www, perm: 0770} /var/run/${NAME}: {uname: root, gname: www, perm: 0770} /var/db/${NAME}: {uname: root, gname: www, perm: 0770} } UCL ## bubblewrap the package and manifest if test "$(uname -s)" == "FreeBSD"; then pkg create --verbose \ --root-dir ${STAGING} \ --manifest ${MANIFEST} \ --out-dir ${BUILD} cp ${MANIFEST} ${BUILD}/ cat << EOF final manifest: $(find ${BUILD} -name manifest.ucl) package complete: $(find ${BUILD} -name *.txz) to install: sudo -s service mu stop pkg install $(find ${BUILD} -name ${NAME}\*.txz) service mu start EOF fi ``` ## package repositories Now that you can create your own packages, how can we make an installable repository from a bunch of these tarballs? Well, it's easy. Ridiculously easy. You'll need 3 things: - a signing key, see the poudriere section below for more details on creating your own one of these - a directory containing a bunch of your tarballs - an output directory, which could be your webserver, or any file or ssh accessible URL that you want to install via ``` # pkg repo -o /var/www/package-site \ /var/www/packages \ /usr/local/etc/ssl/keys/pkg.secret.key ``` That's it. You'll likely spend more time getting your webserver correctly set up to handle it. I suggest starting with a file-based setup - here's mine: ```yaml # /usr/local/etc/pkg/repos/private.conf private: { url: file:///var/www/private-packages mirror_type: http signature_type: pubkey pubkey: /usr/local/etc/ssl/certs/pkg.secret.cert enabled: yes } ``` Once this is in place, you can `sudo pkg install -r private my/foo` until the cows come home. See [pkg repo](https://www.freebsd.org/cgi/man.cgi?query=pkg-repo) docs for more details. # [poudriere(8)] Building all the things, in an isolated jail every time, with a consistent makefile, and creating a custom package repository for installation. ## poudriere quick start - you need at least 20GiB of disk space by the time you build all the packages - zfs, at least for this guide, you can do UFS but then you'll need to read up yourself - FreeBSD of the version you wish to install, or higher - change `s/example.org/your.domain.net/g` below ### setting up poudriere ```shell # pkg install -y devel/dialog4ports devel/poudriere devel/ccache ``` ### basic configuration ``` # the master config file /usr/local/etc/poudriere.conf ZPOOL=zroot FREEBSD_HOST=http://ftp.de.freebsd.org/ RESOLV_CONF=/etc/resolv.conf BASEFS=/usr/local/poudriere USE_PORTLINT=yes USE_TMPFS=yes DISTFILES_CACHE=/usr/ports/distfiles SVN_HOST=svn.FreeBSD.org CHECK_CHANGED_OPTIONS=verbose PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/pkg.example.org.key WRKDIR_ARCHIVE_FORMAT=txz URL_BASE=https://pkg.example.org/poudriere KEEP_OLD_PACKAGES=yes BUILDER_HOSTNAME=pkg.example.org ``` ### set make.conf Tweaking `/usr/local/etc/poudriere.d/make.conf` ensures we can build dtrace-enabled ports and avoid openssl as a dependency: ```shell # /usr/local/etc/poudriere.d/make.conf BATCH=yes DEVELOPER=yes PROXYDEPS_FATAL=yes KEEP_OLD_PACKAGES=yes USE_TMPFS=all TMPFS_LIMIT=2 WRKDIRPREFIX=/tmp CCACHE_DIR=/var/cache/ccache # Erlang/OTP with DTrace STRIP= WITH_CTF=1 CFLAGS+=-fno-omit-frame-pointer # OpenSSL avoidance if you want to try this DEFAULT_VERSIONS+=ssl=libressl OPTIONS_UNSET= GSSAPI_BASE OPTIONS_SET= GSSAPI_MIT # enable our custom ports category VALID_CATEGORIES+=skunkwerks LICENSES_ACCEPTED+=MIT BSD3CLAUSE APACHE20 LGPLV3 BSD2CLAUSE MPL MPL20 NONE ``` Enable `ln -s /usr/local/etc/poudriere.d/make.conf /etc/make.conf` if you are lazy and want to keep the same options for manual port testing. ### set up signing keys for our packages These are the keys referred to in the earlier private repo example - poudriere uses the same setup to sign its own package repo. ``` mkdir -p /usr/local/etc/ssl/keys /usr/local/etc/ssl/certs chmod 0600 /usr/local/etc/ssl/keys openssl genrsa -out /usr/local/etc/ssl/keys/pkg.example.org.key 4096 chmod 0400 /usr/local/etc/ssl/keys/pkg.example.org.key openssl rsa -in /usr/local/etc/ssl/keys/pkg.example.org.key \ -pubout -out /usr/local/etc/ssl/certs/pkg.example.org.cert ``` ### set up ccache for faster builds ``` zfs create -o canmount=off zroot/var/cache zfs create zroot/var/cache/ccache echo 'cache_dir = /var/cache/ccache' > /var/cache/ccache/ccache.conf echo 'max_size = 20G' >> /var/cache/ccache/ccache.conf ``` ### set up our ports tree We can have 2 ports trees - a public one, in `/usr/ports`, and a private one, in `/usr/ports/skunkwerks`: ``` poudriere ports -c -F -f none -M /usr/ports -m git -n default mkdir -p /usr/local/etc/poudriere.d/ports/default/ echo git> /usr/local/etc/poudriere.d/ports/default/method echo /usr/ports> /usr/local/etc/poudriere.d/ports/default/mnt git clone git://github.com/skunkwerks/ports /usr/ports cd /usr/ports git remote add upstream git://github.com/freebsd/freebsd-ports git fetch --all git clone git@github.com:skunkwerks/wharf /usr/ports/skunkwerks # source tarballs from upstream repos are downloaded & cached here zfs create zroot/usr/ports/distfiles ``` ### create a poudriere jail We'll create 3 jails - a standard amd64 one, an i386 one, both from FreeBSD releases downloaded from upstream, and also one built of FreeBSD current using the tarball we built ourselves. Note that we can *only* use a 12.0-CURRENT jail if the host OS is of the same build, or newer. ``` # poudriere jail -c -j 11_2_x64 -v 11.2-RELEASE -a amd64 # poudriere jail -c -j 11_2_x86 -v 11.2-RELEASE -a i386 #poudriere jail -c -j current_x64 -v 12.0-CURRENT -a amd64 \ -m tar=/usr/obj/usr/src/amd64.amd64/release/base.txz ``` ### set up and run a custom port build ``` # make a list of builds we are interested in having a custom port for echo lang/erlang-runtime19 > /usr/local/etc/poudriere.d/packages.txt # or use this pkg trick to list all the "leaf" packages you installed: pkg prime-origins > /usr/local/etc/poudriere.d/leaves.txt ``` ## using custom port builds We have some port, say, [lang/erlang] , where we wish to enable dtrace, hipe, and dirty schedulers. For dtrace support, make sure kernel support is started via `kldload dtraceall` or the build will fail mysteriously. This is typically enabled by adding `dtraceall_load="YES"` to `/boot/loader.conf`. ``` # we could simply customise all packages listed in packages.lst poudriere options -f /usr/local/etc/poudriere.d/packages.lst # or just customise a single package and assume defaults for its descendants poudriere options -cn lang/erlang-runtime19 # look in here for the specific set options /usr/local/etc/poudriere.d/options/lang_erlang-runtime19/options ``` ``` # clear and re-do options for a specific set of ports poudriere options -cn editors/vim lang/erlang net-mgmt/collectd5 lang/erlang-runtime19 devel/protobuf-c ``` ### pulling in a specific port from upstream Sometimes, instead of forking and merging all the updates from the main FreeBSD ports tree, we simply want to cherry-pick a particularly juicy and flavoursome port. The catch is that, there may have been several intermediate updates, and also dependencies that we want, so a simply `git cherry-pick` is not going to cut the mustard. Instead, we remove the entire port, then check it out from the upstream git master branch, and commit that whole lot. Remember to check the dependencies, and if necessary update them too. Once you've done that you can simply ask poudriere to rebuild the port(s) for you: `sudo poudriere bulk -j 11_1_amd64 category/port`, but as your normal user, not as the build user which has no sudo privileges. Note the `-A` to ensure you have ssh-forwarded private keys available for git to use on your build server. Remember *not* to use `-c` or `-C` here as it will wipe *all* our ports. If you do this, recover them from zfs snapsots rather than rebuild all the ports, as the checksums are different in the rebuilt ports, and therefore reinstalls of all packages may be needed. ``` ~ ssh -A build@wolf (34s 913ms) Last login: Mon Mar 13 21:57:39 2017 from 2001:569:77f0:3200:1579:a70e:3f88:af26 FreeBSD 11.0-RELEASE-p8 (GENERIC) #0: Wed Feb 22 06:12:04 UTC 2017 $ cd /usr/ports $ git fetch --all Fetching skunkwerks Fetching upstream remote: Counting objects: 1057, done. remote: Compressing objects: 100% (315/315), done. remote: Total 1057 (delta 478), reused 387 (delta 387), pack-reused 352 Receiving objects: 100% (1057/1057), 1.87 MiB | 397.00 KiB/s, done. Resolving deltas: 100% (519/519), completed with 177 local objects. From git://github.com/freebsd/freebsd-ports a29c5c848d50..9789ea3b3b86 master -> upstream/master d73362eecec2..23e2fff4f6ff svn_head -> upstream/svn_head $ git status On branch 2017Q1 Your branch is up-to-date with 'skunkwerks/2017Q1'. nothing to commit, working tree clean $ more sysutils/graylog/distinfo TIMESTAMP = 1473800220 SHA256 (graylog-2.1.0.tgz) = 76bf9bb4da57bfbfb146f0eaf22458328f5cb76c8e39bf735265886dbb7c2c27 SIZE (graylog-2.1.0.tgz) = 95402379 $ rm -rf sysutils/graylog/ $ git checkout upstream/master -- sysutils/graylog $ tree sysutils/graylog/ sysutils/graylog/ ├── Makefile ├── distinfo ├── files │ ├── graylog.in │ ├── graylog_logging.xml │ ├── pkg-message.in │ └── server.conf.sample.in ├── pkg-descr └── pkg-plist 1 directory, 8 files $ git status On branch 2017Q1 Your branch is up-to-date with 'skunkwerks/2017Q1'. Changes to be committed: (use "git reset HEAD <file>..." to unstage) modified: sysutils/graylog/Makefile modified: sysutils/graylog/distinfo modified: sysutils/graylog/files/graylog.in new file: sysutils/graylog/files/pkg-message.in new file: sysutils/graylog/files/server.conf.sample.in modified: sysutils/graylog/pkg-plist Changes not staged for commit: (use "git add/rm <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory) deleted: sysutils/graylog/files/graylog.conf.example $ git commit -am "sysutils/graylog: import 2.2.2 from upstream" [2017Q1 69a8d5522b1e] sysutils/graylog: import 2.2.2 from upstream 6 files changed, 111 insertions(+), 96 deletions(-) rewrite sysutils/graylog/Makefile (61%) create mode 100644 sysutils/graylog/files/pkg-message.in rename sysutils/graylog/files/{graylog.conf.example => server.conf.sample.in} (99%) rewrite sysutils/graylog/pkg-plist (100%) $ git push skunkwerks master Counting objects: 11, done. Delta compression using up to 12 threads. Compressing objects: 100% (10/10), done. Writing objects: 100% (11/11), 8.66 KiB | 0 bytes/s, done. Total 11 (delta 4), reused 6 (delta 1) remote: Resolving deltas: 100% (4/4), completed with 4 local objects. To github.com:skunkwerks/ports 7dfcf176cfd4..69a8d5522b1e 2017Q1 -> 2017Q1 ``` ### keeping things updated - the port tree itself needs to be merged/rebased into an appropriate upstream ports branch - the jail needs to be updated - the host system also needs to be updated ``` # ports tree cd /usr/ports git fetch --all git rebase upstream/master # .... hackety hackety git push --force origin/master # update jail poudriere jail -u -j 11_1_amd64 # rebuild its packages nicely nice -n 18 idprio 29 poudriere bulk -j 11_1_amd64 -f /usr/local/etc/poudriere.d/server.pkg # once the core server packages are done do your server upgrade and then rebuild all the packages nice -n 18 idprio 29 poudriere bulk -j 11_1_amd64 -f /usr/local/etc/poudriere.d/server.pkg -f /usr/local/etc/poudriere.d/xorg.pkg -f -f /usr/local/etc/poudriere.d/desktop.pkg ``` I have a short helper script to do this which runs as part of cron, for my server that runs `CURRENT`: ``` /root/update.sh #!/bin/sh -e freebsd-update fetch install || /usr/bin/true # poudriere ports -u cd /usr/ports git fetch upstream git rebase upstream/master chown -R root:wheel /usr/ports chmod -R g+rw /usr/ports # update the ports poudriere jail -u -j 11_1_amd64 poudriere bulk -f /usr/local/etc/poudriere.d/core.pkg -j 11_1_amd64 pkg backup -d /var/db/pkg/backup pkg update pkg upgrade -y echo OK done ``` Just keep an eye out for any daemons that may now have updated binaries and need restarting. ## sharing packages ### exporting our packages - rsync and `file:///` urls are perfectly fine, as is zfs send etc - [www/h2o](https://github.com/freebsd/freebsd-ports/blob/master/www/h2o/files/h2o.conf.sample) has a configuration for exporting our custom packages over HTTP(S) ### advanced pkg repo security Some package repositories need security. While a certificate-based approach may be possible, I've successfully used `ssh://`, http basic authentication, and even a humble `rsync` and local `file://` url. ### using file URL and rsync The details of `rsync` or `zfs send|recv` are left to the user. Ensure softlinks are preserved appropriately. Within your `/usr/local/etc/pkg/repos/repo.conf` file, simply use an appropriate path: ``` skunkwerks: { url: file:///usr/local/poudriere/data/packages/11_1_amd64-default mirror_type: http signature_type: pubkey pubkey: /usr/local/etc/ssl/certs/poudriere.cert enabled: yes } ``` ### ssh repo security Use ssh:// urls, including username, url, and path, to retrieve packages securely, in your `/usr/local/etc/pkg/repos/repo.conf` file. Specific ssh keys can be configured in `/root/.ssh/config` as required. Ensure you restrict access on the server side too, via `sshd_config`. url: ssh://pkg.example.org:/usr/local/poudriere/data/packages/11_1_amd64-default ### http basic auth The underlying libfetch library is used by pkg to do the dirty work. See https://www.freebsd.org/cgi/man.cgi?query=fetch&sektion=3 for details. Add this to your `h2o.conf` file for serving packages, making use the jail name you are using in poudriere matches the `file.dir` location -- in this case, it's `11_1_amd64-default`: ```yaml pkg.example.org: paths: "/poudriere": file.dir: "/usr/local/share/poudriere/html/" "/poudriere/data": file.dir: "/usr/local/poudriere/data/logs/bulk/" "/FreeBSD:11:amd64": mruby.handler: | require "htpasswd.rb" Htpasswd.new("/usr/local/etc/h2o/private/htpasswd", "skunkwerks") file.dir: "/usr/local/poudriere/data/packages/11_1_amd64-default/" ``` When running `pkg update` and similar, export HTTP_AUTH into the environment to be consumed by pkg: `HTTP_AUTH="basic:skunkwerks:user:passwd" pkg update` ## integrating custom private ports into the ports tree - we use [git](https://github.com/skunkwerks/ports) to edit and maintain any public-facing port changes in `/usr/ports` - `git clone git://github.com/skunkwerks/wharf /usr/ports/skunkwerks` - add `/usr/ports/Makefile.local` to add our `skunkwerks` category into the ports tree - there is a `VALID_CATEGORIES+=skunkwerks` in `/etc/make.conf` and `/usr/local/etc/poudriere.d/make.conf` for poudriere automated package builds and normal by-hand builds as well ## cross-building ports with qemu and poudriere You'll need build custom poudriere-devel with `qemu` option enabled. ``` # pkg install -y qemu-user-static poudriere-devel # echo qemu_user_static_enable=YES > /etc/rc.conf.d/qemu_user_static # service qemu_user_static start # poudriere jail -c -j 12_0_arm64 -a arm64.aarch64 -v 12.0-RELEASE # poudriere bulk -j 12_0_arm64 -f /usr/local/etc/poudriere.d/server.pkg ``` ### tips - there is a single Makefile for each port to describe the build process - see [simple](https://github.com/skunkwerks/simple) for a minimal port that uses a git tag to build from - [wharf](https://github.com/skunkwerks/wharf) is the equivalent of the https://github.com/freebsd/freebsd-ports repo to provide `Makefiles` for build and packaging scripts for our private ports ```shell # /usr/ports/skunkwerks/Makfile # ensures we include our new subcategory COMMENT = private ports maintained by skunkwerks # add a SUBDIR ... line for every new port we require SUBDIR += simple .include <bsd.port.subdir.mk> ``` ```shell # Created by: Dave Cottlehuber <dch@FreeBSD.org> # $FreeBSD$ PORTNAME= simple PORTVERSION= 0.1 # PORTREVISION= 0 DISTVERSIONPREFIX= v CATEGORIES= skunkwerks MAINTAINER= dch@skunkwerks.at COMMENT= some ports are simpler than others LICENSE= APACHE20 LICENSE_FILE= ${WRKSRC}/LICENSE USE_GITHUB= yes GH_ACCOUNT= skunkwerks NO_BUILD= yes NO_ARCH= yes PORTDOCS= README.md do-install: ${MKDIR} ${STAGEDIR}${DOCSDIR} ${INSTALL_DATA} ${WRKSRC}/README.md ${STAGEDIR}${DOCSDIR}/ .include <bsd.port.mk> ``` ## pkg(8) hacks FreeBSD's [pkg(8)] tool supports a few handy package management commands: ``` pkg lock -y <thing> pkg unlock -y <thing> # get a list of all manually installed packages (i.e. "really useful" vs automatic dependencies) pkg leaf # what package installed this file? pkg which /usr/local/bin/erl # read any major ports-related notes for upgrading packages since a given date pkg updating -d 20160101 ``` You can also retrieve an older version of a package from the local cache in `/var/cache/pkg/<name-version>.txz` and install it directly via `pkg add <path>`, although generally it's better to keep packages consistent within the set built at the same time on the package server pkg.example.org and use them from there. ## resources ### packaging - https://wiki.freebsd.org/PkgPrimer has a handy "Rosetta Stone" table - https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/pkgng-intro.html - hooks into poudriere's lifecycle: [poudriere hooks] / bdrewery@ ### canonical - well documented [handbook](https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/book.html) - https://www.freebsd.org/doc/handbook/ports-poudriere.html - https://github.com/freebsd/freebsd-ports/blob/master/www/h2o is a nice short example of a real world port ### useful but not up to date: - http://blog.ignoranthack.me/?p=129 - http://blog.shatow.net/posts/2015-04-27-Poudriere-FreeBSD-Journal/ - http://www.bsdnow.tv/tutorials/poudriere - https://www.digitalocean.com/community/tutorials/how-to-set-up-a-poudriere-build-system-to-create-packages-for-your-freebsd-servers ## main ports tree diff ```diff diff --git a/.gitignore b/.gitignore index 13ad354..51ffbb7 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,4 @@ *.rej *.orig *.sw[p-z] +/skunkwerks diff --git a/Makefile.local b/Makefile.local new file mode 100644 index 0000000..790f37f --- /dev/null +++ b/Makefile.local @@ -0,0 +1,3 @@ +# $FreeBSD$ +# +SUBDIR += skunkwerks ``` [pkg(8)]: https://www.freebsd.org/cgi/man.cgi?query=pkg [poudriere(8)]: https://www.freebsd.org/cgi/man.cgi?query=poudriere [poudriere hooks]: https://gist.github.com/bdrewery/9438353 [lang/erlang]: https://github.com/freebsd/freebsd-ports/blob/master/lang/erlang/Makefile