FreeBSD Testing & CI Tasks and Ideas
====================================

## CI system itself

* Be able to download the raw junit xml of the test rult, this helps people compare the rsult of two builds.

* publish status to github repository.
* https://github.com/CTSRD-CHERI/jenkins-scripts/blob/master/vars/setGitHubStatus.groovy
* do this after swtiching to git, for easiter matching revision hash
* https://www.jenkins.io/doc/pipeline/steps/pipeline-githubnotify-step/

## Base system

* **Fixing skippped and failing test cases**
* https://ci.freebsd.org/job/FreeBSD-head-amd64-test/lastSuccessfulBuild/testReport/
* https://ci.freebsd.org/job/FreeBSD-head-i386-test/lastSuccessfulBuild/testReport/

* move stress tests to another dedicated job, and see if it's possible to run pho's stress test suite.

* MFC autobuild/tester (kp@)
:::info
Right now developers get a reminder e-mail based on the ‘MFC After’ tag in commit messages. It should be possible to automatically merge, build and test (i.e. run the existing automatic test suite) these.

Ideally that would enable a button in the e-mail (or elsewhere) to then also commit the change, but even a quick sanity check (does it merge and build?) would be valuable extra information in the MFC e-mail.
:::
https://github.com/sobomax/mfc_notifications

* https://issues.tmatesoft.com/issue/SVNKIT-740

* Improve the interafce of tinderbox.freebsd.org
* source: https://github.com/FreeBSDFoundation/jenkins-tinderbox
* Add test result imformation
* better view on mobile device

* Document: how developers should know and work with ci.freebsd.org
* https://wiki.freebsd.org/Jenkins/Debug

* test case history tracker - for finding out flakey tests easier. i.e., we want to have a number "Test case X has failed Y times in last Z builds."
* ***there is "history" in the test result viewer***
* approach 1: a key-value database, to track (test case id: a.b.c, build#, result)
* approach 2: a crawler script to collect states from last X builds.

* A "filter" job/script to only execute test when related code chanced.
* e.g. ignore test if the commit contains only document changes

* A script to help locating the root cause of failure of a build log with `make -jN`
* `grep -AN -BN 'error:'` sometimes helps
* if `error:` is not found, search for: `warnings being treated as errors`
* also: https://ci.freebsd.org/job/FreeBSD-head-amd64-build/13944/console
`00:31:48.358 make[5]: make[5]: don't know how to make /usr/obj/usr/src/amd64.amd64/obj-lib32/tmp/sys/netinet/in.h. Stop`

* Improve test script, add "check points" to exit run-away (e.g., boot failure) jobs early

* Show freebsd-ci helper scripts in the build information page.
This is for analyzing failures.

* A meta port for pulling in all test dependencies
* see "sudo chroot ufs pkg install -y" section of https://github.com/freebsd/freebsd-ci/blob/master/scripts/build/build-test_image-head.sh

* make test execution parallel
* having annotation for marking test cases require specified resource

* Check plugin for jenkins "Test Results Analyzer plugin"

* If kernel panics during the test run, extract core dumps and upload to artifact server (and other analysis service) for further examination

* Check results, fix them or annotate is false positive
* https://ci.freebsd.org/job/FreeBSD-head-scan_build/lastCompletedBuild/clangScanBuildBugs/
* https://ci.freebsd.org/job/FreeBSD-head-amd64-gcc/lastCompletedBuild/warnings28Result/
* https://ci.freebsd.org/job/FreeBSD-head-amd64-build/lastCompletedBuild/warnings10Result/

* bhyve tests
* Currently ci.freebsd.org runs tests in bhyve VMs, but bhyve itself is not tested.
* Bhyve dones't support nested VM, and this might not cover all the cases.
* This is a good case to run when we have infrastructure to install and run tests on headware.

* Script to check which test case increased execution time
* WIP: https://gist.github.com/8979dbfe03d0309ea8e5d3136264e127
* This is useful to debug the case that whole test job timed out.
* another though -> is it possible to ask kyua dump the current execution summary before preempting and abort the job?

* Read-only src & obj scenario:
* https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241402
* https://github.com/freebsd/freebsd-ci/pull/20

* 17:27 <@manu__> lwhsu: emaste_: we should have an hw ci job that takes upstream dts to notice breakage sooner
:::info
17:51 <@manu__> https://gist.github.com/evadot/962fcd8f3df0627aefb45b4243699ffd
17:51 <@manu__> sh build_dtb.sh am335x-boneblack.dts arm `pwd`
17:52 <@manu__> sh build_dtb.sh allwinner/sun50i-a64-pine64-lts.dts arm64 `pwd`
17:52 <@manu__> that will build the dts and put the result in the current dir
17:52 <@manu__> then you just need to copy that to the tftp server
17:52 <@manu__> more logs would be better to have like the git id etc ...
:::

* A "try broekn" job to check if the skipped/disbale tests are fixed silently.

* Fuzz testing for the userspace programs and libraries.
* libarchive comes with test_fuzz but mostly implemented with just calling random(). It will be nice to implement a fuzzer with libFuzzer for the components in the base, then having a perodic job running latest snapshot for certain time.
* Ref: https://reviews.freebsd.org/D27153

* kernel modules kldload/kldunload
* need a minimal kernel to test

## Known issues

*

## Ports / 3rd party software

* Create docker image like https://github.com/madworx/docker-netbsd , which is needed for xorg tests.

## Misc

* https://github.com/freebsd/freebsd-ci/commit/ff0efd80ddb5173980308e86f72f0cd4bd6fff19
* Need to adjust (remove) version string in triple
* Need to discuss on -toolchain@