FreeBSD CI Tasks and Ideas ========================== ## Base system * **Fixing skippped and failing test cases** * * * move stress tests to another dedicated job, and see if it's possible to run pho's stress test suite. * MFC autobuild/tester (kp@) :::info Right now developers get a reminder e-mail based on the ‘MFC After’ tag in commit messages. It should be possible to automatically merge, build and test (i.e. run the existing automatic test suite) these. Ideally that would enable a button in the e-mail (or elsewhere) to then also commit the change, but even a quick sanity check (does it merge and build?) would be valuable extra information in the MFC e-mail. ::: * * Improve the interafce of * source: * Add test result imformation * better view on mobile device * Document: how developers should know and work with * * test case history tracker - for finding out flakey tests easier. i.e., we want to have a number "Test case X has failed Y times in last Z builds." * ***there is "history" in the test result viewer*** * approach 1: a key-value database, to track (test case id: a.b.c, build#, result) * approach 2: a crawler script to collect states from last X builds. * A "filter" job/script to only execute test when related code chanced. * e.g. ignore test if the commit contains only document changes * A script to help locating the root cause of failure of a build log with `make -jN` * `grep -AN -BN 'error:'` sometimes helps * if `error:` is not found, search for: `warnings being treated as errors` * also: `00:31:48.358 make[5]: make[5]: don't know how to make /usr/obj/usr/src/amd64.amd64/obj-lib32/tmp/sys/netinet/in.h. Stop` * Improve test script, add "check points" to exit run-away (e.g., boot failure) jobs early * Show freebsd-ci helper scripts in the build information page. This is for analyzing failures. * A meta port for pulling in all test dependencies * see "sudo chroot ufs pkg install -y" section of * make test execution parallel * having annotation for marking test cases require specified resource * Check plugin for jenkins "Test Results Analyzer plugin" * If kernel panics during the test run, extract core dumps and upload to artifact server (and other analysis service) for further examination * Check results, fix them or annotate is false positive * * * * bhyve tests * Currently runs tests in bhyve VMs, but bhyve itself is not tested. * Bhyve dones't support nested VM, and this might not cover all the cases. * This is a good case to run when we have infrastructure to install and run tests on headware. * Migrate tests to python3 * * src/tests/sys/netpfil * Script to check which test case increased execution time * WIP: * This is useful to debug the case that whole test job timed out. * another though -> is it possible to ask kyua dump the current execution summary before preempting and abort the job? * Read-only src & obj scenario: * * * 17:27 <@manu__> lwhsu: emaste_: we should have an hw ci job that takes upstream dts to notice breakage sooner :::info 17:51 <@manu__> 17:51 <@manu__> sh am335x-boneblack.dts arm `pwd` 17:52 <@manu__> sh allwinner/sun50i-a64-pine64-lts.dts arm64 `pwd` 17:52 <@manu__> that will build the dts and put the result in the current dir 17:52 <@manu__> then you just need to copy that to the tftp server 17:52 <@manu__> more logs would be better to have like the git id etc ... ::: * A "try broekn" job to check if the skipped/disbale tests are fixed silently. ## Known issues * Due to src/sys/conf/ uses `svnversion` directly, the svn revision in `uname -a` will show the revision of the checkout (latest version in the repository), not the "last changed rev" (from `svn info`). This causes the outputs of `uname -a` are different btween two systems build from two `svn co svn://` on different time (where base/head hasn't been changed but other path in base has been modified.) * possible solution: * use `svn info --show-item last-changed-revision svn://` first and then `svn co svn://${SVN_REVISION}` * Difference between "scm polling checked version" and "real svn checked out version" * scm polled r353793 but the checked out version is r353794 ## Ports / 3rd party software * Create docker image like , which is needed for xorg tests. ## Misc * * Need to adjust (remove) version string in triple * Need to discuss on -toolchain@