Be able to download the raw junit xml of the test rult, this helps people compare the rsult of two builds.
publish status to github repository.
Fixing skippped and failing test cases
move stress tests to another dedicated job, and see if it's possible to run pho's stress test suite.
MFC autobuild/tester (kp@)
Right now developers get a reminder e-mail based on the ‘MFC After’ tag in commit messages. It should be possible to automatically merge, build and test (i.e. run the existing automatic test suite) these.
Ideally that would enable a button in the e-mail (or elsewhere) to then also commit the change, but even a quick sanity check (does it merge and build?) would be valuable extra information in the MFC e-mail.
https://github.com/sobomax/mfc_notifications
Improve the interafce of tinderbox.freebsd.org
Document: how developers should know and work with ci.freebsd.org
test case history tracker - for finding out flakey tests easier. i.e., we want to have a number "Test case X has failed Y times in last Z builds."
A "filter" job/script to only execute test when related code chanced.
A script to help locating the root cause of failure of a build log with make -jN
grep -AN -BN 'error:'
sometimes helpserror:
is not found, search for: warnings being treated as errors
00:31:48.358 make[5]: make[5]: don't know how to make /usr/obj/usr/src/amd64.amd64/obj-lib32/tmp/sys/netinet/in.h. Stop
Improve test script, add "check points" to exit run-away (e.g., boot failure) jobs early
Show freebsd-ci helper scripts in the build information page.
This is for analyzing failures.
A meta port for pulling in all test dependencies
make test execution parallel
Check plugin for jenkins "Test Results Analyzer plugin"
If kernel panics during the test run, extract core dumps and upload to artifact server (and other analysis service) for further examination
Check results, fix them or annotate is false positive
bhyve tests
Script to check which test case increased execution time
Read-only src & obj scenario:
17:27 <@manu__> lwhsu: emaste_: we should have an hw ci job that takes upstream dts to notice breakage sooner
17:51 <@manu__> https://gist.github.com/evadot/962fcd8f3df0627aefb45b4243699ffd
17:51 <@manu__> sh build_dtb.sh am335x-boneblack.dts arm pwd
17:52 <@manu__> sh build_dtb.sh allwinner/sun50i-a64-pine64-lts.dts arm64 pwd
17:52 <@manu__> that will build the dts and put the result in the current dir
17:52 <@manu__> then you just need to copy that to the tftp server
17:52 <@manu__> more logs would be better to have like the git id etc …
A "try broekn" job to check if the skipped/disbale tests are fixed silently.
Fuzz testing for the userspace programs and libraries.
kernel modules kldload/kldunload