# Manual test checklist

To best simulate the environment that our users will use our software
in, the manual tests should be done with the tools that are installed by
our installer.

To build and run the installer, run:

INSTALLER=$(./scripts/release/build-installer.sh) && \
"${INSTALLER}" --non-interactive --install-vscode-extension

Then start the local server (with verbose logging to ease debugging):

ci-daemon -v2 --alsologtostderr

### Nightly Build
* [ ] Check the CI/CD pipeline

### CI/CD scripts
* [x] Test CI/CD scripts
* [x] Build and push cictl, github-action and gitlab images tagged as `dev`:

    ./bazelisk.py build //cmd/cictl
    ./build/docker/cictl/build.sh --cictl-version dev <path to cictl binary> --push
    ./build/docker/github-action/build.sh --cictl-version dev --push
    ./build/docker/gitlab/build.sh --cictl-version dev --push
    ./bazelisk.py run //tools/aws/s3 -- upload --bucket="public.code-intelligence.com" --key="cictl/cictl-dev" --file="<absolute path to cictl binary>"

* [x] Build the ci-server installer:

    ./bazelisk.py build --config=release --//cmd/ci-daemon:server_mode //installer:server-installer

* [x] SSH into the `release-cicd-check` VM and delete the installer from your home directory, if there is one.

    gcloud beta compute ssh --zone "europe-west4-a" "release-cicd-test"  --project "ci-fuzz"

* [x] Push the newly built ci-server installer to the VM and install it:

    gcloud compute scp <path to the ci-server installer> release-cicd-test:~/ --zone "europe-west4-a" --project "ci-fuzz"

* [x] At release-cicd-test.code-intelligence.com, generate a GitHub Action CI/CD script for the `zint` project and use it at github.com/ci-fuzz/zint, branch `release-test`. Replace `${{ secrets.CI_FUZZ_API_TOKEN }}` by `${{ secrets.CI_FUZZ_RELEASE_TEST_API_TOKEN }}` and `@v3` by `@dev`. Push and check that the action will run. In [this PR](https://github.com/ci-fuzz/zint/pull/20) two comments should be created by the Action, one with the finding and another with the coverage.
* [x] At release-cicd-test.code-intelligence.com, generate a GitLab CI/CD script for the `zint-gitlab` project and use it at gitlab.com/WillRoque/zint, branch `master`. Replace `cifuzz/gitlab:v3` with `cifuzz/gitlab:dev`. Update the `.gitlab-ci.yaml` file of the repository and check if the pipeline runs and a finding is encountered.
* [x] At release-cicd-test.code-intelligence.com, generate a Jenkins CI/CD script. Update the configuration of https://jenkins.demo.code-intelligence.com/job/cifuzz/ with the recent script. Replace the `CICTL_URL` value with `https://s3.eu-central-1.amazonaws.com/public.code-intelligence.com/cictl/cictl-dev`, the `CICTL_SHA256SUM` value with the output of `sha256sum <path to the recently built cictl binary>`, and `credentials('CI_FUZZ_API_TOKEN')` with `credentials('CI_FUZZ_API_TOKEN_RELEASE')`. Build and check the output, it should encounter a finding.

### Local
Manually test the CI suite with projects in the [fuzz-testing repository][1].

[1]: https://gitlab.com/code-intelligence/fuzz-testing

Before starting, make sure you set these variables in the `.env` file at
the root of the core repository:

DOCKER_REGISTRY
DOCKER_REGISTRY_USER
DOCKER_REGISTRY_PASSWORD

If you don't have a `.env` file yet, copy `.env.template` to `.env` and
replace the placeholders.

When you start to test a project, write your name in parenthesis behind
the "Test $PROJECT" bullet item.

* [x] Test CppCMS (sim)
* [x] Test creating and initializing the CppCMS project:
* Build Script:

      #!/bin/bash -eu
      mkdir build
      cd build
      cmake -DCMAKE_TESTING_ENABLED=OFF -DDISABLE_SHARED=ON -DDISABLE_GCRYPT=ON -DDISABLE_OPENSSL=ON -DDISABLE_FCGI=ON -DDISABLE_SCGI=ON -DDISABLE_HTTP=ON -DDISABLE_CACHE=ON -DDISABLE_TCPCACHE=ON -DDISABLE_GZIP=ON ..
      make -j$(nproc)

* Build Image: `cifuzz/builder-cppcms`
* [x] (VS CTest the fuzz-this-function functionality by trying to generate a
fuzz target from an API function.
* [x] Start the "Fuzz Json" fuzz test by selecting it and clicking "run" (top) or "start run" (bottom).
* [x] Check that charts are showing and updating once the test
collection is running.
* [x] Code Coverage: Navigate to the "Overview" page and click on the "Coverage" card to verify that coverage exists.
* [x] (VS Code) Insert a bug at line 1220 in the file "src/json.cpp"
(e.g., `*(char*)2=1;`). Re-run the campaign to produce a crash
finding. Debug the found crash.
* [x] (VS Code) Click on "All Findings" then click on the finding in the table. Make sure that clicking on the file link in the view would navigating to the erroneous line.
* [x] (VS Code) Test if debugging the finding works and you can step to the error introduced
* [x] Test grammar fuzzer by clicking on the "fuzz_json_grammar" target and clicking "run" or "start run"
* [ ] Test nginx (testing the socket fuzzer)
* [x] Test creating and initializing a fuzzing project using
* The build script:

         #!/bin/bash -eu
         ./auto/configure
         mkdir -p logs
         make

* The build image: cifuzz/builder-nginx
* [x] Create a new socket fuzz test using the "add fuzz target"
functionality in the UI.
* [x] Select TCP port 6666 and Raw data.
* [x] Select the `nginx` binary from the dropdown menu.
* [x] Use the following run arguments: `-p $SRC` and `-c .code-intelligence/nginx2.conf`, each on a new line.
* [ ] Run the the created fuzz test by selecting it and clicking "run" or "start run".
* [ ] Stop the run.
* [x] Test struct fuzzer (core repository)
* [x] Run the struct fuzzer test collection (called "struct") in the example project
in "core/fuzzing/testdata/projects/example/" using (A crash
should be found quickly):
* Build Script: `make`
* Build Image: cifuzz/builder-cppcms
* [x] Code Coverage: Navigate to the "Overview" page and click on the "Coverage" card to verify that coverage exists.
* [x] (VS Code) Debug the crash and verify that you can step through the code
and see the value of the struct that caused the crash.

* [x] Test string instrumentation in the example java project
* [x] Create and initialize the project (found in the core repo:
fuzzing/testdata/projects/java_example) with the settings:
* build script:

       mkdir -p build/libs
       javac $(find src -name "*.java") -d build
       jar cf build/libs/example.jar -C build com

* image: cifuzz/builder-maven
* [x] Select the Fuzz Test that is configured in the proejct and run it.
* [x] (VS Code) Test debugging the finding, and step into the code until
you reach the bug location.
* [x] Test WebGoat
* [x] Create and initialize the fuzzing project. Once completed, you should see a view to select a web app to analyze. Select "StartWebGoat" and click "Analyze Web App", and analyze it wiht the defaults
* Build script: `mvn clean package -DskipTests`
* Build image: cifuzz/builder-maven
* In the "Select and configure Web Applications" step of the project
wizard make sure that the "StartWebGoat" Artifact is selected
and "Spring / Spring Boot" is automatically selected
* Click "Analyse" and verify that no error is reported
* [x] Add a new "Web App Fuzz Test"
* A list of found URLs should displayed in the wizard
* Select only the `/challenge/5` controller and save the fuzz target
* [x] Start created Fuzz Test by selecting it and clicking "Run".
* [x] Verify that there are multiple warnings reported from the ZAP
run.
* [x] Click on the "Overview" page and verify that coverage is reported
* [ ] Test AltoroJ
* [x] Run AltoroJ locally with Tomcat
* [x] If not installed yet download extract Tomcat 7 or 8 (https://tomcat.apache.org/download-80.cgi)
* [x] Build the AltoroJ application with `./gradlew build`
* [x] Copy the web application `build/libs/altoromutual.war` to `<Tomcat_install_dir>/web_apps/AltoroJ.war`
* [x] Start the tomcat server with `<Tomcat_install_dir>/bin/startup.sh`
* [x] The AltoroJ application should be available on `http://localhost:8080/AltoroJ`
* [ ] Initialize the fuzzing project AltoroJ
* [x] Select the "Configure project for web application fuzzing" option
to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name for the service
* [x] Select "com.ibm" as java packages to be instrumented
* [x] Copy the generated java agent argument `-javaagent ...`
* [x] Stop the running Tomcat server (`<Tomcat_install_dir>/bin/shutdown.sh`)
* [x] Start it again with the copied java agent argument `JAVA_OPTS=<copied_command> <Tomcat_install_dir>/bin/startup.sh`
* [x] The web service should appear in the list of web services
* [x] Click on the web service and perform the web service analysis using the OpenAPI
definition `WebContent/swagger/properties.json`.
* [x] Create a new java agent fuzz test and select the web service configured before
* [x] Create a file `.code-intelligence/fuzz_targets/<name_of_fuzz_test>_headers.sh`
and insert the following content:

    token=$(curl -s -X POST "http://localhost:8080/AltoroJ/api/login" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{  \"username\": \"jsmith\",  \"password\": \"demo1234\"}" | jq .Authorization | tr -d '"')
    echo "Authorization: $token"

You can execute the script to verify that the Authorization
token is successfully extracted
* [x] Run the create fuzz test and verify that an SQL injection and more then 10 exceptions are found.
* [x] Verify that ZAP findings are reported (might take ~2 minutes)
* [x] Verify that code coverage is reported
* [x] Test the unit test runner
* [x] Initialize the project `yaml-cpp` from the fuzz_testing repo
* Build script:

      #!/bin/bash -eu
      mkdir -p build && cd build
      cmake -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ..
      make -j$(nproc)

* Build image: cifuzz/builder-zint
* [x] Create a new fuzz target of type "unit test".
* [x] (VS CODE) Click "SAVE AND OPEN FILE" and enter the following test code in the Bash
template that should open:

        build/test/run-tests

* [x] Run the created Fuzz Test
* [x] (VS Code) Make one of the unit tests fail (e.g. insert `EXPECT_TRUE(false);` in `test/regex_test.cpp:14`)
* [x] Run the fuzz test and check if the failed test is reported as a bug
* [x] (VS Code) Make one of the unit tests crash (e.g. insert `*(char *) 1 = 2;` in `test/regex_test.cpp:14`)
* [x] Run the fuzz test and check if the asan report shows up in findings