# Manual test checklist

To best simulate the environment that our users will use our software
in, the manual tests should be done with the tools that are installed by
our installer.

The installation directory is referenced as `CI_FUZZ_INSTALL_DIR` in this checklist.

Download the relevant tentative release from the GitLab [releases](https://gitlab.code-intelligence.com/code-intelligence/core/-/releases)
page.

Alternatively, build and run the installer locally:

INSTALLER=$(./scripts/release/build-installer.sh) && \
./"${INSTALLER}" --non-interactive --install-vscode-extension

Then start the local server (with verbose logging to ease debugging):

ci-daemon -v2 --alsologtostderr

### Nightly Build
* [x] Check the CI/CD pipeline

### Local
Manually test CI Fuzz with projects in the [fuzz-testing repository][1].

[1]: https://gitlab.code-intelligence.com/code-intelligence/fuzz-testing

Before starting, make sure you set these variables in the `.env` file at the root of the core repository:

DOCKER_REGISTRY
DOCKER_REGISTRY_USER
DOCKER_REGISTRY_PASSWORD

If you don't have a `.env` file yet, copy `.env.template` to `.env` and replace the placeholders.

When you start to test a project, write your name in parenthesis behind the "Test $PROJECT" bullet item.

* [x] (Lukas) Test CppCMS
* [x] Test creating and initializing the CppCMS project:
* Build Script:

      #!/bin/bash -eu
      mkdir build
      cd build
      cmake -DCMAKE_TESTING_ENABLED=OFF -DDISABLE_SHARED=ON -DDISABLE_GCRYPT=ON -DDISABLE_OPENSSL=ON -DDISABLE_FCGI=ON -DDISABLE_SCGI=ON -DDISABLE_HTTP=ON -DDISABLE_CACHE=ON -DDISABLE_TCPCACHE=ON -DDISABLE_GZIP=ON ..
      make -j$(nproc)

* Build Image: `cifuzz/builder-cppcms`
* [x] (VS Code) Test the fuzz-this-function functionality by trying to generate a fuzz target from an API function.
* [x] Start the "Fuzz Json" fuzz test by selecting it and clicking "Run".
* [x] Check that charts are showing and updating once the test collection is running.
* [x] Code Coverage: Navigate to the "Overview" page and click on the "Coverage" card to verify that coverage exists. (You might need to stop the run for that.)
* [x] (VS Code) Insert a bug at line 1220 in the file "src/json.cpp"
(e.g., `*(char*)2=1;`). Re-run the campaign to produce a crash
finding. Debug the found crash.
* [x] (VS Code) Click on "All Findings" then click on the finding in the table. Make sure that clicking on the file link navigates you to the erroneous line.
* [x] (VS Code) Test if debugging the finding works and you can step to the error introduced
* [x] Test grammar fuzzer by clicking on the `fuzz_json_grammar` target and clicking "run" or "start run"
* [x] (Lukas) Test nginx (testing the socket fuzzer)
* [x] Test creating and initializing a fuzzing project using
* The build script:

      #!/bin/bash -eu
      ./auto/configure
      mkdir -p logs
      make

* The build image: cifuzz/builder-nginx
* [x] Create a new socket fuzz test using the "add fuzz target"
functionality in the UI.
* [x] Select TCP port 6666 and Raw data.
* [x] Select the "Binary" `nginx` from the dropdown menu.
* [x] Use the following run arguments:

      -p
      $SRC
      -c
      .code-intelligence/nginx2.conf

* [x] Save it and add the following to the fuzz target yaml file at `.code-intelligence/fuzz_targets/<fuzz-test-name>.yaml`:

      engine_options:
        libfuzzer:
          - "-detect_leaks=0"

* [x] Run the the created fuzz test by selecting it and clicking "run" or "start run".
* [x] Stop the run.
* [x] (Lukas) Test struct fuzzer (core repository)
* Use the VS Code Extension Wizard to set up example project `core/fuzzing/testdata/projects/example/`
* Build Script: `make`
* Build Image: cifuzz/builder-cppcms
* [x] Run the struct fuzzer (called "do_struct_fuzzer") in the example project
(A crash should be found quickly)
* [x] Code Coverage: Navigate to the "Overview" page and click on the "Coverage" card to verify that coverage exists.
* [x] (VS Code) Debug the crash and verify that you can step through the code
and see the value of the struct that caused the crash.

* [x] (Lukas) Test string instrumentation in the example java project
* [x] Create and initialize the project (found in the core repo:
fuzzing/testdata/projects/java_example) with the settings:
* build script:

       mkdir -p build/libs
       javac $(find src -name "*.java") -d build
       jar cf build/libs/example.jar -C build com

* image: cifuzz/builder-maven
* [x] Select the fuzz test "FuzzParser" that is already configured and run it.
* [x] (VS Code) Test debugging the finding, and step into the code until you reach the bug location.
* [x] (Willian) Test WebGoat
* [x] Build WebGoat manually using maven

    mvn clean package -DskipTests

* [x] Initialize a fuzzing project for WebGoat
* [x] Select the "Configure project for web application fuzzing" option to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name for the service
* [x] Select "org.owasp" as java packages to be instrumented
* [x] Copy the generated java agent argument `-javaagent ...`
* [x] Run

      java -jar <javaagent_command> webgoat-server/target/webgoat-server-8.0.0-SNAPSHOT.jar

* [x] WebGoat should start and appear in the list of web services
* [x] Add a new "Web App Fuzz Test"
* [x] Select your webservice, leave all other default options
* [x] In a browser navigate to `http://localhost:8080/WebGoat` and create a user with username `testuser` and password `testpass`
* [x] In the file `.code-intelligence/fuzz_targets/<fuzz-target-name>_initial_request.http` insert the following login request

    POST /WebGoat/login HTTP/1.1
    Host: localhost:8080
    Content-Type: application/x-www-form-urlencoded
    Content-Length: 35
    Origin: http://localhost:8080
    Referer: http://localhost:8080/WebGoat/login

    username=testuser&password=testpass

* [x] Start the created Fuzz Test by selecting it and clicking "Run".
* [x] A "Possible SQL Injection" finding and multiple exceptions should be reported
* [x] Verify that the cURL command for the SQL Injection finding reproduces the SQL Exception
* [x] Verify that there are multiple warnings reported from the ZAP run (might take up to 3 minutes).
* Note: ZAP run findings have a locations like `POST http://127.0.0.1:8080/[…]` or `GET http://127.0.0.1:8080/[…]`, examples include types like
* "Application Error Disclosure",
* "Information Disclosure […]",
* "Cookie without SameSite Attribute",
* "Cookie No HttpOnly Flag" or
* "Charset Mismatch […]".
* [x] Click on the "Overview" page and verify that coverage is reported
* [x] Click on "Code Coverage" and check that the coverage view works
* [x] Click on some file that has 0%<coverage<100% and check if source code opens and some lines are highlighted and they make sense

* [x] (Willian) Test AltoroJ
* [x] Run AltoroJ locally with Tomcat

Note that there are helper scripts for more convenience in the ``checklist`` folder.
* [x] If not installed yet download extract Tomcat 7 or 8 (https://tomcat.apache.org/download-80.cgi)
* [x] Build the AltoroJ application with `./gradlew build`. This requires using JDK 8.
* [x] Copy the web application `build/libs/altoromutual.war` to `<Tomcat_install_dir>/webapps/AltoroJ.war`
* [x] Start the tomcat server with `<Tomcat_install_dir>/bin/startup.sh`
* [x] The AltoroJ application should be available on `http://localhost:8080/AltoroJ`
* [x] Initialize the fuzzing project AltoroJ
* [x] Select the "Configure project for web application fuzzing" option
to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name for the service
* [x] Select "com.ibm" as java packages to be instrumented
* [x] Copy the generated java agent argument `-javaagent ...`
* [x] Stop the running Tomcat server (`<Tomcat_install_dir>/bin/shutdown.sh`)
* [x] Start it again with the copied java agent argument `JAVA_OPTS=<copied_command> <Tomcat_install_dir>/bin/startup.sh`.
Remember to replace `CI_FUZZ_INSTALL_DIR` with your actual installation directory.
Verify by checking for `Instrumented ...` messages in Tomcat's logs (`logs/catalina.out`).
* [x] The web service should appear in the list of web services.
* [x] In the `.code-intelligence/web_services.yaml` configure the OpenAPI
spec for the added web service: `WebContent/swagger/properties.json`.
* [x] Create a new "Web App Fuzz Test" and select the web service configured before
* [x] Create a file `.code-intelligence/fuzz_targets/<name_of_fuzz_test>_headers.sh` and insert the following content:

    token=$(curl -s -X POST "http://localhost:8080/AltoroJ/api/login" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{  \"username\": \"jsmith\",  \"password\": \"demo1234\"}" | jq .Authorization | tr -d '"')
    echo "Authorization: $token"

You can execute the script to verify that the Authorization token is successfully extracted
* [x] Run the created fuzz test and verify that an SQL injection and more than 10 exceptions are found.
* [x] Verify that ZAP findings are reported (might take ~2 minutes)
* [x] Verify that code coverage is reported

* [x] (Lukas) Test 32-bit fuzzing
* [x] Initialize the project `32-bit_fuzzing` from the fuzz-testing repo
* Build script is: ```clang software_under_test.c -c -m32 -o software_under_test.o```
* Docker image is: `cifuzz/builder-c:1.1`
* [x] Execute the fuzz test `32-bit_fuzz_test` and check if a double free is found

* [x] (Lukas) Test file input fuzzing
* [x] Initialize the project `file_input_fuzzing` from the fuzz-testing repo
* Build script can be empty
* [x] Execute the fuzz test `file_input_fuzzing_64-bit` and check if a stack buffer overflow is found

* [x] (Willian) Test fuzzing with AFL
* [x] Initialize the project `afl_fuzzing` from the fuzz-testing repo
* Build script can be empty
* [x] Execute the fuzz test `afl_fuzzing_64-bit` and check if a buffer overflow is found
* [x] Execute the fuzz test `afl_fuzzing_32-bit` and check if a buffer overflow is found (Please note: If the 32-bit test fails, one needs to build AFL from source during the installation by adding the flag `-b` when invoking the installer. You also need to make sure that your 64-bit system is capabale of cross-compiling 32-bit binaries by installing the appropriate libraries with `apt install gcc-multilib g++-multilib`)

* [x] (Lukas) Test Cmake Integration
* [x] Open the `cmake_integration` folder from the fuzz-testing repo and adjust the `CIFUZZ_INSTALL_ROOT` in `build_fuzzing_artifacts_and_run_fuzz_test.sh`
* [x] Run the `build_fuzzing_artifacts_and_run_fuzz_test.sh` script
* [x] Confirm that the script was run with the correct version of CI Fuzz, that the fuzz test was build successfully and that the crashing input starts with "CI FUZZ"

* [x] (Willian) Test java-grpc fuzzing with greeter service
* [x] In the core repo, build the greeter service by running

    ./bazelisk.py build //pkg/web_app/tests/integration/grpc/testdata/java/greeter_service:GreeterServer_deploy.jar

* [x] In the core repo, run

    ci-protoc libproto_stub.so  -Ipkg/web_app/tests/integration/grpc/testdata/common/greeter_service  pkg/web_app/tests/integration/grpc/testdata/common/greeter_service/helloworld/a/b/c/d/greeter_service.proto  pkg/web_app/tests/integration/grpc/testdata/common/greeter_service/helloworld/a/b/c2/d2/greeter_messages.proto   --field_hint=authentication.access_token=let_me_in  --field_hint=authentication.access_token=backdoor

to build the libproto_stub.so.
* [x] Move libproto_stub.so the to greeter service project dir

    mv libproto_stub.so pkg/web_app/tests/integration/grpc/testdata/java/greeter_service

* [x] Open the greeter service directory `pkg/web_app/tests/integration/grpc/testdata/java/greeter_service` in VS Code and initialize a web application fuzzing project
* [x] Add a new webservice
* [x] Chose a name (e.g. `test`) and instrument `com.example`
* [x] Copy the generated java agent argument `-javaagent ...`
* [x] Start the previously build jar with the java agent command and specify the port

    java <javaagent_command> -Dserver.port=12345 -jar bazel-bin/pkg/web_app/tests/integration/grpc/testdata/java/greeter_service/GreeterServer_deploy.jar

* [x] Add a new "Web App Fuzz Test"
* [x] Select the previously created webservice
* [x] Modify the fuzz test config `.code-intelligence/fuzz_targets/<fuzztestname>.yaml` and replace the content with (change name of webservice from `test` if necessary)

    type: "java web app"
    run_extra_args:
      - "--proto_stub_path=libproto_stub.so"
    web_services:
      - "test"
    base_url: "127.0.0.1:12345"
    protocol: grpc

* [x] Create the file `.code-intelligence/fuzz_targets/<fuzztestname>_headers.http` with the content

    Authorization: Bearer letmein

* [x] Start the fuzz test
* [x] Verify that the fuzzing metrics graphs are shown
* [x] Verify that you get 3 findings: Two are a `Security Exception`, one with `access_token: "backdoor"`, and one is a `Dereference of Null Object` with `access_token: "let_me_in"` in the Human-readable input.
* [x] Verify that the grpcurl reproducer commands work by running them. The output should be

    ERROR:
      Code: Unknown
      Message:

* [x] Stop the run (if not already done)
* [x] Go to "Code Coverage" and verify the coverage metrics are shown.

* [x] (Lukas) Test tidb (currently branch go-in-process)
* [x] Checkout branch `go-in-process` in repository "fuzz-testing":

    git checkout go-in-process

* [x] Change directory to `tidb` and run `go mod tidy`:

    cd tidb
    go mod tidy

* [x] Test creating and initializing a fuzzing project using
* [x] The build script:

      exit 0

* [x] Build in local mode
* [x] Under "Additional Docker volume mounts", add mount to your `GO_PATH` (usually `$HOME/go` to `$HOME/go`)
* (Note: if your local go installation lives somewhere else, change "Source" accordingly. "Destination" still needs to be `$HOME/go`.)
* If the input makes problem, ask Niklas to schedule work on [APP-471](https://code-intelligence.atlassian.net/browse/APP-471).
* Then edit `.code-intelligence/project.yaml` manually…
* [x] Enable the hidden Fuzz Targets:
* [x] Fuzz Target creation in VSCODE UI is not supported and not planned. You need to add project.yaml entry:

      go_fuzz_packages:
      - github.com/pingcap/tidb/types

* [x] In the VSCode UI you should see afterwards that three fuzztargets were found in the specified package.
* [x] Run the FuzzMarshalJSON target and verify that fuzzing metrics graphs are shown
* [x] Run the FuzzNewHexLiteral target and verify that we get a "Go Panic" finding
* [x] Verify that you can jump to the bug location from the UI
* [x] Verify that the stack trace contains three elements and that you can jump to the corresponding location
* [x] Add a new fuzz test to the file types/fuzzNewHexLiteral.go (copy/paste/rename the existing fuzz test FuzzNewHexLiteral). Verify that the newly added fuzz test appears in the list of fuzz targets and that you can go to the file from the UI. (This takes several seconds, have patience!)

* [x] (Willian) Test `go-grpc` fuzzing with greeter service
* [x] In the [grpc-go](https://gitlab.code-intelligence.com/code-intelligence/fuzz-testing/-/tree/master/grpc-go) project in the fuzz-testing repo, build `libproto_stub.so` via:

    ci-protoc libproto_stub.so -Ihelloworld/helloworld/ helloworld/helloworld/helloworld.proto

* [x] Create and initialize a fuzzing project in VS Code
* [x] Select the "Configure project for web application fuzzing" option
to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name e.g. go-grpc-server for the service and click "Next"
* [x] Click on "go" and follow the instructions to execute `ci-gofuzz` in your terminal.
You will have to make the following changes to the command:

1. Replace `CI_FUZZ_INSTALL_DIR` with the actual path of your CIFuzz installation
2. Add `google.golang.org/grpc/examples/helloworld*` to the list of included packages
3. Replace the path to the main package with `./helloworld/greeter_server`
* [x] Start the SUT with the command shown in the UI, again replacing `CI_FUZZ_INSTALL_DIR` with the actual installation directory.
* [x] Add a Fuzz test with name `test` and modify the fuzz test config `.code-intelligence/fuzz_targets/test.yaml` and replace the content with

    type: "java web app"
    run_extra_args:
      - "--proto_stub_path=libproto_stub.so"
    web_services:
      - "go-grpc-server"
    base_url: "127.0.0.1:50051"
    protocol: grpc
    tag_header: X-Request-Id

* [x] Start the fuzz test
* [x] Verify that you get 2 findings:
* [x] Verify that the fuzz target stops.
* [x] One is a `Go Panic` with `name: "Critical Bug"` and the second a `Crash (exit code: 2)` also with `name: "Critical Bug"` in the Human-readable input. Check that the first one has reasonable SUMMARY, DESCRIPTION, STACK TRACE and LOG in the finding. The second finding is expected to only have Content in SUMMARY and DESCRIPTION.
* [x] Check that the summary contains a request tag header called `X-Request-Id` as specified in the fuzz test config.
* [x] Verify that you can jump to the bug location from the UI
* [x] Verify that one of the grpcurl reproducer commands work by running them after you have restarted the SUT. It will not work, because the SUT does not support the reflection API. You will need to add the proto file path to the grpcurl command `-proto helloworld/helloworld/helloworld.proto -import-path helloworld/helloworld`. The command should look similar to

    grpcurl \
    -proto helloworld/helloworld/helloworld.proto -import-path helloworld/helloworld \
    -plaintext \
    -d '{"name":"Critical Bug"}' \
    127.0.0.1:50051 helloworld.Greeter/SayHello

`grpcurl` should print `Code: Unavailable` and the SUT should have crashed.
* [x] Comment out the vulnerability (lines 45-47) in grpc-go/helloworld/greeter_server/main.go and recompile the fuzz target using the same invocation as before.
* [x] Restart the application once more using the same invocation as before.
* [x] Set fuzz target runtime to 5 minutes in project.yaml
* [x] Set coverage interval to 10000 by adding `coverage_update_interval: 10000` to the fuzz target yaml.
* [x] Start the fuzz test
* [x] Verify that the fuzzing metrics graphs are shown.
* [x] Verify that the Source Code Coverage is available during runtime after a minute and that clicking on a source file name opens the corresponding file.
* [x] Stop the fuzz target. Let the target application run withouth termination.
* [x] Check that the webservice light goes back to green again
* [x] Restart fuzzing and check that fuzzing is successfully started again.