# Manual test checklist

To best simulate the environment that our users will use our software
in, the manual tests should be done with the tools that are installed by
our installer.

To build and run the installer, run:

INSTALLER=$(./scripts/release/build-installer.sh) && \
"${INSTALLER}" --non-interactive --install-vscode-extension

Then start the local server (with verbose logging to ease debugging):

ci-daemon -v2 --alsologtostderr

### Nightly Build
* [x] Check the CI/CD pipeline

### CI/CD scripts
* [x] Test CI/CD scripts (Willian)
* [x] Start the `release-cicd-test` VM in GCloud.
* [x] SSH into the `release-cicd-test` VM, download the installer
from the tentative release and install it.

    gcloud beta compute ssh --zone "europe-west4-a" "release-cicd-test"  --project "ci-fuzz"
    curl -o installer "<installer-URL>"
    chmod +x installer
    ./installer -n

* [x] Test the GitHub action
* At the [github-actions](https://github.com/CodeIntelligenceTesting/github-actions)
repository, merge the changes from the `v3` branch onto the `dev` branch.
* Update the images in the actions on the `dev` branch to use `<CICTL_VERSION>`
instead of `v3`, for example: `image: "docker://cifuzz/github-action:v3"`
will be updated to `image: "docker://cifuzz/github-action:<CICTL_VERSION>"`,
where `<CICTL_VERSION>` is the new cictl version.
* At release-cicd-test.code-intelligence.com, generate a GitHub Action
CI/CD script for the `zint` project and use it at
github.com/ci-fuzz/zint, branch `release-test`.
* Replace
`${{ secrets.CI_FUZZ_API_TOKEN }}` by
`${{ secrets.CI_FUZZ_RELEASE_TEST_API_TOKEN }}`,
`branches: [ master ]` by `branches: [ release-test ]` in `push` and
`@v3` with `@dev`.
* Check if the action will run and a finding is encountered.
* [x] At release-cicd-test.code-intelligence.com, generate a GitLab CI/CD
script for the `zint-gitlab` project and use it at
gitlab.com/WillRoque/zint, branch `master`. Replace
`cifuzz/gitlab:v3` with `cifuzz/gitlab:<CICTL_VERSION>` where
`<CICTL_VERSION>` is the new cictl version. Update the
`.gitlab-ci.yaml` file of the repository and check if the
pipeline runs and a finding is encountered.
* [x] At release-cicd-test.code-intelligence.com, generate a Jenkins
CI/CD script for the `zint-gitlab` project. Update the
configuration of
https://jenkins.demo.code-intelligence.com/job/cifuzz/ with the
recent script. Add and random value to the end of the value of
`CICTL` so the new cictl will be downloaded, replace the
`CICTL_URL` value with
`https://s3.eu-central-1.amazonaws.com/public.code-intelligence.com/cictl/cictl-<CICTL_VERSION>-linux`
and `credentials('CI_FUZZ_API_TOKEN')` with
`credentials('CI_FUZZ_API_TOKEN_RELEASE')`. Build and check the
output, it should encounter a finding.
* [x] Stop the `release-cicd-test` VM in GCloud.

### Local
Manually test CI Fuzz with projects in the [fuzz-testing repository][1].

[1]: https://gitlab.com/code-intelligence/fuzz-testing

Before starting, make sure you set these variables in the `.env` file at
the root of the core repository:

DOCKER_REGISTRY
DOCKER_REGISTRY_USER
DOCKER_REGISTRY_PASSWORD

If you don't have a `.env` file yet, copy `.env.template` to `.env` and
replace the placeholders.

When you start to test a project, write your name in parenthesis behind
the "Test $PROJECT" bullet item.

* [x] Test CppCMS (Dae)
* [x] Test creating and initializing the CppCMS project:
* Build Script:

      #!/bin/bash -eu
      mkdir build
      cd build
      cmake -DCMAKE_TESTING_ENABLED=OFF -DDISABLE_SHARED=ON -DDISABLE_GCRYPT=ON -DDISABLE_OPENSSL=ON -DDISABLE_FCGI=ON -DDISABLE_SCGI=ON -DDISABLE_HTTP=ON -DDISABLE_CACHE=ON -DDISABLE_TCPCACHE=ON -DDISABLE_GZIP=ON ..
      make -j$(nproc)

* Build Image: `cifuzz/builder-cppcms`
* [x] (VS Code) Test the fuzz-this-function functionality by trying to generate a
fuzz target from an API function.
* [x] Start the "Fuzz Json" fuzz test by selecting it and clicking "Run".
* [x] Check that charts are showing and updating once the test
collection is running.
* [x] Code Coverage: Navigate to the "Overview" page and click on the "Coverage" card to verify that coverage exists.
* [x] (VS Code) Insert a bug at line 1220 in the file "src/json.cpp"
(e.g., `*(char*)2=1;`). Re-run the campaign to produce a crash
finding. Debug the found crash.
* [x] (VS Code) Click on "All Findings" then click on the finding in the table. Make sure that clicking on the file link in the view would navigating to the erroneous line.
* [x] (VS Code) Test if debugging the finding works and you can step to the error introduced
* [x] Test grammar fuzzer by clicking on the "fuzz_json_grammar" target and clicking "run" or "start run"
* [x] Test nginx (testing the socket fuzzer) (jan)
* [x] Test creating and initializing a fuzzing project using
* The build script:

         #!/bin/bash -eu
         ./auto/configure
         mkdir -p logs
         make

* The build image: cifuzz/builder-nginx
* [x] Create a new socket fuzz test using the "add fuzz target"
functionality in the UI.
* [x] Select TCP port 6666 and Raw data.
* [x] Select the `nginx` binary from the dropdown menu.
* [x] Use the following run arguments: `-p $SRC` and `-c .code-intelligence/nginx2.conf`, each on a new line.
* [x] Save it and add the following to the fuzz target yaml file at `.code-intelligence/fuzz_targets`:

      engine_options:
        libfuzzer:
          - "-detect_leaks=0"

* [x] Run the the created fuzz test by selecting it and clicking "run" or "start run".
* [x] Stop the run.
* [x] Test struct fuzzer (core repository) (jan)
* [x] Run the struct fuzzer (called "do_struct_fuzzer") in the example project
in "core/fuzzing/testdata/projects/example/" using (A crash
should be found quickly):
* Build Script: `make`
* Build Image: cifuzz/builder-cppcms
* [x] Code Coverage: Navigate to the "Overview" page and click on the "Coverage" card to verify that coverage exists.
* [x] (VS Code) Debug the crash and verify that you can step through the code
and see the value of the struct that caused the crash.

* [x] Test string instrumentation in the example java project
* [x] Create and initialize the project (found in the core repo:
fuzzing/testdata/projects/java_example) with the settings:
* build script:

       mkdir -p build/libs
       javac $(find src -name "*.java") -d build
       jar cf build/libs/example.jar -C build com

* image: cifuzz/builder-maven
* [x] Select the Fuzz Test that is configured in the proejct and run it.
* [x] (VS Code) Test debugging the finding, and step into the code until
you reach the bug location.
* [x] Test WebGoat (Vojtech)
* [x] Build WebGoat manually using maven `mvn clean package -DskipTests`
* [x] Initialize a fuzzing project for WebGoat
* [x] Select the "Configure project for web application fuzzing" option
to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name for the service
* [x] Select "org.owasp" as java packages to be instrumented
* [x] Copy the generated java agent argument `-javaagent ...`
* [x] Run `java -jar <javaagent command> webgoat-server/target/webgoat-server-8.0.0-SNAPSHOT.jar`
* [x] WebGoat should start and appear in the list of web services
* [x] Add a new "Java Web App Fuzz Test"
* [x] In a browser navigate to `http://localhost:8080/WebGoat` and create a user with username `testuser` and password `testpass`
* [x] In the file `.code-intelligence/fuzz_targets/<fuzz-target-name>_initial_request.http` insert the following
login request

    POST /WebGoat/login HTTP/1.1
    Host: localhost:8080
    Content-Type: application/x-www-form-urlencoded
    Content-Length: 35
    Origin: http://localhost:8080
    Referer: http://localhost:8080/WebGoat/login

    username=testuser&password=testpass

* [x] Start the created Fuzz Test by selecting it and clicking "Run".
* [x] A "Possible SQL Injection" finding and multiple exceptions should be reported
* [x] Verify that the cURL command for the SQL Injection finding reproduces the SQL Exception
* [x] Verify that there are multiple warnings reported from the ZAP
run (might take up to 3 minutes).
* [x] Click on the "Overview" page and verify that coverage is reported
* [x] Test AltoroJ (Simon)
* [x] Run AltoroJ locally with Tomcat
* [x] If not installed yet download extract Tomcat 7 or 8 (https://tomcat.apache.org/download-80.cgi)
* [x] Build the AltoroJ application with `./gradlew build`
* [x] Copy the web application `build/libs/altoromutual.war` to `<Tomcat_install_dir>/web_apps/AltoroJ.war`
* [x] Start the tomcat server with `<Tomcat_install_dir>/bin/startup.sh`
* [x] The AltoroJ application should be available on `http://localhost:8080/AltoroJ`
* [x] Initialize the fuzzing project AltoroJ
* [x] Select the "Configure project for web application fuzzing" option
to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name for the service
* [x] Select "com.ibm" as java packages to be instrumented
* [x] Copy the generated java agent argument `-javaagent ...`
* [x] Stop the running Tomcat server (`<Tomcat_install_dir>/bin/shutdown.sh`)
* [x] Start it again with the copied java agent argument `JAVA_OPTS=<copied_command> <Tomcat_install_dir>/bin/startup.sh`
* [x] The web service should appear in the list of web services
* [x] In the `.code-intelligence/web_services.yaml` configure the OpenAPI
spec for the added web service: `WebContent/swagger/properties.json`.
* [x] Create a new "Java Web App Fuzz Test" and select the web service configured before
* [x] Create a file `.code-intelligence/fuzz_targets/<name_of_fuzz_test>_headers.sh` and insert the following content:

    token=$(curl -s -X POST "http://localhost:8080/AltoroJ/api/login" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{  \"username\": \"jsmith\",  \"password\": \"demo1234\"}" | jq .Authorization | tr -d '"')
    echo "Authorization: $token"

You can execute the script to verify that the Authorization token is successfully extracted
* [x] Run the create fuzz test and verify that an SQL
injection and more then 10 exceptions are found.
* [x] Verify that ZAP findings are reported (might take ~2 minutes)
* [x] Verify that code coverage is reported
* [x] Test file input fuzzing (Simon)
* [x] Initialize the project `file_input_fuzzing` from the fuzz_testing repo
* Build script can be empty
* [x] Execute the fuzz test file_input_fuzzing_64-bit and check if a stack buffer overflow is found
* [ ] Test fuzzing with afl (Simon) NOT WORKING! Sceduled for next sprint
* [x] Initialize the project `afl_fuzzing` from the fuzz_testing repo
* Build script can be empty
* [ ] Execute the fuzz test AFL_fuzzing_64-bit and check if a stack buffer overflow is found
* [ ] Edit the project.yaml so that instead of afl_llvm_mode afl is used as the fuzzing engine
* [ ] Execute the fuzz test AFL_fuzzing_64-bit and check if a stack buffer overflow is found
* [x] Test the unit test runner (jan)
* [x] Initialize the project `yaml-cpp` from the fuzz_testing repo
* Build script:

      #!/bin/bash -eu
      mkdir -p build && cd build
      cmake -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ..
      make -j$(nproc)

* Build image: cifuzz/builder-zint
* [x] Create a new fuzz target of type "unit test".
* [x] (VS CODE) Click "SAVE AND OPEN FILE" and enter the following test code in the Bash
template that should open:

        build/test/run-tests

* [x] Run the created Fuzz Test
* [x] (VS Code) Make one of the unit tests fail (e.g. insert `EXPECT_TRUE(false);` in `test/regex_test.cpp:14`)
* [x] Run the fuzz test and check if the failed test is reported as a bug
* [x] (VS Code) Make one of the unit tests crash (e.g. insert `*(char *) 1 = 2;` in `test/regex_test.cpp:14`)
* [x] Run the fuzz test and check if the asan report shows up in findings

* [x] Test java-grpc fuzzing with greeter service (Simon)
* [x] In the core repo, build the greeter service by running

        ./bazelisk.py build //pkg/web_app/tests/integration/grpc/testdata/java/greeter_service:GreeterServer_deploy.jar

* [x] In the core repo, run

        ci-protoc libproto_stub.so  -Ipkg/web_app/tests/integration/grpc/testdata/common/greeter_service  pkg/web_app/tests/integration/grpc/testdata/common/greeter_service/helloworld/a/b/c/d/greeter_service.proto  pkg/web_app/tests/integration/grpc/testdata/common/greeter_service/helloworld/a/b/c2/d2/greeter_messages.proto   --field_hint=authentication.access_token=let_me_in  --field_hint=authentication.access_token=backdoor

to build the libproto_stub.so.
* [x] Move libproto_stub.so the to greeter service project dir
`mv libproto_stub.so pkg/web_app/tests/integration/grpc/testdata/java/greeter_service`
* [x] Open the greeter service directory `pkg/web_app/tests/integration/grpc/testdata/java/greeter_service` in VS Code and initialize a web application fuzzing project
* [x] Add a new webservice with the name `test` and start the previously build jar with the java agent command and specify the port

        java -javaagent*** -Dserver.port=12345 -jar bazel-bin/pkg/web_app/tests/integration/grpc/testdata/java/greeter_service/GreeterServer_deploy.jar

* [x] Add a Fuzz test with name `test` and modify the fuzz test config .code-intelligence/fuzz_targets/test.yaml and replace the content with

    type: "java web app"
    run_extra_args:
      - "--proto_stub_path=libproto_stub.so"
    web_services:
      - "test"
    base_url: "127.0.0.1:12345"
    protocol: grpc

* [x] Create the file `.code-intelligence/fuzz_targets/test_headers.http` with the content `Authorization: Bearer letmein`
* [x] Start the fuzz test
* [x] Verify that the fuzzing metrics graphs are shown
* [x] Verify that you get 3 findings: Two are a `Security Exception` with `access_token: "backdoor"` and the second a `Dereference of Null Object` with `access_token: "let_me_in"` in the Human-readable input.
* [x] Verify that the grpcurl reproducer commands work by running them. The output should be

        ERROR:
          Code: Unknown
          Message:

* [x] Go to "Code Coverage" and verify the coverage metrics are shown.

* [ ] Test tidb (currently branch go-in-process) (Simon)
* [x] Test creating and initializing a fuzzing project using
* [x] The build script:

      exit 0 

* [x] Build in local mode
* [x] Add mount to GO_PATH (usually $HOME/go to $HOME/go)
* [x] Create a Fuzz Target.
* [x] Fuzz Target creation in VSCODE UI is not supported and not planned. You need to add project.yaml entry:

      go_fuzz_packages:
        - github.com/pingcap/tidb/types

* [x] In the VSCode UI you should see afterwards that three fuzztargets were found in the specified package.
* [x] Run the FuzzMarshalJSON target and verify that fuzzing metrics graphs are shown
* [ ] Run the FuzzNewHexLiteral target and verify that we get a "Go Panic" finding
* [ ] Verify that you can jump to the bug location from the UI
* [ ] Verify that the stack trace contains three elements and that you can jump to the corresponding location
* [x] Add a new fuzz test to the file types/fuzzNewHexLiteral.go (copy/paste/rename the existing fuzz test FuzzNewHexLiteral). Verify that the newly added fuzz test appears in the list of fuzz targets and that you can go to the file from the UI

* [x] Test `go-grpc` fuzzing with greeter service (currently on branch go-out-of-process) (jan)
* [x] In the fuzz-testing repository go to grpc-go/examples and build libproto_stub.so

    cd grpc-go/examples
    ci-protoc libproto_stub.so -Ihelloworld/helloworld/ helloworld/helloworld/helloworld.proto

* [x] Create and initialize a fuzzing project in VSCode
* [x] Select the "Configure project for web application fuzzing" option
to skip the build script and docker image configuration
* [x] In the web service configuration page click on "Add Web Service"
* [x] Select a name e.g. go-grpc-server for the service
* [x] Click on "go" and follow the instructions in the terminal. They should be similar to:

      ci-gofuzz --include "google.golang.org/grpc/examples/helloworld*" \
      -cover \
      -cifuzz_agent_lib $HOME/bin/libfuzzing_agent.so \
      -o output_binary_with_instrumentation ./helloworld/greeter_server/

* [x] Start the SUT, but replace the CIFUZZ_SERVICE_NAME with your generated service name in the instructions:

      LD_LIBRARY_PATH=$HOME/bin/ GLOG_v=2 GLOG_alsologtostderr=1 \
      CIFUZZ_SERVICE_NAME=projects/grpc-go-bf1747a2/web_services/go-grpc-helloword2 \
      ./output_binary_with_instrumentation

* [x] Add a Fuzz test with name `test` and modify the fuzz test config .code-intelligence/fuzz_targets/test.yaml and replace the content with

    type: "java web app"
    run_extra_args:
      - "--proto_stub_path=./examples/libproto_stub.so"
    web_services:
      - "go-grpc-server"
    base_url: "127.0.0.1:50051"
    protocol: grpc
    tag_header: X-Request-Id

* [x] Start the fuzz test
* [x] Verify that the fuzzing metrics graphs are shown. If the crash occurs too fast it can happen that no metrics are shown
* [x] Verify that you get 2 findings:
* [x] Verify that the fuzz target stops.
* [x] One is a `Go Panic` with `name: "Critical Bug"` and the second a `Crash (exit code: 2)` also with `name: "Critical Bug"` in the Human-readable input. Check that the first one has reasonable SUMMARY, DETAILS, STACKTRACE and LOG in the finding. The second finding is expected to only have Content in SUMMARY and DETAILS.
* [x] Check that the summary contains a request tag header called `X-Request-Id` as specified in the fuzz test config.
* [x] Verify that you can jump to the bug location from the UI (not implemented in 2.28.0, skip)
* [x] Verify that one of the grpcurl reproducer commands work by running them. It will not work, because the SUT does not support the reflection API. You will need to add the proto file path to the grpcurl command ´-proto helloworld/helloworld/helloworld.proto -import-path helloworld/helloworld´. The command should look similar to

    grpcurl \
    -proto helloworld/helloworld/helloworld.proto -import-path helloworld/helloworld \
    -plaintext \
    -d '{"name":"Critical Bug"}' \
    127.0.0.1:50051 helloworld.Greeter/SayHello

* [x] Uncomment the vulnerability (lines 45-47) in grpc-go/examples/helloworld/main.go and recompile the fuzztarget again.

    ci-gofuzz --include "google.golang.org/grpc/examples/helloworld*" \
    -cover \
    -cifuzz_agent_lib $HOME/bin/libfuzzing_agent.so \
    -o output_binary_with_instrumentation ./helloworld/greeter_server/

* [x] Restart application again with correct service name:

      LD_LIBRARY_PATH=$HOME/bin/ GLOG_v=2 GLOG_alsologtostderr=1 \
      CIFUZZ_SERVICE_NAME=projects/grpc-go-bf1747a2/web_services/go-grpc-helloword2 \
      ./output_binary_with_instrumentation

* [x] Set fuzz target runtime to 5 minutes in project.yaml
* [x] Set coverage intervall to 10000 by adding ```coverage_update_interval: 10000``` to the fuzz target yaml.
* [x] Start the fuzz test
* [x] Verify that the fuzzing metrics graphs are shown.
* [x] Verfiy that the Source Code Coverage is available during runtime after a minute
* [x] Stop the fuzz target. Let the target application run withouth termination.
* [x] Check that the webservice light goes back to green again
* [x] Restart fuzzing and check that fuzzing is successfully started again.