# Chromium/V8 TSan was developed by Google for testing Chrome. Thus, when modifying TSan, it is useful to evaluate the changes on Chromium. ## V8 V8 is Chromium's JavaScript engine. It is a lightweight alternative to building the full Chromium binary when testing changes on TSan. First, [fetch](https://v8.dev/docs/source-code) v8. The following steps are based on [v8's build documentation](https://v8.dev/docs/build-gn). ### Get Chromium's LLVM If you have followed the steps above for fetching v8, you should have this folder structure in your home directory: ~/v8/v8. Run the following command to clone LLVM, download other necessary files, set up LLVM with Chromium's CMake flags, and build it. ``` tools/clang/scripts/build.py --without-android --without-fuchsia --with-ccache ``` :::warning Note: On focs-server, it seems that running a build command with all CPUs might cause the server to hang. It is preferable to perform the build with less CPUs (e.g. 96). When the script above runs to the point where it prints something like the following, the build has started, and we can cancel the build by pressing Ctrl+C. ``` [_/_] CXX ... ``` We can then manually run the build command as below, specifying the number of concurrent jobs (e.g 96). Anyway we will need to manually do this after modifying LLVM. ``` ninja -C third_party/llvm-build/Release+Asserts/ -j96 ``` ::: ### Build V8 To build V8 with our custom LLVM/TSan, run `gn args out/tsan` (you can replace `out/tsan` with any folder name), and fill in the following fields. ``` # Build arguments go here. # See "gn args <out_dir> --list" for available build arguments. dcheck_always_on = false target_cpu = "x64" is_component_build = true is_debug = false v8_enable_google_benchmark = true v8_enable_test_features = true v8_enable_fast_torque = true is_tsan = true # this should be false if you didn't get LLVM in the way described above clang_use_chrome_plugins = true # the full path must be specified here, i.e. cannot use ~/ clang_base_path = "<your home directory>/v8/v8/third_party/llvm-build/Release+Asserts/" ``` [Reference](https://chromium.googlesource.com/chromium/src/+/main/docs/clang.md#using-a-custom-clang-binary) `gn args` will open the file with vim, and after closing the file, `gn` will generate the necessary build files in `out/tsan`. Before building V8, we need to change some files. ```diff diff a/build/config/compiler/BUILD.gn b/build/config/compiler/BUILD.gn --- a/config/compiler/BUILD.gn +++ b/config/compiler/BUILD.gn @@ -2547,7 +2547,7 @@ config("optimize") { # vectorization while otherwise optimizing for size. rustflags = [ "-Copt-level=s" ] } else { - cflags = [ "-O2" ] + common_optimize_on_cflags + cflags = [ "-O3" ] + common_optimize_on_cflags # The `-O3` for clang turns on extra optimizations compared to the standard # `-O2`. But for rust, `-Copt-level=3` is the default and is thus reliable ``` ```diff diff a/build/BUILD.gn b/build/BUILD.gn --- a/BUILD.gn +++ b/BUILD.gn @@ -894,7 +894,7 @@ external_v8_defines = [ "V8_IMMINENT_DEPRECATION_WARNINGS", "V8_USE_PERFETTO", "V8_MAP_PACKING", - "V8_IS_TSAN", + # "V8_IS_TSAN", "V8_ENABLE_CONSERVATIVE_STACK_SCANNING", "V8_ENABLE_DIRECT_HANDLE", "V8_MINORMS_STRING_SHORTCUTTING", @@ -945,7 +945,7 @@ if (v8_enable_map_packing) { enabled_external_v8_defines += [ "V8_MAP_PACKING" ] } if (is_tsan) { - enabled_external_v8_defines += [ "V8_IS_TSAN" ] + # enabled_external_v8_defines += [ "V8_IS_TSAN" ] } if (v8_enable_conservative_stack_scanning) { enabled_external_v8_defines += [ "V8_ENABLE_CONSERVATIVE_STACK_SCANNING" ] diff --git a/src/base/macros.h b/src/base/macros.h index 4eb891dfcbd..db3e0d7311b 100644 --- a/src/base/macros.h +++ b/src/base/macros.h @@ -456,7 +456,7 @@ bool is_inbounds(float_t v) { // there if TSAN is enabled. #ifdef V8_IS_TSAN // EXPAND is needed to work around MSVC's broken __VA_ARGS__ expansion. -#define IF_TSAN(V, ...) EXPAND(V(__VA_ARGS__)) +#define IF_TSAN(V, ...) #else #define IF_TSAN(V, ...) #endif // V8_IS_TSAN ``` These changes are important if you want to run benchmarks and compare the score against V8 built without TSan. This is because without TSan, V8 will be built with optimization level O3, whereas V8 with TSan will be built with O2, with some additional code enabled by the `IF_TSAN` flag. The changes above ensure that V8 with TSan is built exactly the same way as without TSan. We are now ready to build V8. ``` ninja -C out/tsan -j64 d8 ``` Here, `d8` is specified so that only the command line program `d8` is built, skipping all the tests which will take up quite some time. ### V8 without TSan To build V8 without TSan, just change `is_tsan = false` in the `gn args` file. ### After modifying compiler-rt If you want to rebuild d8 after modifying some code in compiler-rt, you don't need to start from scratch. Just remove the `d8` binary and run `ninja` again to link it with the new compiler-rt library. ``` rm out/tsan/d8 ninja -C out/tsan d8 ``` ### After modifying LLVM However, if you have modified code in the LLVM framework itself, you will need to rebuild the whole codebase, since you likely want to recompile all sources with new instrumentation. ``` ninja -C out/tsan -t clean ninja -C out/tsan -j64 d8 ``` ### Benchmarks Running benchmarks is easy. For the first run, run the benchmark script specifying that it is the baseline. Then for the subsequent runs, run the script in compare mode. For example, ``` test/benchmarks/csuite/csuite.py kraken baseline out/no-tsan/d8 test/benchmarks/csuite/csuite.py kraken compare out/tsan/d8 ``` There are 3 different benchmarks: kraken, octane, sunspider. [Reference](https://v8.dev/docs/benchmarks) For comparing many different versions of V8 built with different modifications to TSan, I recommend creating a build directory for each of them. For example, `out/tsan-a` for modification A, `out/tsan-b` for modification B, etc. ## Old instructions (deprecated) Run `gn args out/build` and fill it with the following options: ``` dcheck_always_on = false is_component_build = false is_debug = false is_tsan = true target_cpu = "x64" ``` Overwrite the build system's TSan libraries with ours (modify the llvm-project path accordingly) and build d8. Always repeat this step after rebuilding llvm-project. ``` rm out/build/d8 cp ~/llvm-project/build/lib/clang/19/lib/x86_64-unknown-linux-gnu/libclang_rt.tsan* third_party/llvm-build/Release+Asserts/lib/clang/17/lib/x86_64-unknown-linux-gnu/ ninja -C out/build d8 ``` Below are steps for reproducing a data race bug on V8. We will reproduce [this bug](https://logs.chromium.org/logs/v8/buildbucket/cr-buildbucket/8786697506901472385/+/u/Num_Fuzz_-_combined__flakes_/regress-crbug-1420860). In the v8 directory, checkout the buggy version. ``` git checkout 80fc53dd3ac gclient sync ``` Run `gn args out/build` and fill it with the following options: ``` dcheck_always_on = false is_component_build = false is_debug = false is_tsan = true target_cpu = "x64" ``` Overwrite the build system's TSan libraries with ours (modify the llvm-project path accordingly) and build d8. Always repeat this step after rebuilding llvm-project. ``` rm out/build/d8 cp ~/llvm-project/build/lib/clang/19/lib/x86_64-unknown-linux-gnu/libclang_rt.tsan* third_party/llvm-build/Release+Asserts/lib/clang/17/lib/x86_64-unknown-linux-gnu/ ninja -C out/build d8 ``` Run the test case. ``` out/build/d8 --test test/mjsunit/mjsunit.js test/mjsunit/mjsunit_numfuzz.js test/mjsunit/regress/regress-crbug-1420860.js --random-seed=-452346216 --nohard-abort --exit-on-contradictory-flags --testing-d8-test-runner --no-fail --always-turbofan --assert-types --minor-mc --no-enable-bmi1 --no-lazy-feedback-allocation --no-regexp-tier-up --stress-flush-code --stack-size=434 --random-gc-interval=859 --fuzzer-random-seed=818416949 ``` I have compiled a list of [data race bug reports](https://github.com/focs-lab/v8-data-races/tree/7f1b70e3c131cbf480d196c741a2c5642217ce74/logs) on GitHub. It turns out that the bug reports are only kept on the Chromium CI server for around 2 years. Those that came earlier are already gone (hence I'm backing them up on GitHub!). The report files follow the structure below: ``` Test: mjsunit/wasm/grow-huge-memory Flags: --test /b/s/w/ir/test/mjsunit/mjsunit.js /b/s/w/ir/test/mjsunit/wasm/grow-huge-memory.js --isolate /b/s/w/ir/test/mjsunit/mjsunit.js /b/s/w/ir/test/mjsunit/wasm/grow-huge-memory.js --random-seed=-1206385236 --nohard-abort --testing-d8-test-runner --no-liftoff --stress-lazy-source-positions --no-wasm-generic-wrapper --no-wasm-lazy-compilation --no-wasm-to-js-generic-wrapper --multi-mapped-mock-allocator Command: out/build/d8 --test test/mjsunit/mjsunit.js test/mjsunit/wasm/grow-huge-memory.js --isolate test/mjsunit/mjsunit.js test/mjsunit/wasm/grow-huge-memory.js --random-seed=-1206385236 --nohard-abort --testing-d8-test-runner --no-liftoff --stress-lazy-source-positions --no-wasm-generic-wrapper --no-wasm-lazy-compilation --no-wasm-to-js-generic-wrapper --multi-mapped-mock-allocator Variant: stress Shard: 0:7 GN arguments: dcheck_always_on = false is_component_build = false is_debug = false is_tsan = true target_cpu = "x64" use_remoteexec = true v8_enable_google_benchmark = true v8_enable_test_features = true Trigger flake bisect on command line: bb add v8/try.triggered/v8_flako -p 'bisect_builder_group="client.v8"' -p 'bisect_buildername="V8 Linux64 TSAN - builder"' -p 'revision="5742e8f05cc68e46cd47fbef7e9352ff0dac9e7c"' -p 'swarming_dimensions=["cpu:x86-64", "pool:chromium.tests", "os:Ubuntu-22.04"]' -p 'isolated_name="bot_default"' -p 'test_name="mjsunit/wasm/grow-huge-memory"' -p 'timeout_sec=60' -p 'total_timeout_sec=120' -p 'variant="stress"' -p 'extra_args=["--isolates"]' Local flake reproduction on command line: tools/run-tests.py --outdir=SET_OUTDIR_HERE --variants=stress --random-seed-stress-count=1000000 --total-timeout-sec=120 --exit-after-n-failures=1 --isolates mjsunit/wasm/grow-huge-memory ``` The line starting with `bb add v8...` contains the buggy V8 commit hash (`revision="5742e8f05cc68e46cd47fbef7e9352ff0dac9e7c"`). The line starting with `Command: out/build/d8` contains the command to run the test case that triggers this bug. The line starting with `tools/run-tests.py` runs a script that runs the test case many times until TSan reports a bug (or timeout). You will need to replace some parts of the command (e.g. `out/build` or `SET_OUTDIR_HERE`) to your build directory. The part under `GN arguments` give the GN build arguments. Remove the `use_remoteexec` argument. ## Chromium I did not test Chromium for my experiments, so I don't have any summarized steps for building it. However, the [official documentation](https://chromium.googlesource.com/chromium/src/+/main/docs/linux/build_instructions.md) worked well for me. Remember to add the line `is_tsan=true` when running `gn gen`.