Moving Gadgetron testing to pytest
1. Add the basic pytest framework located in test\e2e
2. Create a fixture for fetching the files
* Download files a cache folder if not already there.
* Validate file checksum/hash before using (source?)
* Can disable caching with a pytest command line option. (--no-cache)
* If cache is disabled, deleted the download file
3. Create a fixture for reading in the config data
* Move to a yaml format for the config data
4. Build test objects
* Using the read in config data to parametrize the actuall test cases.
5. Run the tests
* Convert the dat files to h5 files, storing them into a temperary location.
6. Run comparisons
Default Caching location (--cache-path): current_folder/data
Default file host (--host): http://gadgetrondata.blob.core.windows.net/gadgetrontestdata/
Both should be setable via custom pytest command line options or env variables.
Temperary location used for the converted h5 files:
1. Use the temp_path fixture in pytest. This will provide a temp location under the base temp path (which can be changed via the command line) and automaticaly cleaned up after a point.
2. Use current_folder/temp and manage the files ourselves
Questions:
1. Should multiple tests be allowed in a single yaml file?
* This will allow for grouping of related tests
* Allow for multiple yaml files, each with a number of tests.
* Might make specifying the exact test to run a bit more convulted, but not to bad i.e.: "filename-testid"
* Can still have one test per a file if desired.
Example config:
```yaml=
cases:
- name: cmr_cine_binning_2slices
tags: slow
reconstruction:
data_file: cmr/CineBinning/meas_MID838_PK_rt_test_2slice_FID22519/meas_MID838_PK_rt_test_2slice_FID22519.dat
data_file_hash: 7f52ef8abeb44b7a23648b195307ccc8
measurement: 2
# [Optional] parameter_xml:
# [Optional] parameter_xsl:
# [Optional] data_conversion_flag:
configuration: CMR_2DT_RTCine_KspaceBinning.xml
validation:
images:
CMR_2DT_RTCine_KspaceBinning.xml/image_2:
reference_file_path: cmr/CineBinning/meas_MID838_PK_rt_test_2slice_FID22519/cmr_cine_binning_2slice_ref_20220817.mrd
reference_file_hash: ddd08991b7837dc37d7457cc74fb40cb
scale_comparison_threshold: 0.01
value_comparison_threshold: 0.01
- name: generic_cartesian_cine_denoise
tags: generic
reconstruction:
data_file: generic/meas_MID00051_FID110020_SAX_realtime_gt_TPAT4/meas_MID00051_FID110020_SAX_realtime_gt_TPAT4.dat
measurement: 2
configuration: Generic_Cartesian_Grappa_Cine_Denoise.xml
noise:
data_file: generic/meas_MID00051_FID110020_SAX_realtime_gt_TPAT4/meas_MID00051_FID110020_SAX_realtime_gt_TPAT4.dat
measurement: 1
configuration: default_measurement_dependencies.xml
validation:
images:
Generic_Cartesian_Grappa_Cine_Denoise.xml/image_1:
reference_file_path: denoise/meas_MID00051_FID110020_SAX_realtime_gt_TPAT4/denoise_ref_20220817.mrd
scale_comparison_threshold: 0.005
value_comparison_threshold: 0.001
```