---
# System prepended metadata

title: File mismatching in omni_clustering
tags: [omb]

---

# File mismatching in omni_clustering

###### tags: `omb`

### Problem
`Count files` from the koh dataset were matched with `meta data` files from the koh_ttc dataset. The current input file matching based on pattern matching is not robust for files with similar names. We need lineage based matching!

```python=1
omni_obj.inputs.input_files
> {'filter_expression_1d45b_koh': 
   {'filtered_rawcounts': 
    'data/filter_expression/filter_expression_koh_filtered_counts.mtx.gz',
    'meta_file': 
    'data/filter_expression/koh_ttc_meta.json'},
   ... }

```

:::info
#### Manual fix:
1.  Remove mismatched files and revert associated activities
3.  Manually force correct matching
4.  Check downstream activities
:::

### Expected hashes


| Dataset        | File      | correct | incorrect |
| -------------- | --------- | ------- | --------- |
| filter_express | logcounts | 38d89   | 59b80     |
| filter_express | counts    | 9ad5c   | 1d45b     |
| filter_m3drop  | logcounts | 2eae1   | edc0f     |
| filter_m3drop  | counts    | 7dfac   | cfc86     |




---


### Step 1: Remove mismatched files and revert related activities:

1. Check, if files are mismatched

*within console:*
```python=1
omni_obj.inputs.input_files
``` 
*within terminal:*
```bash!
renku workflow outputs | grep "koh__"
```
 
2. Get outputs from mismatched files:
```python=1
from omnibenchmark.utils.build_omni_object import get_omni_object_from_yaml
from omnibenchmark.renku_commands.general import renku_save
from omnibenchmark.management.run_commands import revert_run

## Load config
omni_obj = get_omni_object_from_yaml('src/config.yaml')

# Get mismatched outputs
out_mis = [out_map['output_files']['cluster_res'] for out_map in omni_obj.outputs.file_mapping if "koh__" in out_map['output_files']['cluster_res']]

#Revert run
revert_run(out_files = out_mis, dataset_name = omni_obj.name)
renku_save()
```

3. Check from terminal
```bash!
renku workflow outputs | grep "koh__"
``` 
:::warning
If this still shows files (e.g. with the correct hash) after the clean up, __please__ check the datasets these files belong to. Some of the methods have old output datasets left. __Only__ keep those activities if you are sure they represent valid runs! Otherwise manually remove them by replacing `OUTFILES` with the remaining files and `DATASET_NAME`:

```python=1
from renku.api import Activity, Plan
from omnibenchmark.renku_commands.workflows import renku_workflow_revert
from omnibenchmark.management.data_commands import unlink_dataset_files

# Get activities for those orphan outputs
activity_ids = [Activity.filter(outputs=out_fi) for out_fi in OUT_FILES]

# Make sure they are not linked to any dataset any longer
unlink_dataset_files(out_files=out_fis, dataset_name=DATASET_NAME, remove=True)

# Revert the activities
[renku_workflow_revert(activity_id=act_id[0]._activity.id, plan=False) for act_id in activity_ids]
```
:::


### Step 2: Manually force correct matching

This is very ugly and I wish there was an easier way to overwrite automatic inputs (one day ...). As we anyways plan to switch towards the use of lineages I would rather not try to wrangle to much with the current system? 

Add this to `run_workflow.py` after `omni_obj.update_object()`:

For logcounts:
```python=1

######################## fix for file matching of koh dataset #######################
#remove mismatched inputs
del omni_obj.outputs.inputs.input_files['filter_expression_59b80_koh']
del omni_obj.outputs.inputs.input_files['filter_m3drop_edc0f_koh']
omni_obj.outputs.file_mapping = []
omni_obj.outputs.inputs.default = 'filter_expression_38d89_koh'

#add correctly matched
omni_obj.outputs.inputs.input_files['filter_expression_38d89_koh']= {'filtered_logcounts': 'data/filter_expression/filter_expression_koh_filtered_logcounts.mtx.gz',
  'meta_file': 'data/filter_expression/koh_meta.json'}
omni_obj.outputs.inputs.input_files['filter_m3drop_2eae1_koh']= {'filtered_logcounts': 'data/filter_m3drop/filter_m3drop_koh_filtered_logcounts.mtx.gz',
  'meta_file': 'data/filter_m3drop/koh_meta.json'}

#update outputs
omni_obj.outputs.update_outputs()
######################################################################################

```
For counts:
```python=1
######################## fix for file matching of koh dataset #######################
#remove mismatched inputs
del omni_obj.outputs.inputs.input_files['filter_expression_1d45b_koh']
del omni_obj.outputs.inputs.input_files['filter_m3drop_cfc86_koh']
omni_obj.outputs.file_mapping = []
omni_obj.outputs.inputs.default = 'filter_expression_9ad5c_koh'

#add correctly matched
omni_obj.outputs.inputs.input_files['filter_expression_9ad5c_koh']= {'filtered_rawcounts': 'data/filter_expression/filter_expression_koh_filtered_counts.mtx.gz',
  'meta_file': 'data/filter_expression/koh_meta.json'}
omni_obj.outputs.inputs.input_files['filter_m3drop_7dfac_koh']= {'filtered_rawcounts': 'data/filter_m3drop/filter_m3drop_koh_filtered_counts.mtx.gz',
  'meta_file': 'data/filter_m3drop/koh_meta.json'}

#update outputs
omni_obj.outputs.update_outputs()
omni_obj.command.outputs = omni_obj.outputs
omni_obj.command.update_command()
######################################################################################
```

If possible check with:
```python=1
omni_obj.check_run()
```

### Step 3: Check downstream activities:
Theoretically things should clean up by itself in downstream projects, but I'm not sure that works. So check outputs from the koh dataset in downstream projects by:

```bash!
renku workflow inputs | grep "koh__"
```

Outputs from the following project are already used in metric workflows:
* monocle (both)
* flowsom 
* raceid
* cidr
* ascent
* r-seurat (Is this on purpose calculated on logcounts and raw counts????)


### Checklist:
- [x] [tscan](https://renkulab.io/projects/omnibenchmark/omni_clustering/tscan-clustering)
- [x] [ascend](https://renkulab.io/projects/omnibenchmark/omni_clustering/ascend-clustering)
- [x] [pca-hc](https://renkulab.io/gitlab/omnibenchmark/omni_clustering/pca-hc)
- [x] [cidr](https://renkulab.io/gitlab/omnibenchmark/omni_clustering/cidr-clustering)
- [x] [sc3-svm](https://renkulab.io/gitlab/omnibenchmark/omni_clustering/sc3-svm-clustering)
- [x] [flowsom](https://renkulab.io/gitlab/omnibenchmark/omni_clustering/flowsom-clustering)
- [x] [monocle](https://renkulab.io/gitlab/omnibenchmark/omni_clustering/monocle-clustering)
- [x] [sc3](https://renkulab.io/gitlab/omni_hackathon/omni_clustering/clustering-sc3)
- [x] [seurat](https://renkulab.io/gitlab/omni_hackathon/omni_clustering/r-seurat)
- [x] [rtsne-kmeans](https://renkulab.io/gitlab/omni_hackathon/omni_clustering/r-tsne-k-means-clustering)
- [x] [race_id2](https://renkulab.io/gitlab/omni_hackathon/omni_clustering/raceid2-clustering)
- [x] [pca_kmeans](https://renkulab.io/gitlab/omni_hackathon/omni_clustering/pcakmeans-clustering)

