# 3rd Year Project Meetings (December, January, February)

## Previous notes: [before December](https://hackmd.io/@W9LPPdtDTQCwdxGEjX9N8A/B1vfEHEuv)

## 7 December 2020

### Changes
- rotated images automatically when cropping a portrait photo
- reduced lambda finding iteration count to 52
- notif script exits with the same code as the child process
- avoid useless diff calculation when -nodiff specified
- save plots in .fig files
- save classifier data in model file (should be possible to load later)
- added "multi" script which runs multiple processes for *d.pgm where d is a digit in {0..9}
- experiments 29+ use "multi" and run faster
- added stego stats that will be evaluated in the linear classifier

### Calling conventions

<details> <summary> See Calling conventions </summary>

- Cropper
- [width] [height] [source dir] [destination dir] [delta row / -1 random] [delata col / -1 random] [regex match]
- ./crop 300 500 ../images/ cropped/ 50 -1 .*pgm
- HILL
- --input [source] --output [destination]
- python3 cost.py --input "../images/*" --output /costs
- CostDisplay
- [source dir] [destination dir] [regex match]
- ./costDisplay costs/ display/ .*cost
- Stego
- [source dir] [cost dir] [destination dir] [diff dir / -nodiff] [stats dir / -nostats] [payload bits] [regex match]
- ./stego images/ cost/ res/ diff/ stats/ 10000 .*pgm
- SRMQ1 feature extractor
- matlab -nodisplay < exit -r "extract [sourceDir] [destinationDir] [fileMatch]"
- matlab -nodisplay < exit -r "extract '../Pipe/Cropped/' './' '*.pgm'"
- echo "exit()" > Tools/exit
- Linear Classifier
- matlab -nodisplay < exit -r < exit "classify [sourceDir] [destinationDir] [statsDir] [resultDir] [fileMatch]"
- matlab -nodisplay < exit -r "classify('./CoverFeatures/', './StegoFeatures/', './Stats/', './Result/', '*.fea')"
- Statistics Expander
- ./statExpand [source dir ] [cost source dir ] [destination dir] [keep old 0/1] [regex match]
- ./statExpand ./from/ ./fromCost/ ./to/ 0 .*.stats

</details>

### Current pipeline

graph TD
Cover(Cover .pgm) -- Crop --> Cropped(Cropped .pgm)
Cropped -- HILL --> Costs(Costs .cost)
Costs -- CostDisplay --> Display(CostDisplay .pgm)
Costs --> Embedder
Cropped --> Embedder(Embedder)
Embedder --> Stats(Stats .stats)
Stats -- Opt. Expander --> NewStats(Expanded Stats .stats)
Embedder --> Stego(Stego .pgm)
Payload(Payload bits) --> Embedder
Stego --> Diff(Difference .pgm)
Cropped --> Diff
Cropped -- SRMQ1 --> CFea(Cover Features .fea)
Stego -- SRMQ1 --> SFea(Stego Features .fea)
CFea --> Lin(Linear Classifier)
SFea --> Lin
Stats --> Lin
Lin --> ResultError(result)
Lin --> Result(result.fig)
Lin --> Tolerance(tolerance.fig)
Lin --> Model(model)
Lin --> Corr(correlationX.fig)

### Suggested size & payload

| Size | Width | Height | O(1) | O(sqrt(n)) | O(sqrt(n)log(n)) | O(n) |
| - | - | - | - | - | - | - |
| 0.5 | 864 | 576 | 100000 | 40000 | 40000 | 20000 |
| 1.0 | 1248 | 832 | 100000 | 57000 | 61000 | 41000 |
| 1.5 | 1536 | 1024 | 100000 | 71000 | 77000 | 63000 |
| 2.0 | 1752 | 1168 | 100000 | 81000 | 89000 | 82000 |
| 2.5 | 1968 | 1312 | 100000 | 91000 | 102000 | 103000 |
| 3.0 | 2160 | 1440 | 100000 | 100000 | 113000 | 125000 |
| 3.5 | 2328 | 1552 | 100000 | 107000 | 124000 | 145000 |
| 4.0 | 2496 | 1664 | 100000 | 115000 | 134000 | 166000 |
| 4.5 | 2640 | 1760 | 100000 | 122000 | 143000 | 186000 |
| 5.0 | 2784 | 1856 | 100000 | 128000 | 151000 | 207000 |
| Est | | | 10-45% | 25-35% | 25-30% | 45%-10% |

### Useful commands
- { time make test > results; } 2> time &
- notif "( time make test > results ) 2> time" &
- alias: run_pipe
- { time make test_classify >> extra; } 2>> time_extra &
- jobs, kill %1, bg %1, fg %1
- notif [command] - will send a Telegram notification when a process finishes
- multi [command with @] - run multiple commands in background with @ replaced with {0..9}
- new_exp [name] [image pattern] [image pattern]? - create a new experiment placeholder with images that follow those patterns
- pipe [images] - runs the experiments described in the bash script
- notif "pipe '../new_images/*.pgm'" > out 2> err &
- out: init + results + wait error
- err: progression info

### Experiments

<details> <summary> See Experiments </summary>

#### Setup

| Id | Width | Height | Payload | Images |
| - | - | - | - | - |
| 19 | 512 | 512 | 14000 bits | 000*.pgm (99) |
| 20 | 512 | 512 | 14000 bits | 000*.pgm (99) |
| 21 | 512 | 512 | 14000 bits | 000*.pgm (99) |
| 22 | 512 | 512 | 14000 bits | 000*.pgm (99) |
| 23 | 512 | 512 | 14000 bits | 0[0/3]*.pgm (1999) |
| 24 | 1024 | 512 | 20000 bits | 0[0/3]*.pgm (1999) |
| 25 | 1024 | 1024 | 28000 bits | 0[0/3]*.pgm (1999) |
| 26 | 2048 | 1024 | 40000 bits | 0[0/3]*.pgm (1999) |
| 27 | 1024 | 512 | 20000 bits | 0[0/3]*.pgm (1999) |
| 28 | 2048 | 1024 | 40000 bits | 0[0/3]*.pgm (1999) |
|
| 29 | 512 | 512 | 14000 bits | all (10k) |
| 30 | 1024 | 512 | 20000 bits | all (10k) |
| 31 | 1024 | 1024 | 28000 bits | all (10k) |
| 32 | 2048 | 1024 | 40000 bits | all (10k) |

#### Time

| Id | Crop | Cost | Cost Display | Stego | Features | Classifier | Total |
| - | - | - | - | - | - | - | - |
| 19 | 4s | 33s | 0s | 37s | 2m x2 | 10s | 3m38s |
| 20 | 4s | 33s | 0s | 37s | 2m x2 | 10s | 3m38s |
| 21 | 4s | 33s | 0s | 37s | 2m x2 | 10s | 3m38s |
| 22 | 4s | 33s | 0s | 37s | 2m x2 | 10s | 3m38s |
| 23 | 1m35s | 11m27s | 2s | 13m9s | 41m x2 | 5m51s | 75m |
| 24 | 1m37s | 21m55s | 3s | 26m30s | 1h15m x2 | 5m35 | 132m |
| 25 | | | | | | | 255m |
| 26 | | | | | | | 506m |
| 27 | | | | | | | 135m |
| 28 | | | | | | | 485m |
|
| 29 | | | | | | | 259m |
| 30 | | | | | | | 311m |
| 31 | | | | | | | 394m |
| 32 | | | | | | | 528m |

#### Results

| Id | Error | Obs |
| - | - | - |
| 19 | 0.0859 | random crop |
| 20 | 0.0707 | .. |
| 21 | 0.0707 | .. |
| 22 | 0.0556 | .. |
| 23 | 0.3319 | 50/50 crop |
| 24 | 0.2319 | random crop |
| 25 | 0.1243 | 50/50 crop |
| 26 | 0.3942 | random crop |
| 27 | 0.3009 | 50/50 crop |
| 28 | 0.0978 | 50/50 crop |
|
| 29 | 0.3533 | - |
| 30 | 0.3678 | - |
| 31 | 0.4107 | - |
| 32 | 0.4128 | - |

</details>

### Pipe experiments

<details> <summary> See Pipe experiments </summary>

#### p1: 1k images, 2h10m, 14.5MB, 1h30m/10GB
- It has the same groups as **p2** but it run only on 1000 images.
- Unfortunately, the results are uncorrelated because of the small number of training images.

#### p2: 10k images, 17h38m, 14.5MB, 1h10m/10GB

Adding group 256KB (512 x 512) with stego 14000bits, 25000bits, 8000bits, 17000bits, 
Adding group 512KB (1024 x 512) with stego 20000bits, 25000bits, 16000bits, 
Adding group 1MB (1024 x 1024) with stego 28000bits, 25000bits, 32000bits, 34000bits, 
Adding group 2MB (2048 x 1024) with stego 40000bits, 25000bits, 64000bits, 50000bits, 
Image source: ../new_images/image*

| Group | O(1) | O(sqrt(n)) | O(sqrt(n)log(n)) | O(n) |
| - | - | - | - | - |
| 256KB | 25000 | 14000 | 17000 | 8000 |
| | 0.25 | 0.35 | 0.31 | 0.43 |
| 512KB | 25000 | 20000 | 25000 | 16000 |
| | 0.35 | 0.36 | 0.35 | 0.40 |
| 1MB | 25000 | 28000 | 34000 | 32000 |
| | 0.39 | 0.37 | 0.37 | 0.38 |
| 2MB | 25000 | 40000 | 50000 | 64000 |
| | 0.44 | 0.41 | 0.37 | 0.36 |
| | up | stable-up? | stable-up? | down |

#### p3: 10k images, 15h30m, 11.75MB, 1h20m/10GB

Adding group 256KB (512 x 512) with stego 50000bits, 30000bits, 15000bits,
Adding group 512KB (1024 x 512) with stego 50000bits, 42500bits, 45000bits, 30000bits,
Adding group 1MB (1024 x 1024) with stego 50000bits, 60000bits, 67000bits,
Adding group 2MB (2048 x 1024) with stego 83000bits, 99000bits, 120000bits,
Image source: ../new_images/*.pgm

| Group | O(1) | O(sqrt(n)) | O(sqrt(n)log(n)) | O(n) |
| - | - | - | - | - |
| 256KB | 50000 | 30000 | 30000 | 15000 |
| | 0.09 | 0.20 | 0.20 | 0.33 |
| 512KB | 50000 | 42500 | 45000 | 30000 |
| | 0.18 | 0.22 | 0.21 | 0.31 |
| 1MB | 50000 | 60000 | 67000 | 60000 |
| | 0.29 | 0.28 | 0.25 | 0.28 |
| 2MB | 50000 | 83000 | 99000 | 120000 |
| | 0.44 | 0.30 | 0.26 | 0.21 |
| | up | stable-up | stable-up | down |

#### p4: 10k images, 27h26m, 25MB, 1h6m/10GB
- p5: 10k images, 40h30, 39MB, 1h2m/10GB
- p6: 10k images, 5h30m, 1.5MB, 3h40m/10GB (run with p7)
- p7: 10k images, 40h, 38MB, 1h3m/10GB

Adding group 0.5MB (864 x 576) with stego 100000bits, 40000bits, 20000bits, 
Adding group 1.0MB (1248 x 832) with stego 100000bits, 57000bits, 61000bits, 41000bits, 
Adding group 1.5MB (1536 x 1024) with stego 100000bits, 71000bits, 77000bits, 63000bits, 
Adding group 2.0MB (1752 x 1168) with stego 100000bits, 81000bits, 89000bits, 
Adding group 2.5MB (1968 x 1312) with stego 100000bits, 91000bits, 103000bits, 
Image source: ../new_images/*.pgm

Adding group 3.0MB (2160 x 1440) with stego 100000bits, 113000bits, 125000bits, 
Adding group 3.5MB (2328 x 1552) with stego 100000bits, 107000bits, 124000bits, 145000bits, 
Adding group 4.0MB (2496 x 1664) with stego 100000bits, 115000bits, 134000bits, 166000bits, 
Image source: ../new_images/*.pgm

Adding group 0.5MB (864 x 576) with stego 100000bits, 40000bits, 20000bits, 
Image source: ../new_images/*.pgm

Adding group 4.5MB (2640 x 1760) with stego 100000bits, 122000bits, 143000bits, 186000bits, 
Adding group 5.0MB (2784 x 1856) with stego 100000bits, 128000bits, 151000bits, 207000bits, 
Image source: ../new_images/*.pgm

| Group | Width | Height | O(1) | O(sqrt(n)) | O(sqrt(n)log(n)) | O(n) |
| - | - | - | - | - | - | - |
| 0.5 | 864 | 576 | 100000 | 40000 | 40000 | 20000 |
| | | | 0.06 | 0.23 | 0.23 |0.36 |
| 1.0 | 1248 | 832 | 100000 | 57000 | 61000 | 41000 |
| | | | 0.15 | 0.25 | 0.28 | 0.34 |
| 1.5 | 1536 | 1024 | 100000 | 71000 | 77000 | 63000 |
| | | | 0.20 | 0.28 | 0.26 | 0.31 |
| 2.0 | 1752 | 1168 | 100000 | 81000 | 89000 | 82000 |
| | | | 0.26 | 0.31 | 0.29 | 0.30 |
| 2.5 | 1968 | 1312 | 100000 | 91000 | 102000 | 103000 |
| | | | 0.30 | 0.28 | 0.30 | 0.29 |
| 3.0 | 2160 | 1440 | 100000 | 100000 | 113000 | 125000 |
| | | | 0.34 | 0.34 | 0.31 | 0.28 |
| 3.5 | 2328 | 1552 | 100000 | 107000 | 124000 | 145000 |
| | | | 0.33 | 0.33 | 0.27 | 0.23 |
| 4.0 | 2496 | 1664 | 100000 | 115000 | 134000 | 166000 |
| | | | 0.36 | 0.32 | 0.29 | 0.28 |
| 4.5 | 2640 | 1760 | 100000 | 122000 | 143000 | 186000 |
| | | | 0.37 | 0.33 | 0.29 | 0.27 |
| 5.0 | 2784 | 1856 | 100000 | 128000 | 151000 | 207000 |
| | | | 0.39 | 0.34 | 0.30 | 0.26 |

</details>

### See all tables at [result tables](https://hackmd.io/@W9LPPdtDTQCwdxGEjX9N8A/B1rLaiIg_).

## Towards the second part of the project

We started with an important assumption, i.e. a good way to measure detectability is to compute the total cost $\pi^Tc$. With that assumption, we computed $\pi_i$ which where further used to simulate stego embedding in some cover images with a fixed payload.

$$ min\ \pi^Tc,\ subject\ to\ \sum_{i}H(\pi_i) = m $$

After running multiple tests with different sizes and payloads, we reached the conclusion that the error of the linear classifier is maintained when $payload=O(\sqrt{size})$. However, all images were embedded with the same number of bits even though some of them have higher or lower costs. I will further inspect the relation between the cost (assumed initially to be $\pi^Tc$) and the detectability of each specific image.

Detectability will be measured as the difference in the classifier values assigned to the stego and cover features. More precisely, if the output of the linear classifier is $w$, then detectabily will be measured as $w^T(s_i-c_i)$, where $s_i$ and $c_i$ are the stego and cover features.

Apart from the initial assumption that the cost is $\pi^Tc$, my supervisor suggested that $\sum_i{\pi_i^2c_i}$ would be a better indicator of the final detectability. To check these claims, I see multiple possible approaches:

1. (Ker's suggestion) Solve a different optimization problem that maximizes the total entropy while keeping the cost constant. Then, see if the detectability remains constant among all images in the classifier.

$$ max\ \sum_{i}H(\pi_i) = m,\ subject\ to\ \sum_i{\pi_i^2c_i}=d $$

2. Compute for each image in the current pipe its associated cost $\sum_i{\pi_i^2c_i}$ and compare it to the detectability $w^T(s_i-c_i)$. We want to find a correlation between these two if possible. Maybe I could compute the correlation of the two random variables?
3. I remember you also suggested a correlation between detectability and $\sum_i{\frac{1}{c_i}}$. I suppose that requires a fixed payload size.

Before I start to work on this part, I would be grateful if you could answer some of my question:

1. Did I understand it correctly that we are currently challenging the initial assumption of a good measurement of detectability? Then haven't we used it in the first place? Maybe because it is harder to solve?

**A**: Yes. It might be the case that the real probability should be $\pi_ic_i$ leading to the average cost $\sum_i\pi_i^2c_i$.

2. Are methods (1) and (2) above attempting to prove the same thing? Is there are difference that I should consider? In method (1) we train a classifier on stego images of different payloads but the same cost. In method (2) we train stego images with the same payload but different costs. Is that going to influence the results?

**A**: Those are different things but should lead to similar results. Method (1) is encoding the most value givea a detectability limit while method (2) assumes some detectability limit and encodes a fixed payload.

3. Which method do you think is more appropriate? I think that (2) is easier to integrate in the current pipeline but I am happy to write another pipe for (1).

**A**: Both should lead to similar results. Let's try both.

4. Is method (3) proving the same claim? Did I understand it correctly? What is the equivalence between (1) and (3) or (2) and (3). I see that ADK72B proves this correlation under some conditions.

**A**: In fact, $M = \sum_i H(\pi_i) \approx \sum_i\frac{1}{c_i}$ is the approximative suggested payload size for an image. Doesn't that contradict the square root law?

5. Do we expect these relations to hold only under some specific circumstances (e.g. payload = sqrt(size)) or to be universally true?

**A**: I think they should be universally true.

6. If you suggest to continue with method (1), what $d$ should I use? What size and payload should I choose to choose the most useful $d$.

**A**: Use your current data with $\sim 20\%$ error and find a suitable $d$.

7. In the first part we prove that $payload$ should be $O(\sqrt{size})$ in order to maintain the error. In the second part we aim to find the constant that satisfies this critical relationship. Is that what we are trying to do?

**A**: Yes, approx.

8. Based on my questions, what do you think that I might be missing or not understanding? I want to make sure that I completely understand what we are trying to prove here.

**A**: Mr Ker explained the relations once again during our last meeting.

9. I did some more experiments below. Please answer my **Unfortunately** questions.

**A**: Try using MaxSRM or adjusting the detectability measure.

10. What do you think about the results of p4-p7? Are they appropriate for the first part of the project?

**A**: Yes. Repeat the experiment with $1.5\times$ payload.

These are all the questions and I hope to be able to start writing the code for the second part after reading your answers. Enjoy your Winter Holiadays with your family! I am looking forward to hearing back from you!

## Evaluating different detectability estimations

I implemented the necessary tools to evaluate more possible estimations based on method (2). More precisely, when computing $\lambda$ and the stego image, I also generated a $.stats$ file that contains multiple possible estimations $\sum_i{f(\pi_i, c_i)}$ where $f$ is one of the following functions:
- $f(\pi_i, c_i) = \pi_i \times c_i$ (corr 0.90)
- $f(\pi_i, c_i) = \pi_i^2 \times c_i$ (corr 0.23)
- $f(\pi_i, c_i) = \pi_i \times c_i^2$ (corr 0.85)
- $f(\pi_i, c_i) = \pi_i$ (corr -0.14)
- $f(\pi_i, c_i) = c_i$ (corr 0.84)
- $f(\pi_i, c_i) = \frac{1}{\pi_i}$ (corr -0.01)
- $f(\pi_i, c_i) = \frac{1}{c_i}$ (corr -0.70)

I created a scatter plot with $x = w^T(s_i-c_i)$ and $y=\sum_i{f(\pi_i, c_i)}$ and computed the correlation between these two "random variables". The results were similar for different sizes and payloads (at least on my current data). You can find the correlation above for 40k bits of payload hidden in 0.5MB cover images. The plots can be found below.

**Unfortunately**, $\pi_i^2 \times c_i$ did not provide the expected correlation, the initial approach $\pi_i \times c_i$ leading to the best results in most cases. Maybe I misunderstood what you wanted to say? Maybe the fact that we optimise $\pi_i \times c_i$ for a fixed payload hides the desired result?

<details> <summary> See Correlation images </summary>

</details>

## 6 January 2021

### Changes
- stego embedder generates .probs file
- added maxSRM to the feature extractor

### Calling conventions

<details> <summary> See Calling conventions </summary>

- Cropper
- [width] [height] [source dir] [destination dir] [delta row / -1 random] [delata col / -1 random] [regex match]
- ./crop 300 500 ../images/ cropped/ 50 -1 .*pgm
- HILL
- --input [source] --output [destination]
- python3 cost.py --input "../images/*" --output /costs
- CostDisplay
- [source dir] [destination dir] [regex match]
- ./costDisplay costs/ display/ .*cost
- Stego
- [source dir] [cost dir] [destination dir] [diff dir / -nodiff] [stats dir / -nostats] [probs dir / -noprobs] [payload bits] [regex match]
- ./stego images/ cost/ res/ diff/ stats/ probs/ 10000 .*pgm
- SRMQ1 feature extractor
- matlab -nodisplay < exit -r "extract [sourceDir] [probsDir / -noprobs] [destinationDir] [fileMatch]"
- matlab -nodisplay < exit -r "extract '../Pipe/Cropped/' '../Pipe/Probs/' './' '*.pgm'"
- echo "exit()" > Tools/exit
- Linear Classifier
- matlab -nodisplay < exit -r < exit "classify [sourceDir] [destinationDir] [statsDir] [resultDir] [fileMatch]"
- matlab -nodisplay < exit -r "classify('./CoverFeatures/', './StegoFeatures/', './Stats/', './Result/', '*.fea')"
- Statistics Expander
- ./statExpand [source dir ] [cost source dir ] [destination dir] [keep old 0/1] [regex match]
- ./statExpand ./from/ ./fromCost/ ./to/ 0 .*.stats

</details>

### Current pipeline

graph TD
Cover(Cover .pgm) -- Crop --> Cropped(Cropped .pgm)
Cropped -- HILL --> Costs(Costs .cost)
Costs -- CostDisplay --> Display(CostDisplay .pgm)
Costs --> Embedder
Cropped --> Embedder(Embedder)
Embedder --> Stats(Stats .stats)
Embedder --> Probs(Probs .probs)
Stats -- Opt. Expander --> NewStats(Expanded Stats .stats)
Embedder --> Stego(Stego .pgm)
Payload(Payload bits) --> Embedder
Stego --> Diff(Difference .pgm)
Cropped --> Diff
Probs --> FeaExtr(Feature Extractor)
Cropped --> FeaExtr
Stego --> FeaExtr
FeaExtr -- SRMQ1/maxSRMQ1 --> CFea(Cover Features .fea)
FeaExtr -- SRMQ1/maxSRMQ1 --> SFea(Stego Features .fea)
CFea --> Lin(Linear Classifier)
SFea --> Lin
Stats --> Lin
Lin --> ResultError(result)
Lin --> Result(result.fig)
Lin --> Tolerance(tolerance.fig)
Lin --> Model(model)
Lin --> Corr(correlationX.fig)

### Experiments

#### Setup

| Id | Width | Height | Payload | Images |
| - | - | - | - | - |
| 33 | 512 | 512 | 14000 bits | all (10k) |
| 34 | 1024 | 512 | 20000 bits | all (10k) |
| 35 | 1024 | 1024 | 28000 bits | all (10k) |
| 36 | 2048 | 1024 | 40000 bits | all (10k) |

#### Time

| Id | Crop | Cost | Cost Display | Stego | Features | Classifier | Total |
| - | - | - | - | - | - | - | - |
| 33 | | | | | | | 5h |
| 34 | | | | | | | 6h |
| 35 | | | | | | | 13h |
| 36 | | | | | | | 17h |

#### Results

| Id | Error | Obs |
| - | - | - |
| 33 | 0.31 | maxSRMQ1 |
| 34 | 0.30 | maxSRMQ1 |
| 35 | 0.32 | maxSRMQ1 |
| 36 | 0.34 | maxSRMQ1 |

#### Correlations
| Exp | $\pi_i \times c_i$ | $\pi_i^2 \times c_i$| $\pi_i \times c_i^2$| $\pi_i$| $c_i$| $\frac{1}{\pi_i}$| $\frac{1}{c_i}$ |
| - | - | - | - | - | - | - | - |
| 33 | 0.29 | 0.24 | 0.28 | 0.23 | 0.27 | 0 | -0.27 |
| 34 | 0.44 | 0.68 | 0.38 | 0.49 | 0.51 | 0 | -0.43 |
| 35 | 0.28 | 0.65 | 0.21 | 0.53 | 0.39 | 0 | -0.33 |
| 36 | 0.22 | 0.64 | 0.15 | 0.54 | 0.35 | 0 | -0.29 |

<details> <summary> See correlation for e33 </summary>


</details>

<details> <summary> See correlation for e34 </summary>


</details>

### Questions
1. The error improved after replacing SRMQ1 with maxSRMQ1 by the correlation decreased dramatically. Is that expected?

**A**: Sure.

2. However, e34 was more correlated to $\pi_i^2\times c_i$ than the other functions. Is that the effect we want?

**A**: We should see the results of SRMQ1 and maxSRMQ1 side-by-side.

3. I suppose that we assume here that a possible attacker knows the estimated payload size in the image so that he can generate the correct probabilities. Is that what we want?

**A**: Yes. Make sure to include these assumptions in the final project.

4. I used the same probability map when extracting the cover and stego features (computed on the cover image). Would that have a significant impact?

**A**: That should be fine. The costs don't change significantly.

5. I currently have two pipelines. The small one is running a single experiment. The big one is running multiple experiments in a tree structure (multiple sizes, each size with multiple payloads). The maxSRMQ1 is now running only in the small pipe. Should I make it compatible to the big pipe, too? If we need only a few experiments with their correlations then the small pipe will do the job. However, if we want to analyze different sizes and payloads then the big one will be more appropriate.

**A**: Add it to the big pipeline so that we can get more data and also enhance the first part.

5. It seems that there is also a strong correlation between detectability and $\sum_i\pi_i$. However, this correlation was almost 0 when I used SRMQ1. Why?

**A**: I don't remember the answer here :(.

## 13 January 2021

### Changes
- Deletion toolfor old experiments
- Added multiple correlation types (Pearson, Kendall, Spearman)
- Added another detectability measure (normal, normalised)
- Added inverted sums estimates
- Added a nice table generator from results
- Added maxSRMQ1 to the big pipe
- Added pseudo-random stego embedder based on file name

### Current pipeline

graph TD
Cover(Cover .pgm) -- Crop --> Cropped(Cropped .pgm)
Cropped -- HILL --> Costs(Costs .cost)
Costs -- CostDisplay --> Display(CostDisplay .pgm)
Costs --> Embedder
Cropped --> Embedder(Embedder)
Embedder --> Stats(Stats .stats)
Embedder --> Probs(Probs .probs)
Stats -- Opt. Expander --> NewStats(Expanded Stats .stats)
Embedder --> Stego(Stego .pgm)
Payload(Payload bits) --> Embedder
Stego --> Diff(Difference .pgm)
Cropped --> Diff
Probs --> FeaExtr(Feature Extractor)
Cropped --> FeaExtr
Stego --> FeaExtr
FeaExtr -- SRMQ1/maxSRMQ1 --> CFea(Cover Features .fea)
FeaExtr -- SRMQ1/maxSRMQ1 --> SFea(Stego Features .fea)
CFea --> Lin(Linear Classifier)
SFea --> Lin
Stats --> Lin
Lin --> ResultError(result)
Lin --> Result(result.fig)
Lin --> Tolerance(tolerance.fig)
Lin --> Model(model)
Lin --> Corr(correlationX.fig)
ResultError -- Table generator --> Table(result.table) 

### Folder Structure

.
├── Images (initial images)
├── 0.5MB/
│   └── ...
└── 1.0MB/
    ├── crop (cropped cover images)
    ├── cost (HILL costs)
    ├── features (SRMQ1 cover features)
    └── stego/
        ├── 10000bits/
        │   └── ...
        └── 14000bits/
            ├── images (stego images)
            ├── probs (probability map)
            ├── features (stego features)
            ├── cover_features (maxSRMQ1 cover features)
            └── classifier (results)

### Useful commands
- new_exp [name] [image pattern] [image pattern]? - create a new experiment placeholder with images that follow those patterns
- { time make test > results; } 2> time &
- notif "( time make test > results ) 2> time" &
- alias: run_pipe
- { time make test_classify >> extra; } 2>> time_extra &
- new_pipe [name] - create a new pipe experiment
- pipe [images] - runs the experiments described in the bash script
- notif "pipe '../new_images/*.pgm'" > out 2> err &
- alias: pipe_some = notif "pipe '../some_images/*.pgm'" > out 2> err &
- alias: pipe_here = notif "./pipe_here '../new_images/*.pgm'" > out 2> err &
- out: init + results + wait error
- err: progression info
- jobs, kill %1, bg %1, fg %1
- notif [command] - will send a Telegram notification when a process finishes
- multi [command with @] - run multiple commands in background with @ replaced with {0..9}
- rm_stego [see script] -- removes unncessary files to free space
- folder structure gen https://tree.nathanfriend.io

### Pipe experiments

#### p8: 10k images, 22h25m, 5.5MB, 4h/10GB
- p9: 10k images, 86h, 14MB, 6.14h/10GB
- BOTH EXPERIMENTS ENCOUNTERED THE "Your MATLAB session has timed out. All license keys have been returned." ERROR. MUST BE REPEATED

Using maxSRMQ? 1
Adding group 0.5MB (864 x 576) with stego 100000bits, 40000bits, 20000bits, 
Adding group 1.0MB (1248 x 832) with stego 100000bits, 57000bits, 61000bits, 41000bits, 
Image source: ../new_images/*.pgm

Using maxSRMQ? 1
Adding group 1.5MB (1536 x 1024) with stego 100000bits, 71000bits, 77000bits, 63000bits, 
Adding group 2.0MB (1752 x 1168) with stego 100000bits, 81000bits, 89000bits, 82000bits, 
Image source: ../new_images/*.pgm

| Group | Width | Height | O(1) | O(sqrt(n)) | O(sqrt(n)log(n)) | O(n) |
| - | - | - | - | - | - | - |
| 0.5 | 864 | 576 | 100000 | 40000 | 40000 | 20000 |
| | | | 0.06 | 0.21 | 0.21 | 0.19 |
| 1.0 | 1248 | 832 | 100000 | 57000 | 61000 | 41000 |
| | | | 0.12 | 0.21 | 0.20 | 0.27 |
| 1.5 | 1536 | 1024 | 100000 | 71000 | 77000 | 63000 |
| | | | 0.17 | 0.20 | 0.21 | 0.25 |
| 2.0 | 1752 | 1168 | 100000 | 81000 | 89000 | 82000 |
| | | | 0.21 | 0.24 | 0.23 | 0.24 |

### Questions

1. I implemented the maxSRMQ1 extractor in the big pipe but it seems to take a very long time to process each imag
e (~75s). Unfortunately, MATLAB has a timeout which stopped my feature extractions after ~20 hours for some small experiments. The error is "Your MATLAB session has timed out. All license keys have been returned.". There could be a few solutions. Which one seems better? Maybe both?
- Remove the timeout but the execution will still take a very long time
- maxSRM computes many features (not only with q=1) and I discard them after that. I am not sure but it might be possible to stop it from computing the other features with higher q.

**A**: You should compile your matlab with "mcc" and try to avoid computing q > 1.

2. However, previous experiments 33-36 had smaller times of extraction per image (9s for 34, 43s for 36). I suppose that's because the big pipe runs multiple processes in parallel and puts more load on each cpu. Is that a reasonable explanation or is it more probable that I have a bug?

**A**: You are running a lot of instances. Maybe serialize them?

3. It seems that the bug appeared also in p8 so the results displayed above are computed on ~30% of data.

**A**: Repeat it.

## 20 January 2021

### Changes
- compiled all matlab files with mcc (& changed both pipes)
- added parallel/sequential options for diff sizes and payloads (current sequential sizes, parallel payload)
- removed the computation of features with q > 1 I had to use some lib not available in matlab2014. The maxSRM script works only when called from my modified ./run_extract.sh. (/usr/local/matlab-r2020b-installer/bin/glnxa64/libstdc++.so.6.0.25)

### Pipe experiments

- p10-p12 moved to the results table below.

#### See all tables at [result tables](https://hackmd.io/@W9LPPdtDTQCwdxGEjX9N8A/B1rLaiIg_).

#### See correlation tables [here](https://hackmd.io/@W9LPPdtDTQCwdxGEjX9N8A/SyFkCL31O).

### Observations
1. The square root law holds also when using the maxSRMQ1 feature extractor.
2. $\sum_i\pi_i^2c_i$ is, in fact, a better detectability predictor than $\sum_i\pi_ic_i$, especially for large images. The difference becomes larger for bigger images.
3. Correlations are not influenced by the payload size.
4. Kendall and Spearman correlations removed the effect of the outliers.
5. An even better estimate seems to be the sum of costs ($\sum_ic_i$). I suspect that happens because we used a fixed payload. However, I did not expect that to happen.
6. The second detectability measure leads to a lot of outliers. There is some correlatin when using Kendall or Spearman.

### Questions
1. What do you think about the observations above?
2. Why do you think that the sum of costs is a better predictor?
3. I compute the second detectability measure as (first detec measure) / abs(distance to chosen separation line). That may lead to some really bad outliers that make the resulting graphs useless. Should I do something about them?

## 27 January 2021

The server was down after rebooting so I paused coding and running experiments and started the initial draft.

### Changes
- Moved all results to [results tables](https://hackmd.io/@W9LPPdtDTQCwdxGEjX9N8A/B1rLaiIg_) and [correlation tables](https://hackmd.io/@W9LPPdtDTQCwdxGEjX9N8A/SyFkCL31O)
- added support for random payload embeddding with -1, -2, ... (randomly chosen 1%-9%)

### Calling Conventions

<details> <summary> See Calling conventions </summary>

- Cropper
- [width] [height] [source dir] [destination dir] [delta row / -1 random] [delata col / -1 random] [regex match]
- ./crop 300 500 ../images/ cropped/ 50 -1 .*pgm
- HILL
- --input [source] --output [destination]
- python3 cost.py --input "../images/*" --output /costs
- CostDisplay
- [source dir] [destination dir] [regex match]
- ./costDisplay costs/ display/ .*cost
- Stego
- [source dir] [cost dir] [destination dir] [diff dir / -nodiff] [stats dir / -nostats] [probs dir / -noprobs] [payload bits / <0 if random] [regex match]
- ./stego images/ cost/ res/ diff/ stats/ probs/ 10000 .*pgm
- SRMQ1 feature extractor
- matlab -nodisplay < exit -r "extract [sourceDir] [probsDir / -noprobs] [destinationDir] [fileMatch]"
- matlab -nodisplay < exit -r "extract '../Pipe/Cropped/' '../Pipe/Probs/' './' '*.pgm'"
- echo "exit()" > Tools/exit
- Linear Classifier
- matlab -nodisplay < exit -r < exit "classify [sourceDir] [destinationDir] [statsDir] [resultDir] [fileMatch]"
- matlab -nodisplay < exit -r "classify('./CoverFeatures/', './StegoFeatures/', './Stats/', './Result/', '*.fea')"
- Statistics Expander
- ./statExpand [source dir ] [cost source dir ] [destination dir] [keep old 0/1] [regex match]
- ./statExpand ./from/ ./fromCost/ ./to/ 0 .*.stats

</details>

### Correlations for random payload on fixed size images

/ssd2/tatomir/rand/0.5MB/stego/-1bits/classifier
| Det | Corr | | | | | | | | | |
| - | - | - | - | - | - | - | - | - | - | - |
| 0 | Pearson | 0.57 | 0.56 | 0.55 | 0.40 | 0.45 | -0.00 | 0.25 | -0.41 | 0.54 |
| | Kendall | 0.54 | 0.56 | 0.56 | 0.33 | 0.49 | -0.20 | 0.20 | -0.49 | 0.49 |
| | Spearman | 0.72 | 0.75 | 0.74 | 0.47 | 0.66 | -0.29 | 0.29 | -0.67 | 0.67 |
| 1 | Pearson | 0.02 | 0.01 | 0.02 | 0.02 | 0.01 | -0.00 | 0.00 | -0.01 | 0.01 |
| | Kendall | 0.33 | 0.34 | 0.32 | 0.25 | 0.23 | -0.14 | 0.14 | -0.23 | 0.23 |
| | Spearman | 0.48 | 0.49 | 0.46 | 0.36 | 0.34 | -0.21 | 0.21 | -0.34 | 0.34 |

/ssd2/tatomir/rand/1.0MB/stego/-1bits/classifier
| Det | Corr | | | | | | | | | |
| - | - | - | - | - | - | - | - | - | - | - |
| 0 | Pearson | 0.67 | 0.73 | 0.63 | 0.46 | 0.60 | -0.01 | 0.32 | -0.49 | 0.61 |
| | Kendall | 0.56 | 0.58 | 0.59 | 0.33 | 0.52 | -0.25 | 0.25 | -0.51 | 0.51 |
| | Spearman | 0.75 | 0.78 | 0.77 | 0.48 | 0.70 | -0.36 | 0.36 | -0.70 | 0.70 |
| 1 | Pearson | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | -0.00 | 0.01 | -0.01 | 0.01 |
| | Kendall | 0.31 | 0.32 | 0.30 | 0.22 | 0.24 | -0.14 | 0.14 | -0.24 | 0.24 |
| | Spearman | 0.44 | 0.46 | 0.44 | 0.32 | 0.35 | -0.22 | 0.21 | -0.35 | 0.35 |

### Project draft
https://www.overleaf.com/read/zvfzbdmjhdsf

### Observations with random payload
1. $\pi_i^2c_i$ seems to have a stronger correlation on each experiment. However, the margin is quite small.
2. $\pi_ic_i^2$ seems to provide a good lower bound on the detectability.
3. $c_i$ lost its advantage after adding random payload.

### Questions
1. Do you have any specific project structure? Any example?

**A**: Abstract, Intro & Conclusions last. Each chapter with sections. Write one statement for each paragraph then write all paragraphs. Use detex for word count. Move "Three approaches" & "Reproducing" to Ch 2 & 3. Begin/End with "why it matters", "what did i learn", time spent, disk space. Don't reformulate hypothesis in conclusion. Add confidence intervals (user R).

2. I remember you told me about a framework for experiments. Could you give me more details, please?

**A**: Hypo, Experiments, Results, Conclusions

3. I added a title and the abstract paragraph. What do you think?

**A**: We will come back later.

4. Do you have any suggestion about code embedding, images, tables, diagrams?

**A**: Make sure to include images without compression.

5. Should I include code now or wait until the entire codebase is done and experiments finished? I expect that there will be some changes after:
- adding support for the random payload experiment
- adding support for DeLS. Should I also repeat the experiment with DiLS?

**A**: You can add code now even if it won't be identical to the final one.

6. Would it be a good idea to finish the second and third part first and then re-run the experiment with x1.5 payload so that I will be able to write the report while waiting for those results?

**A**: Sure.

7. Should I write on my behalf (I), our behalf (we) or neutral?

**A**: We = author + reader, not supervisor. You choose.

8. Am I remembering correctly that distortion is the name of the average cost?

**A**: Yes. Di = Distortion, De = Deflection

9. I started some experiments with fixed size and random payload (1-9% of size). Should I use random size, too? Should I use a bigger variation for payload (like 1-20%)?

**A**: Random size might be useful.

10. I run the same random experiment twice (with names -1bits and -2bits) and expected them to be the same. However, because I am computing the seed based on the full path, the results were slightly different. Would that be a problem?

**A**: No, as long as there are no big differences.

10. What do you think about the observations above?

**A**: We will come back later.

## 3 February 2021

### Changes
- added mean & std for detectability and estimates
- added density histograms
- added support for DiLS
- added support for DeLS

### Calling conventions
<details> <summary> See Calling conventions </summary>

- Cropper
- [width] [height] [source dir] [destination dir] [delta row / -1 random] [delata col / -1 random] [regex match]
- ./crop 300 500 ../images/ cropped/ 50 -1 .*pgm
- HILL
- --input [source] --output [destination]
- python3 cost.py --input "../images/*" --output /costs
- CostDisplay
- [source dir] [destination dir] [regex match]
- ./costDisplay costs/ display/ .*cost
- Stego
- [source dir] [cost dir] [destination dir] [diff dir / -nodiff] [stats dir / -nostats] [probs dir / -noprobs] [payload bits / <0 if random] [embedder 1 = PayloadLS, 2 = DiLS, 3 = DeLS] [regex match]
- ./stego images/ cost/ res/ diff/ stats/ probs/ 10000 1 .*pgm
- SRMQ1 feature extractor
- matlab -nodisplay < exit -r "extract [sourceDir] [probsDir / -noprobs] [destinationDir] [fileMatch]"
- matlab -nodisplay < exit -r "extract '../Pipe/Cropped/' '../Pipe/Probs/' './' '*.pgm'"
- echo "exit()" > Tools/exit
- Linear Classifier
- matlab -nodisplay < exit -r < exit "classify [sourceDir] [destinationDir] [statsDir] [resultDir] [fileMatch]"
- matlab -nodisplay < exit -r "classify('./CoverFeatures/', './StegoFeatures/', './Stats/', './Result/', '*.fea')"
- Statistics Expander
- ./statExpand [source dir ] [cost source dir ] [destination dir] [keep old 0/1] [embedder 1, 2, 3] [regex match]
- ./statExpand ./from/ ./fromCost/ ./to/ 0 1 .*.stats

</details>

### Average & Std of detectability 0.5 MB

- 20000 bits, 430 Di, 6 De

| | PayloadLS | DiLS | DeLS |
| - | - | - | - |
| Average | 0.41 | 0.39 | 0.52 |
| Std | 0.42 | 0.39 | 0.28 |

- 40000 bits, 1011 Di, 22 De

| | PayloadLS | DiLS | DeLS |
| - | - | - | - |
| Average | 0.69 | 0.72 | 0.92 |
| Std | 0.51 | 0.42 | 0.42 |

- 100000 bits, 3301 Di, 156 De

| | PayloadLS | DiLS | DeLS |
| - | - | - | - |
| Average | 1.40 | 1.59 | 1.72 |
| Std | 0.62 | 0.42 | 0.40 |

Scroll down on the [correlation page](https://hackmd.io/@W9LPPdtDTQCwdxGEjX9N8A/SyFkCL31O) to get more detailed results.

### Observations

1. It seems that both DeLS and DiLS improve the standard variation of the detectability.
2. DeLS seems to be marginally better because it provides at most the same variation for a larger average.

### Questions

1. Do you agree with the observations above?

**A**: It is surprising that the std is not significantly improved in the case of DeLS.

2. I solved DiLS with two nested interval bisections - one for $\lambda$ and one for each $\pi_i$. Therefore, it took 28 times longer to compute the stego objects. Fortunately, I was able to replace the nested bisection with 5 iterations of Newton's method. Does that sound all right? The overall time is 2-3x longer for the 0.5MB experiment.

**A**: Try precomputing the answer for some values

3. For the final results of parts II and III, I suggest running 6 experiment on 5MB images: (SMRQ1 / maxSRMQ1) x (PayloadLS / DiLS / DeLS). 2 of them are necessary in part II, the other 4 in part III. What do you think? Would one limit be enough or should I use 4 limits as before? (+ random payload experiments for PayloadLS)

**A**: Sounds ok. Maybe 4 as before would be more helpful.

4. I used 25 bars for the graphs. Should I use 100 to make the figure more useful?

**A**: Certainly

## 10 February 2021

### Changes
- added Skew & Kurtosis
- added Normal and Pearson approximations on the graphs
- started experiments again with the correct images
- new results link [HERE](https://hackmd.io/@W9LPPdtDTQCwdxGEjX9N8A/BkJu_GvZu).
- new correlations & density link [HERE](https://hackmd.io/@W9LPPdtDTQCwdxGEjX9N8A/rJsGs3PZu)

### Questions
1. I suppose I should describe how HILL works or at least what it does. Could you send me some documentation, please?

**A**: Done.

2. The new results are very similar to the old ones (on the wrong image set). Only some errors are improved which I think it is expected in out case (because the noise is reduced). What do you think?

**A**: That is the expected outcome.

3. The following bar chart has two approximations attached to it. The red one is a normal distribution while the blue one is a pearson distribution (with mean, std, skew, kurtosis). There are 10.000 images in 100 bars. Hence, I suppose that the total area is 10^(4-2) = 100 units. A distribution has area 1 so I multiplied it by 100. However, the result does not cover the bar chart. I can't understand why that happens. Could you help me, please? L.E. I also have to multiply by the width of the spanned x axis.


### Plain diagram

graph TD
Cover(Cover .pgm) --> Cropper{{Cropper}}
Cropper --> Cropped(Cropped .pgm)
Cropped --> HILL{{HILL}}
HILL --> Costs(Costs .cost)
Costs --> Embedder{{Embedder}}
Cropped --> Embedder
Embedder --> Stego(Stego .pgm)
Embedder --> Probs(Probabilities .probs)
Probs --> FeaExtrS
Probs --> FeaExtrC
Payload(Embedding Limit & Method) --> Embedder
Cropped --> FeaExtrC{{Feature Extractor}}
Stego --> FeaExtrS{{Feature Extractor}}
FeaExtrC --> CFea(Cover Features .fea)
FeaExtrS --> SFea(Stego Features .fea)
ExtrMethod--> FeaExtrS
ExtrMethod(Extracting Method) --> FeaExtrC
CFea --> Lin{{Linear Classifier}}
SFea --> Lin
Lin --> Resuls(Errors, Correlations, Graphs)

### Annotated diagram

graph TD
Cover(Cover .pgm) -- 100 GB --> Cropper{{Cropper}}
Cropper -- 45 m --> Cropped(Cropped .pgm)
Cropped -- 25 GB --> HILL{{HILL}}
HILL -- 1h 52m --> Costs(Costs .cost)
Costs -- 49 GB --> Embedder{{Embedder}}
Cropped -- 25 GB --> Embedder
Embedder -- 3h 42m --> Stego(Stego .pgm)
Embedder -- 3h 42m --> Probs(Probabilities .probs)
Probs -- 217 GB --> FeaExtrS
Probs -- 217 GB --> FeaExtrC
Payload(Embedding Limit & Method) --> Embedder
Cropped -- 25 GB --> FeaExtrC{{Feature Extractor}}
Stego -- 25 GB --> FeaExtrS{{Feature Extractor}}
FeaExtrC -- 12h 50m --> CFea(Cover Features .fea)
FeaExtrS -- 12h 50m --> SFea(Stego Features .fea)
ExtrMethod--> FeaExtrS
ExtrMethod(Extracting Method) --> FeaExtrC
CFea -- 1.5 GB --> Lin{{Linear Classifier}}
SFea -- 1.5 GB --> Lin
Lin -- 75m --> Resuls(Errors, Correlations, Graphs)

graph TD
Cover(Cover .pgm) -- 45m --> Cropped(Cropped .pgm)
Cropped -- 1h52m --> Costs(Costs .cost)
Costs --> Embedder(Embedder)
Cropped --> Embedder
Embedder -- 3h42m --> Stego(Stego .pgm)
Embedder --> Probs(Probabilities .probs)
Probs --> FeaExtrS
Probs --> FeaExtrC
Payload(Embedding limit) --> Embedder
Cropped --> FeaExtrC(Feature Extractor)
Stego --> FeaExtrS(Feature Extractor)
FeaExtrC -- 12h50m --> CFea(Cover Features .fea)
FeaExtrS -- 12h50m --> SFea(Stego Features .fea)
CFea --> Lin(Linear Classifier)
SFea --> Lin
Lin -- 75m --> Resuls(Errors, Correlations, Graphs)

graph TD
Cover(Cover .pgm) -- 25GB --> Cropped(Cropped .pgm)
Cropped -- 49GB --> Costs(Costs .cost)
Costs --> Embedder(Embedder)
Cropped --> Embedder
Embedder -- 25GB --> Stego(Stego .pgm)
Embedder -- 217GB  --> Probs(Probabilities .probs)
Probs --> FeaExtrS
Probs --> FeaExtrC
Payload(Embedding limit) --> Embedder
Cropped --> FeaExtrC(Feature Extractor)
Stego --> FeaExtrS(Feature Extractor)
FeaExtrC -- 1.5GB --> CFea(Cover Features .fea)
FeaExtrS -- 1.5GB --> SFea(Stego Features .fea)
CFea --> Lin(Linear Classifier)
SFea --> Lin
Lin -- 18MB --> Resuls(Errors, Correlations, Graphs)