# TODO
- [x] Rewrite introduction and related work to focus to visapp
- [x] Abstract: "data augmentation techniques are helping"
- [x] Put appendix in main text
- [x] Introduction: Emphasis on the problems related to manual annotation and cost related problems
=> Anh: i think we have already stated this via:
> Active learning (AL) utilizes uncertainty-based sam-pling to: 1. Select informative unlabeled examplesto annotate to reduce annotation costs 2. Select core-subsets from a redundant dataset, to reduce training samples and thus dataset size
- [x] Introduction: delete 1/ 2/ 3/ lists
- [x] Correct "very, fast training time" expressions in model description
changed version:
> which performs well with a fast inference speed compared to other architectures, reducing training and uncertainty computation time

- [x] Detail the stopping criterion in AL pipeline
- [ ] Detail the impact of DA on first AL steps and last AL steps
- [ ] Add a plot for xp B. to show the decreasing effect of DA over AL steps + add an analysis
- [x] Fig3: comapre BALD and random on a single figure
=> Anh: do we really need to compare BALD and random on a single figure?
the fig3 is for experiment A, "explore the performances of heuristics over random"
- [x] Section3: too many acronyms
=> Anh: i dont see section 3 has many acronyms
- [ ] Emphasis in results on gain on computation time
=> computation time measurement requires another settings such as early stopping on a validation set to be fair.
- [x] Conclusion: emphasis on practical gain in term of annotations costs
- [x] Add this reference on baseline sampling method (https://arxiv.org/pdf/1911.09168.pdf)
- [ ] Native English reading
- [x] The results are promising, though seemingly not stunning. I do wonder if it would help to consider a better method than dropout for uncertainty quantification. For example, there's a growing body of evidence that explicit ensembles provide better uncertainties than dropout (see, e.g., https://arxiv.org/abs/1612.01474), and recent work on improving their computational efficiency (e.g., https://arxiv.org/abs/2110.03360) that might be interesting to explore.
--> Maybe cite those papers, and relates that mc dropout for uncertainty estimation maybe be not the best method, but it has proven good results in the past on multiple known tasks, and it offers a stronger baseline method.

# Introduction:
>
> The motivations behind the studied problem could perhaps be clarified more precisely.
>
> I think it could be more clearly explained that dataset redundancy is a critical issue as it leads to large datasets.

=> AA: to take into account, as VISAPP is a more global conf

> And the drawbacks of large datasets (annotation cost, training time) could next be explained and active learning and data augmentation techniques introduced as possible solutions to handle such issues.

=> we've already explain via the main goals of AL
=> AA: the drawbacks of large datasets is not explicit in the intro, and we're not making a direct link with DA on this topic

> As annotating large dataset is a very significant issue in practice, more stress should be put on this aspect (since active learning techniques can reduce the burden and cost of ground truth annotation).
> It should be precise that even though acquiring data is expensive, the cost of labelled data is higher compared to the cost of unlabelled data.

=> AA: to add

> The end of the introduction could also explained the practical benefits of reducing the dataset size of Semantic-Kitti by 40% (mainly annotation costs and possibly training time).
>
=> we've already explain via the main goals of AL
=> AA: No, we are not explicitly talking about the reduction of labeling effort for our task
=> Anh: we did.
> Active learning (AL) utilizes uncertainty-based sam-pling to: 1. Select informative unlabeled examplesto annotate to reduce annotation costs 2. Select core-subsets from a redundant dataset, to reduce trainingsamples and thus dataset size
>
# Related work:
> Not a lot of remarks here.

# Method:
> Inside the model section, the expression "very, fast training time" seems a bit imprecise. If possible, give some comparison to other approaches (number of epochs, training time by epochs).

=> inference time?
=> AA: training time per epoch + number of parameters to train can be good. Graph on salsanext paper can help
=> Anh: training time per epoch + number of parameters ~= inference speed. ssv2 authors only mentioned inference speed in the paper.

# Experimental setup:
> It's a bit unclear for me how each train step is stopped and when the whole training is stopped

=> I think we ve aldready describe that in the experiments, the whole training is stop when there is no unlabeled data left.
=> AA: can be added, indeed not explicit

> do we continue to query samples while non-annotated data is available).

=> out of the paper focus, interesting in practice

> At the end of the section it is explained that an early stopping policy is used for each train step, but I assume that it's not the only criterion (like a maximum number of epochs).
> There is a max train iterations hyper parameter in the table 1, but it's not obvious if it's for the model trained without the AL setup or for the train steps inside each AL step.

=> the model trained without the AL setup is the training at the last step.
=> we should precise more in the table that LRLR decay,Weight decay,Batch size,Max train iterations are hyperparameters for each AL step.
=> AA: agreed

> I think the stopping criterion for the whole AL loop should be explained.
I understand that you query all the available data (on the studied subset) to study the performances on the model with more data.
But if this active learning setup is used in practice, we don't want to annotate all the available data and I would typically stop it when the trained model is achieving a given perfomance on a test set.
It's maybe obvious but I think it could be written rapidly in the paper.

=> Our focus is DA on AL (improving AL heuristics/strategy), but not stopping criteria. Stopping criteria is another problem in AL, interesting in practice.
=> AA: it's not our main focus, but it should be adress, as our application is on a very practical task.

# Experiments:
> In section A, the random heuristic is considered as a baseline query method.
> If the data is organized in temporal sequences of point clouds, another basic acquisation method is to query pointclouds that are the most temporally distant from already annotated data.
> Because it can be assumed that the pointclouds distant in time differ more compare to consecutive pointclouds.
> The random strategy is still a good baseline to study but the one proposed previoulsy could also be interesting.

=> there is a paper using this technique. (https://arxiv.org/pdf/1911.09168.pdf)

> In section A I'm not sure the "labeling efficiency ratio" was defined before.
> Even if it's not difficult to undertand it, it could be explained more precisely if not defined before in the paper.
> Note: I saw later it was in the appendix (so if it was not done before, put a link to this appendix).

=> Although we explained evaluation metrics in Experimental Setup (also linked to appendix), we should a link to this appendix to every labeling efficiency appear in the paper

> It seems that the data augmentation is improving significantly the performances of the models during the first AL steps (which makes sense as the dataset size used to train the model for the first AL steps is small).
> However it seems that the data augmentation does not bring a significant improvement for the final AL steps and for the model trained directly on the whole data.
> It indicates that the data augmentation strategy should perhaps depend on the AL step and/or the dataset size.
> Also it shows that the data augmentation policy used for the AL setting should not be selected based on results on the whole dataset, but more on small subsets of the data.

=> in practice.
=> AA: I think it's a good analysis, and adding it will give a more precise view on the effect of DA

> In the figure 2, the resuts are reported with the nuber of training samples.
> It could be good, if possible, to also report the MeanIOU with an estimation of the total computation time for the AL loop (for instance the total number of pointclouds seen during all the previous training steps).
> For me the number of training samples shows the performance we can expect with a reduced dataset (and thus with a reduced annotation cost), but it does not indicate the potential gain of the AL loop in term of computational ressources.

> We could expect to speed-up the total model training time with the described AL setup, as the first train step should be faster due to the small size of the training data (but thereis maybe no speed-up when taking into account all the AL steps).

=>

# Discussion and conclusions:
> As for the introduction, I think the practical gains in term of annotation costs (and possibly training time) could be stated more explicity.
> It could also be noted that the gain of data augmentation depends greatly on the AL step (being significant for the early steps only), and could perhaps lead to design data augmentations strategies more specific to each AL step.
>

=> we should mention that the gain of data augmentation depends size of train set.

# General remarks:
> I find this paper correctly written and the experiments are globally well described.
> I think the practical benefits of the studied approach should be explained more clearly inside the introduction and the conclusion (especially concerning the annotation cost, which is a critical issue when we want to apply DL methods in practice).