InCorr cover letter

--- tags: academic, incorr title: InCorr cover letter --- --- *Summary of revisions made to submission 1130 of VAST 2020* # InCorr: Interactive Data-Driven Correlation Panels for Digital Outcrop Analysis *Thomas Ortner, Andreas Walch, Rebecca Nowak, Robert Barnes, Thomas Höllt, and M. Eduard Gröller* --- We thank all the reviewers for their positive reviews and the constructive comments that have helped us to improve our paper. Almost all reviewers agreed on InCorr being a questionable fit for VAST, so we lead with a clarification of our reasoning behind the decision to submit to said track. Then we provide an overview of the major changes in response to the reviewers' comments. These changes are enumerated as A to E, and we refer to them in the detail section, where we respond to each reviewer's comments individually. # Submission to VAST We understand that our solution does not contain any data analytics algorithms, but rather relies on the domain expert performing the visual analysis, e.g. judging the similarity of strata based on rock characteristics and the distribution of dipping orientations. Nevertheless, we felt that a VAST submission would be the best fit, because our solution covers the whole workflow from logging and data transformations, to interactive annotation of logs and correlation analysis resulting in an exportable illustration that is the basis for scientific dissemination. In this regard, one of the key goals of InCorr is to allow for the reproducibility and traceability of the resulting correlation analysis. However, in retrospect we could also see InCorr as a design study and application paper in the InfoVis track. Given the wide range of aspects covered by InCorr, we could not identify a single track as clear-cut for the submission of our work. Our decision for VAST was supported by the following process of elimination. Although the digital outcrop models (DOMs) could qualify as scientific data, their characteristics do not pose the typical challenges discussed in SciVis, as it is the case with volume data or data from computational fluid dynamics. Ultimately, we decided against submitting to SciVis, because the solutions discussed in InCorr do not present typical SciVis contributions, which we see as focussing on efficient data structures, and interaction and rendering techniques for said data. A similar reasoning lead to our decision against an InfoVis submission. We did not invent a new visual representation, instead we translated the existing correlation panel encodings from a static illustration to a dynamic visualization to support an integrated workflow. As commented by some of the reviewers, this representation is rather specific in terms of what data it can represent and what use case it can serve, so it does not serve the broad applicability typically found in InfoVis contributions. On a meta-level, currently the IEEE Vis conference is restructured with the three tracks (VAST, InfoVis, SciVis) being replaced by an area model. So starting from next year on the historically developed three track system will be replaced anyway. # Overview of Major Changes * **(A) Revision of the user study and more information on design process** We added a detailed description of our 3-phase design process and the evolution of InCorr to Section 4 and added the suggested reference to "Participatory Visualization Design". We clarified the roles of our collaborators also in relation to the participants of our Design Validation session. We changed the section header "User Study" to "Design Validation", which reflects our intentions more accurately. * **(B) Geological terminology** We understand that introducing a whole domain at once can be overwhelming. We try to keep geological terminology to a necessary minimum, while still explaining all terms from first principle to provide a complete context. We added a glossary table at the end of Section 3, which summarizes the most important terms and shall serve as a lookup table throughout the paper. * **\(C\) Data and mapping** We revised Section 4.2 and removed all programming specific terms from the model descriptions. Instead of presenting the finished data model we provide details on how to infer a hierarchical structure of strata from a (non-hierarchical) set of contact annotations. * **(D) Revised structure** We moved the Previous Work section from the end (Section 7) to the front (Section 2) of the paper. We integrated Implementation (formerly Section 5) into the main section InCorr (Section 5.3). # Detailed Response to the Comments by each Reviewer In this section we respond to each statement individually by referring to a letter **(X)** from the changes list and/or by a direct comment indicated by ... > ... this quoted text style. > We submitted an additional version of our paper where we indicate all changed portions by green coloring. ---------------------------------------------------------------- ## Coordinator Review R3 ### Paper type Application & Design Study ### Expertise Expert ### Overall Rating 4.5 - Between Accept and Strong Accept ### Supplemental Materials Acceptable with minor revisions (specify revisions in The Review section) ### Justification ➕ interesting design study in an application domain rarely targeted in VIS ➕ involves domain experts and features a participatory design process ➕ nice visual representation of the data ➖ VAST relation questionable ➖ minor presentation issues ### The Review The paper presents the visual interface InCorr, which supports geologists in analyzing and annotating 3D outcrop model that have been measured previously on site. The heart of InCorr is a correlation panel that offers a simplified representation of the sediment layers, that can be enriched with domain-specific information. Considering the fact that the tool has been developed in close collaboration with geologists, and that it seemingly supports workflows, I find the project as a whole highly relevant for presention in the context of VIS. #### General Feedback: In the past decade, VIS featured numerous interdisciplinary design studies in a variety of application areas, e.g., bioinformatics, medicine and the digital humanities. The paper at hand is a valuable addition to this pool as it documents a design study in geology, and it provides a well-justified and well-comprehensible visual typology for the addressed topic. However, I do not particularly understand why the authors targeted the VAST conference, as the presented solution does neither support a visual analytics task nor does it discuss a large-scale data set. I think SciVis would have been a much better address, but also InfoVis would have been a better fit in my point of view. However, I am tending towards acceptance as I really appreciate the work that has been done, irrespective of the *right* VIS subcommunity. However, in case of a VAST-acceptance, the authors should clearly highlight why, in their point of view, the presented topic relates. > We elaborate on our decision process within this cover letter, to make our decision for the VAST track more comprehensible. We cannot change the contributions of InCorr to be a perfect fit for VAST, since there is no data analytics or statistical modelling involved. However, we argue that the transformation of the input data, its visual representation, and the integration into a complete workflow from measuring to publishing may be of interest to the VAST community. #### Interdisciplinary project and user study: The work is presented in the light of the task taxonomy by Brehmer, which makes perfect sense. Being involved in numerous close interdisciplinary collaborations in the past decade myself, I can see in-between the lines the influence of the domain experts on the design. I suggest to better highlight and discuss the interdisciplinary aspects on how the design evolved. I expect the geologists had quite a huge share, but it is only marginally mentioned at different sections of the paper. Maybe the authors could discuss the project in the light of paticipatory visual design [1] which seems to have been employed in the project. Also, a more detailed information on how the tool has been used by the domain experts at the hand of a simple example would be nice. A process overview could also help to better understand how the tool is embedded in current workflows. It would also be good to know how straightforward the results of the tool can be used in geology publications, and if the design is accepted in the field. > Please refer to change **(A)**. The tool has been used in the context of the validation and verification of InCorr. Our design validation aimed to answer if such panels could be found in published work, which was agreed upon by all participants. However, our collaborators are currently working on a publication utlizing the tools provided by InCorr, and only after we will be able gauge the acceptance in the community reliably. #### User Study: It maybe relates to general expectations of what a paper requires to be accepted, but the presented approach and project does not require a traditional *user study*. The tool has been developed in cooperation, and seemingly, design considerations are likely heavily influenced by domain experts (again, this is my guess and should be clearly articulated). In this context, the visuals do make sense and the tool does not necessarily needs a typical user study, at least I would not call it as such. It is more of an informal meeting that evaluates possible extensions and adaptions in other, closely related contexts. And this is fine, but the paper itself does not need reflections like "I can’t wait to get my hands on this and give it a shot". > Please refer to change **(A)**. We further removed the quotation "I can’t wait to get my hands on this and give it a shot". #### Minor Remarks: - The chapter "Previous Work" comes inappropriately at the end. I suggest to discuss those after the introduction as to contextualize and motivate their solution accordingly. > Please refer to change **(D)**. - Are adjustments after re-measurements, like for example mofifying the height of segments, possile (or not necessary)? > Our data-driven approach does support that, but it was not of primary concern with our collaborators when only dealing with a handful of logs. This features will become more important when revisiting larger correlation analysis or collaborating on a large analysis. - I like to see a more formal description of the data features, and how those are mapped to visual features. > Please refer to change **\(C\)**. We think that the changes to 'Section 4.2 Data Transformation' made the mapping to visual features described in 'Section 5.1.1' clearer. - I found the explaination of the data model including the names of classes and programming-specific terms inappropriate. A rather abstract definition of how the shared mental model evolved are sufficient. > Please refer to change **\(C\)**. Instead of the evolution of the shared mental model we described the algorithm to translate from annotations to logs. This transformation is also reflected in the stages of our 3-phase design process. - the implementation does not deserve an own chapter > Please refer to change **(D)**. - There are many terms that are specific to geology. Many of those are somehow explained, but the reader could benefit from a specific terminology chapter that explains the most important terms accordingly, and briefly that it serves as a lookup if necessary. > Please refer to change **(B)**. - Figure 3 contains the information "Illustrated Manually", it was not clear to me what is meant by it. > We clarified what we mean by "illustrated manually" in the figures's caption. We revised all sections to use consistent phrasing: "illustrated manually", "manually illustrated", or "manually created illustration". [1] Participatory Visualization Design as an Approach to Minimize the Gap between Research and Application. Proceedings of the Workshop on the Gap between Visualization Research and Visualization Software (VisGap) ### Supplemental review notes Uploaded file. See the online review for the file. ### Summary Rating Accept The paper should be accepted with some minor revisions. Once these have been completed it will meet the quality standard. ### The Summary Review There was a consensus among the reviewers that, with geology, the paper addresses an interesting domain alongside with a design study that is somewhat special as it deals with a problem that has not yet been tackled. Further, the paper is well-written, and it contains all major components that are necessary for a TVCG publication. Therefore, we suggest conditionally accepting the paper. However, final acceptance is subject to the following major changes to be prepared for the second round review cycle: - clarify contribution to VAST (R1,R3,R4) > Please refer to 'Submission to VAST' and changes **(A)** and **\(C\)** - revise user study (R3) > Please refer to change **(A)** - improve discussion of domain-specific terminology (R1,R3) > Please refer to change **(B)** Especially the first requested change is crucial as the reviewers do not follow why the paper was submitted to VAST as most value was seen by the project as a whole and the rather data-driven visualization metaphor. Please also take a look at the individual reviews to revise your manuscript accordingly. ### Second round comments (public) (blank) ### Second round supplementary materials check (blank) ### Second round supplementary materials comments (blank) ---------------------------------------------------------------- ## Committee Member Review R1 ### Paper type Application & Design Study ### Expertise No or passing Knowledge ### Overall Rating 3 - Possible Accept The paper is not acceptable in its current state, but might be made acceptable with significant revisions within the conference review cycle. If the specified revisions are addressed fully and effectively I may be able to return a score of '4 - Accept'. ### Supplemental Materials Acceptable with minor revisions (specify revisions in The Review section) ### Justification The paper presents an interactive system for analyzing 3D digital outcrop models, allowing to specify the different details of a single crop and correlating them to get an overview of ancient landscapes. The project raises in the context of an ESA project for the reconstruction of ancient environments on Mars. Pros: ➕ It is a solid project, clearly addressing a complex and useful activity; ➕ It has clearly developed with real expert users, addressing real needs; ➕ It has been validated with a convincing use case and user study. Cons: ➖ It is mainly a sophisticated "annotation" system and it does not use analytics nor allows for cycling among the classical VA visual analysis/model analysis steps; > Please refer to 'Submission to VAST' and changes **(A)** and **\(C\)**. ➖ It deals with an intricate and hard to grasp domain; > Please refer to change **(B)**. ➖ The terminology and the details involved in the process are somehow overwhelming; > Please refer to change **(B)**. ### The Review The paper presents a valuable solution for the interactive analysis of 3D digital outcrop models (DOM); the expert geologist can interact with the digital images, specifying the aspects of the DOM, i.e., different strata, contacts, grain sizes, etc. Moreover, the system supports the (physical) correlation among a series of DOMS, connecting corresponding contact points and getting an overview of complex landscapes. The paper aims at addressing a key point in the analysis of Mars landscapes. I like the system described in the paper, but I'm concerned about its scientific contribution and its relationship with VAST objectives. In particular, it seems mainly a sophisticated annotation system, in which the "Integration of data analysis, interaction, and visualization" does not take place. > Please refer to changes **(A)** and **\(C\)**. The second point is that half of the paper deals with the details of the sedimentology analysis, making the paper hard to read. Moreover, the proposed solution is very specific, and I do not see an immediate way of reusing it in a different context. As a consequence, I'm wondering if the paper will be able to attract the interest of the VAST audience. > Please refer to changes **(A)**, **(B)**, and **\(C\)**. Detailed review The terminology used in the paper is very specific, and I suggest authors to anticipate all the concepts and words in a subsection at the end of the introduction, producing a sort of informal glossary. > Please refer to change **(B)**. Section 2.2. It is not clear to me why "true thickness" "complicates this matter significantly". While the difference between true and apparent thickness is obvious I miss the implications: please elaborate on it. > We added the text *'Hence instead of simply using global elevation values over all strata and logs, each apparent thickness needs to be corrected by a stratum's dipping angle. However, in stratigraphy it is an accepted simplification to use one angle per log, which still results in each log creating its own coordinate system.'* to Section 3.3. Section 3.1 Likely a summary table of tasks and subtasks will help the readability of the section. > We orginally considered using the visual representation provided Brehmer and Munzner's Multi-level Task Typology to illustrate the tasks but decided against it due to space restrictions. Section 3.2. The description of the Data model is too technical; I suggest to rephrase the section, focusing on the modeled concepts. > Please refer to change **\(C\)**. The provided video will benefit from some descriptive audio. > We added an audio track to the supplemental video. ### Supplemental review notes (no file) ---------------------------------------------------------------- ## Reviewer 2 Review ### Paper type Application & Design Study ### Expertise Knowledgeable ### Overall Rating 5 - Strong Accept The paper is an excellent contribution and should certainly be accepted, possibly with some minor revisions. It more than meets the quality standard. ### Supplemental Materials Acceptable ### Justification This paper has a big job to do: not only do the authors describe their work and its benefits, but they also take the time to tell the users enough about stratigraphy to appreciate the work. I am impressed with how thoroughly they accomplish both goals without making their paper seem cramped. Strengths: The application they have chosen is an obvious candidate for visual analytics tools. The authors have (sensibly) gone after the low-hanging fruit in order to make an order-of-magnitude difference in how quickly geologists can analyze the data they have gathered. The influence of their user community is evident in the decisions of which parts of the correlation panel to manage automatically and which to leave to manual specification. The clear statement of design goals and tasks makes it easier to assess this system as a research effort in visual analysis. Weaknesses: I would very much like to see an example of InCorr being put to use on data from Mars! I think that the lower-quality data and multispectral images may bring to light some challenges that are not apparent when working with arbitrarily-high-quality terrestrial data. I am, however, aware that solar system exploration missions are not conducted on our publication schedule. > We do have digital outcrop models from ongoing and past Mars missions (Spirit, Opportunity, and Curiosity). At the time of this submission we were not able to include a Martian case study, because the work is still ongoing in the form of a publication by our collaborators. We are eager to see the results and share them at our prospective conference talk. So, why the high score and recommendation for Best Paper? This is the kind of paper that I would get up early to go see. I find the application compelling. The work done is clearly described and well motivated. The results are clear. Task and goal analyses are stated clearly and tied explicitly to design decisions. The paper is a model of clarity. ### The Review First off: I am very impressed with this work and I commend the authors. Brief summary: Stratigraphic analysis is an area of study in geology related to the arrangement and fabric of layers of sedimentary rock. Strata are only visible when some event exposes them, whether natural (an outcropping) or man-made (an excavation or core sample). By annotating strata and correlating these annotations across multiple measurements in an area, geologists assemble a larger picture of what they believe the subsurface structure is. This is a highly labor-intensive process, most of which is done manually. InCorr addresses two phases of stratigraphy: constructing a contact log from a digital outcrop model, then constructing a correlation panel from a series of contact logs. At the most basic level, constructing the contact log is a matter of creating a polyline on a triangulated mesh by choosing vertices at discontinuities in the texture. Dip-and-strike measurements are associated manually with the segments of this polyline. In a second phase, each segment of the polyline becomes an entry in a contact log. Users select material types for each segment. Finally, multiple annotated contact logs are semi-automatically assembled into a correlation panel that can be exported as SVG for further refinement. Positives: The methods applied here are as simple as they can possibly be. I mean that as a compliment: they free the user up to concentrate on the geology instead of wrestling with an over-complex model or application. The application is compelling. Until we can go to Mars in person with compass clinometers, we have to make do with software. The design goals and interaction tasks are stated clearly and tied explicitly to parts of the application. The paper itself is a model of clarity. Negatives: There are a couple of items that are not detailed. How do users differentiate between apparent and true thickness for bedsets? How do they estimate dip and strike angles? How are contacts identified on a DOM before the contact log is created? These might be good things to note in the live presentation. > Due to space restrictions we focus on the part of the workflow, where InCorr contributes. Details on the other parts of the workflow, such as data acquisition, processing, or interpretations can be found in Barnes et al. [[1]](https://www.vrvis.at/publications/PB-VRVis-2018-019). There isn't much discussion of other systems that can perform stratigraphic analysis. I would appreciate knowing what parts of LIME and VRGS (for example) the authors found inspiring. I recognize that the 9-page limit makes it difficult to discuss these in depth. > Existing tools are rather focused on measurements and data integration. For instance, LIME and VRGS allow geologists to project manually illustrated logs onto the 3D surface. The closest to our approach only supports logging through a customized multi-tool workflow described by Nesbit et al. [19]. The inspiration for our work mostly came from manually illustrated and some-times hand-drawn correlation panels found in geological publications. We do not yet have a demonstration of this system on real data from Mars. Harrumph. I am forced to concede that we cannot stage a mission to Mars to gather this data before the conference later this year. > Please refer to the statement of the `Justification` section above. Revisions Requested: I spotted a couple of minor typos. The word "logging" is repeated twice in the abstract -- "logging logging mechanism". In the conclusion, there is an apostrophe missing in the phrase "We demonstrate InCorrs functionality...". > We corrected the text accordingly. The fact that the outcroppings in the Hanksville-Burpee are named Sol2 through Sol5 is slightly confusing since days on Mars are also referred to as sols. I don't know that there's much to do about this. > Rover missions on Mars often do not and probably cannot return data products on the very first day. In this particular terrestrial case, the reconstruction from pictures taken on Sol1 did not yield a usable digital outcrop and therefore Sol1 was discarded. ### Supplemental Material: Big thanks to the authors for making the entire video accessible without audio. ### Supplemental review notes (no file) ---------------------------------------------------------------- ## Reviewer 4 Review ### Paper type Application & Design Study ### Expertise Knowledgeable ### Overall Rating 3.5 - Between Possible Accept and Accept ### Supplemental Materials Acceptable ### Justification This paper presents a visualization solution, i.e., InCorr, that encompasses a 3D logging tool and an interactive correlation panel that evolves with the stratigraphic analysis. However, the paper is not well written in the aspect of the data-driven technique, which can be improved. ### The Review Positives: ➕ With InCorr, this paper addresses a set of tasks that bridges the gap between outcrop interpretation and the creation of a geological model based on logs and correlations. ➕ InCorr consists of three components: (1) the InCorrPanel, a 2D interactive correlation panel, (2) a logging tool, and (3) a list view for assigning rock types to strata in the InCorrPanel. ➕ InCorr is verified through a use case from a terrestrial campaign and a small-scale user-study to collect feedback from a broader range of geologists is conducted. Negatives: ➖ One of the key themes in the proposed work is to design a completely data-driven panel, but how to realize the data-driven technique and what is the innovation of it are not clearly discussed in the paper. > Please refer to change **\(C\)**. ### Supplemental review notes (no file) ----------------------------------------------------------------