Research Project Spring-2023

# Research Project Spring-2023 ## Speleograph - Model, Manage, and Visualize Sensor Data from Underground Environment ### Shubhika Garg --- ## Principle: We want to store underground data, and in particular the time series of data from our sensors on a server, and give users access to it in a simple way, through a tool querying and viewing this data. --- ## Architecture: - One or more host serves store the .csv files describing the sensor data series, as well as the metadata describing these series. - A user form specifying search criteria for cavities and time series. - Speleograph to view the results. --- #### **10/03/2023 Discussion:** - [x] Github account - [ ] Ontology UiS - [ ] Mapping (Direct Mapping or R2RML) - [ ] Use Case of Data Analytics --- #### **Some documentation for the Speleograph project:** - [ ] [The project home page](https://ontology.uis-speleo.org/) - [ ] [A human-oriented description of the Ontology structure](https://ontology.uis-speleo.org/ontology/) - [ ] [An example](https://ontology.uis-speleo.org/ontology/) - [ ] [explanation for Database managers how to develop a karstlink compliant conection](https://ontology.uis-speleo.org/ontology/how-to) - [ ] [The Core Ontology, in Turtle format](https://ontology.uis-speleo.org/ontology/karstlink.owl.ttl) - [ ] [The UIS Karstlink sub-commission page, explaining the overall organisation and plans](https://www.uisic.uis-speleo.org/exchange/karstlink/index-en.html) - [ ] [Le wiki, with explanations of the main concepts, and more details/refs/discussion on each main Entities of the ontology](http://uisic.uis-speleo.org/wiki/karstlink/index.php/KarstLink) (Note: this not yet including the entities of "Season2, i.e. the time-series paradigms) - [ ] [General Presentation](http://uisic.uis-speleo.org/wiki/karstlink/images/9/9c/Karstlink_ICS2021.pdf) --- #### **17/03/2023 Discussion:** - [ ] [Grottocenter (Caves+People+Organization)](https://grottocenter.org/) - [ ] [Karsteau (Restricted)](https://www.karsteau.org/karsteau/index.php) - [ ] BBS Documentation - [ ] All the above are connected in Karstlink - [ ] Grottocenter doesn't have SPARQL, so last year it was Speleograph connecting to Grottocenter using API; and sending HTTP Request to get csv files from web (INRIA) - [ ] Contributors can contribute in the database (Implementation required) - [ ] Getting a list of caves you need; Then you filter again based on type of data (Pressure,etc) and the time duration: Frontened - [ ] Metadata list of quantities you are measuring along with units - [ ] Speleograph to be a separate web-app for now. - [ ] Frontened architecture to be defined - [ ] We might need to pre-process the csv file - [ ] Need to decide for mapping for metadata --- ##### **Frontened architecture:** - **Login Page**: Licence Number, Name, Email, Forgot Password - **Contribution Page**: Author, No files to be uploaded but the author should upload a link to his/her .csv file Example Link: (http://www-sop.inria.fr/agos-sophia/sis/KarstLinkS2/TimeSeries/Beget/Beget-perte-SU15681-jan-avril2021.csv) - **Filtering list page**: Choose multiple cave options (maybe something like a checkbox) - **Time series Filtering**: Adding date_From and date_To and also the physical quantities like Pressure, Temperature, etc.(maybe dropdown+checkbox); Also, we get .csv files for the results - **Speleograph**: to be another web-app for visualization maybe with .csv files which were provided on sorting in Time Series as an input - **Admin** --- #### **24/03/2023 Discussion:** - Contribution Page with an option of file upload also - Then, an option to check if metadata properly there or not - If user checks "No", then multiple options to add the fields like "Time", "Pressure" etc and that should be sent as a list to the backened. - Figma maybe for frontend - **Contributor Id**: To be defined elsewhere. - **Point of observation**: Can be either the entrance point of a "Cave", or some Point inside a cave. The way that this point is built is not important right now, it will boil down to some (lat/long) coordinates, you can assume for the moment that we only use entrance points, that you will get from form Grottocenter. - **Sensor Type**: It will be either "Reefnet", "CTD", or "Hobo Pluviometer". These are defined in the extended ontology. --- **Some Details on the Backened:** - [Postman](https://documenter.getpostman.com/view/19206411/2s93RMWFkz) - **Technologies:** The Project’s Backend is a crucial component that handles server-side logic, database operations, and communication with the front-end. In this report, I will explain the various technologies you will be using for your backend project. • [Mongoose MongoDb](https://en.wikipedia.org/wiki/MongoDB) • [NodeJs](https://en.wikipedia.org/wiki/Node.js) • [Typescript](https://en.wikipedia.org/wiki/TypeScript) • Joi Validation • [JWT (Json Web Tokens)](https://en.wikipedia.org/wiki/JSON_Web_Token) • [Passport](https://www.passportjs.org/) • [Docker](https://www.docker.com/) • Multer • GrottoCenter APIs • Github - **Mongoose - MongoDB:** Mongoose is an Object Data Modeling (ODM) library that provides a structured schema-based approach to model your data. It is widely used with MongoDB, a popular NoSQL database that stores data in a JSON-like format. Mongoose simplifies interactions with MongoDB by providing a fluent API, schema validation, and middleware hooks. - **Node.js:** Node.js is a server-side JavaScript runtime that is built on top of the Chrome V8 JavaScript engine. It enables developers to write server-side applications using JavaScript. Node.js provides a non-blocking I/O model, which allows for fast and efficient handling of large amounts of data. - **TypeScript:** TypeScript is a strongly-typed superset of JavaScript that adds optional static typing, interfaces, classes, and other features to the language. It helps developers catch type-related errors at compile time rather than runtime, which improves code quality and reduces bugs. - **Joi Validation:** Joi is a library used for validation of JavaScript objects. It provides a powerful schema description language that allows developers to define constraints for objects and validate them easily. It is widely used in Node.js applications to validate user input, API requests, and database entries. - **JWT:** JWT (JSON Web Token) is an open standard for securely transmitting information between parties as a JSON object. It is commonly used for authentication and authorization purposes in web applications - **Passport:** Passport is a popular authentication middleware for Node.js that simplifies the process of authenticating users in web applications. It provides a simple, yet flexible framework for implementing authentication strategies, such as username/password - **Docker:** Docker is a containerization platform that allows developers to package an application and its dependencies into a single container. It provides an isolated environment for running applications, which ensures consistency and portability across different environments. - **Multer for file uploads:** Multer is a middleware used for handling file uploads in Node.js applications. It supports various storage options, including in-memory storage, disk storage, and cloud storage. It simplifies the process of handling file uploads and provides a powerful API for validating and manipulating uploaded files. - **GrottoCenter APIs:** GrottoCenter Gives us access to a countless number of caves we can query caves and get this data to be used in our application - **Github:** GitHub is a web-based platform that allows developers to collaborate on and share their code with others using the Git version control system. It provides tools for code hosting, issue tracking, pull requests, code reviews, and more. It is widely used by open-source projects and has become an essential tool for software development teams. - **Conclusion on the backened architecture:** The technologies we have chosen for the backend project are well-suited for building a robust and scalable application. Mongoose and MongoDB provide a reliable database solution, Node.js and Typescript offer an efficient and powerful server-side environment, while Joi, JWT Passport, and Multer add essential functionality for handling user input, authentication, and file uploads. Docker enables you to deploy your application easily and consistently across different environments. By using these technologies, we can build a high-quality backend for your project that meets your requirements and exceeds your users' expectations. --- **Eric's Email Details dated 28/03/2023:** - The Time series is a subclass of the SOSA "Observation" class. To create a Time series instance, the contributor form should ask for the following information: - Contributor - Point - Sensor - ObservedProperty - Begin Date_Time - End Date_Time - "Underground Cavity" it belongs to (which is a URI, e.g. grottocenter ID). - To create predefined instances, you will need to have a Sensor from SOSA that has a Sensor_Type among known types, which should be provided in a fixed list. Currently, the known types are Reefnet, CTD, and later on, Hobo_pluviometer. - The extended ontology file that was provided may not fully implement this, but it should have resources specifying the sensor's serial number and one or more properties it measures. Additionally, there is a discussion about whether the sensor instance should store calibration information. - The Sensor_type is currently not used in the extended ontology, but it should hold information about a specific model of sensor, with its provider, model, and observed properties. - Observed properties are "ObservableProperties" from the SOSA ontology and should be attached with a (QUDT) Unit. Currently, we have Pressure (Pression) with unit#Pascal, Temperature with unit#DegreeCelsuis, and Conductivity with unit#MilliSiemens_by_centimeter. - DateTimes are from xsd schemas (http://www.w3.org/2001/XMLSchema#) and can be used with full information or only the date. - Open problems to be resolved between front-end and back-end teams: - Sensors observe multiple properties, but for storage efficiency, all properties are encoded in a single CSV file. However, users and apps may need to search and use these properties separately. - For example, a user may want to search for all pressure time series in a geographic area and find them in both reefnet and CTD CSV files. - Currently, the time-series from the Lea example specify only one physical property per "time-series" instance, resulting in duplicate metadata objects. - This problem could be approached from the opposite direction. --- #### **31/03/2023 Discussion:** **Contributor Form changes** * Cave ID (Grottocenter ID) * DateTime along with the timezone * On uploading a file from the local, validate whether it is .csv or not, else ask user to upload a valid .csv file * Point of Observation: different coordinate systems all over the world * Name/email of contributor can be fetched from the user authentication * Select either the sensor type or the observed properties * Reefnet for example, has predefined properties (Pressure and Temperature) * Add another sensor type as "Other" and if he chooses this, then he should be directed to another form where he should enter the "Observed Properties" on his own and also the file headers if the file is not in the standard format **Other functionalities proposed** * An option of use-profile on top. There may be an option of the contributions done by the user like: * CONTRIBUTION 1: * The CSV FILE URL to be displayed * And maybe a sort of pdf version of his contribution to be viewed and downloaded #### **07/04/2023 Discussion:** - Sensor type drop-down should have Standard Reefnet and Compact Reefnet - For other sensor type, sensor name, type, manufacturer and a list of observed properties which the user needs to check and also their units - Each sensor has a different unit for observed properties - CSV Compacted version = backened maybe (adding headers and storing locally) - select, define, done - Pre-defined proeperties: Drop-down and suppose he selects "Temperature", a drop-down adjacent to it should come with the units drop-down - if he selects a "define", then observed property should be an input text field with a drop-down of all units #### **14/04/2023 Discussion:** - access to server - connect frontend and backend - grottocenter API in search form - Reefnet Temperature in Kelvin not Celsius - Provide user-documentation for the cavers and technical documentation for future developers - Display country name when user selects a country code in search form Reset Password enter OTP newpassword confirm password Contribute CAVE ID For upload, get file URL from server and then send that url on submit begin and end date sensor id sensor if default Home should have list of contributions of user add sensor serial number in drop-down alongwith sensor type