# Introduction
The Label Repository is a unique repository within TelOS, designed to store label metadata associated with a specific record_key. The primary responsibility of the Label Repository is to handle CRUD operations and search functionality. Labels stored in this repository can be generated through various sources, including human input, AutoML, or other processes.

For instance, consider an image where the system has identified various types of fruits and labeled them accordingly.

image source: https://www.superannotate.com/blog/guide-to-data-labeling

The Label Repository is responsible for storing these labels at an atomic level, which includes:

1. **Record Key** – Identifying what we are labeling.
2. **Label Type** – Determining the type of labeling.
3. **Label Value** – Storing the value of the label type for the given record key.
4. **Label Source** - Identifying whether it is auto-labeled (with metadata of the model) or human-labeled (with metadata of the individual who labeled it).
5. **Label Status** – Indicating whether it is at the final stage of labeling or not.
6. **Label Confidence** - Providing the level of confidence in the label.

### Functional Requirements:

1. **Create Labels**: The service should be able to create new labels with all required attributes such as label key, label type, label value, label source, label status, and confidence score.

2. **Read Labels**: The service should be able to retrieve labels based on various parameters such as record key, label type, label source, and status.

3. **Update Labels**: The service should be able to update existing labels. This could include changing the label value, status, or confidence score.

4. **Delete Labels**: The service should be able to delete labels based on the record key.

5. **Search Labels**: The service should provide a search functionality to find labels based on various parameters such as label type, label source, Confidence, and status.

6. **Label History**: The service should maintain a history of changes to a label, allowing users to track how a label has evolved over time.

### Non-Functional Requirements:

1. **Performance**: The service should be able to handle a large number of labels and perform CRUD operations quickly.

2. **Scalability**: The service should be able to scale to support growth in data volume and user traffic.

3. **Reliability**: The service should be highly reliable, with minimal downtime.

4. **Security**: The service should ensure that the data is securely stored and accessed. This includes implementing proper access controls and encryption.

5. **Data Integrity**: The service should ensure the accuracy and consistency of stored data.

6. **Usability and User Interface**: The service should be easy to use, with a well-designed API that makes it easy for developers to perform CRUD operations. Additionally, it should provide a user-friendly interface for manual labeling. Users should be able to query for labels under a certain confidence level and manually label them through this interface. The user interface should be intuitive, responsive, and accessible, allowing users to efficiently and effectively interact with the system.

7. **Dashboarding**: The service should provide a comprehensive dashboard that allows users to visualize the performance of the models over time. This dashboard should present key metrics and trends in an easy-to-understand format, enabling users to gain insights into model performance and make informed decisions. It should support various types of visualizations and allow users to customize the dashboard based on their specific needs.

7. **Maintainability**: The service should be easy to maintain, with clear documentation and well-structured code.

8. **Data Backup and Recovery**: The service should have mechanisms in place for backing up data and recovering it in case of a failure.

9. **Monitoring and Logging**: The service should have robust monitoring and logging capabilities to track its performance and troubleshoot issues.

# Proto

syntax = "proto3";


package telos.repository.label;

option java_package = "com.telos.repository.label";
option java_multiple_files = true;

import "telos/core/base/token/token.proto";
import "telos/core/record/record.proto";
import "telos/cortex/model/input.proto";
import "telos/cortex/model/output.proto";
import "telos/core/quantity/quantity.proto";
import 'google/protobuf/timestamp.proto';
import 'telos/core/base/status.proto';

message Label {
  telos.core.record.RecordKey label_key = 1;
  LabelType type = 2;
  LabelValue label_value = 3;
  LabelSource label_source = 4;
  LabelStatus status = 5;
  double confidence_score = 6;
  google.protobuf.Timestamp label_time = 10;
}

//message LabelKey {
//  oneof label_key_is_one_of {
//    telos.core.base.Token id = 1; // ex. Entity Instance Type/Instance
//    telos.core.record.RecordKey record_key = 2; // ex. Any feature
//  }
//}

// LabelType ontology is stored in the KG
// label_type is coming from a Label Collection in KG which can have all possible types
enum LabelType {
  LABEL_TYPE_UNKNOWN = 0;
  // image
  LABEL_TYPE_IMAGE_CLASSIFICATION = 1;
  LABEL_TYPE_IMAGE_BOUNDING_BOX = 2;
  LABEL_TYPE_IMAGE_SEGMENTATION = 3;
  LABEL_TYPE_IMAGE_SENTIMENT = 4;
  LABEL_TYPE_IMAGE_OBJECT_DETECTION = 5;
  LABEL_TYPE_IMAGE_OBJECT_TRACKING = 6;
  // text
  LABEL_TYPE_TEXT_CLASSIFICATION = 101;
  LABEL_TYPE_TEXT_SENTIMENT = 102;
  // entity
  LABEL_TYPE_NAMED_ENTITY_RECOGNITION = 200;
}

message LabelValue {
  telos.core.quantity.Quantity value = 1;
  // Label Should one_of ModelOutput: ModelClassificationOutput, ModelRegressionOutput?
}

message LabelSource {
  google.protobuf.Timestamp labeling_time = 1; // redundant??
  oneof label_source_is_one_of {
    ManualLabelSource manual = 2; // rename to HumanLabelSource?
    AutomatedLabelSource automatic = 3; // rename to MachineLabelSource?
  }
}

message ManualLabelSource {
  uint64 user_id = 1;
}

message AutomatedLabelSource {
  uint64 model_instance_id = 1;
}

enum LabelStatus {
  LABEL_STATUS_UNKNOWN = 0;
  LABEL_STATUS_START = 1;
  LABEL_TO_BE_REVIEWED = 2;
  LABEL_UNDER_REVIEW = 3;
  LABEL_REVIEWED = 4;
  LABEL_APPROVED = 5;
  LABEL_REJECTED = 6;
  LABEL_STATUS_COMPLETE = 7;
  // add more label statuses as needed
}

message CreateLabelsRequest {
  repeated Label labels = 1;
}

message CreateLabelsResponse {
  telos.core.base.Status status = 1;
  repeated Label labels = 2;
}

message UpdateLabelsRequest {
  telos.core.record.RecordKey record_key = 1;
}

message UpdateLabelsResponse {
  telos.core.base.Status status = 1;
  repeated Label labels = 2;
}

message GetLabelsRequest {
  telos.core.record.RecordKey record_key = 1; // What about getting the latest of all the labels or query by confidence score?
}

message GetLabelsResponse {
  telos.core.base.Status status = 1;
  repeated Label labels = 2;
}

message DeleteLabelsRequest{
  telos.core.record.RecordKey record_key = 1; // What about deleting the latest of all the labels or query by confidence score?labels or delete where (confidence score < x)?
}

message DeleteLabelsResponse {
  telos.core.base.Status status = 1;
}

# HLD


# Design decisions
### CAP Theorem: Consistency Over Availability

In the context of our Label Repository service, we prioritize consistency over availability. This decision is rooted in the potential business implications of dealing with inconsistent data.

Labels play a critical role in our system. They are used to train machine learning models, make predictions, and ultimately drive business decisions. **If the labels are inconsistent, it could lead to inaccurate model training and erroneous predictions**. This, in turn, could lead to poor business decisions and potential loss.

For instance, consider a scenario where we have a label that classifies a particular product. If due to a lack of consistency, the same product gets different labels, it could lead to confusion and misinterpretation. This could affect various downstream processes, including product recommendations, inventory management, and sales forecasting.

Therefore, ensuring the consistency of our labels is of utmost importance. While availability is also important, we believe that it is more crucial to ensure that every read receives the most recent write. Even if this approach might lead to the service being temporarily unavailable under certain rare conditions (like network partitions), we consider it a worthwhile trade-off to prevent the potential business loss that could result from inconsistent data.

This approach aligns with the principles of the CAP theorem, which states that it is impossible for a distributed data store to simultaneously provide more than two out of the following three guarantees: Consistency, Availability, and Partition tolerance. In our case, we choose to prioritize Consistency and Partition tolerance over Availability.

### Asynchronous Mode

Given the potential for dealing with large volumes of data in our Label Repository service, we have incorporated an asynchronous mode of operation. This mode is particularly useful when users need to create and upload large batches of labels.

In the asynchronous mode, **users can create upload jobs that are processed in the background**. This means that the user doesn't have to wait for the upload to complete and can continue with other tasks. The system handles the upload process, freeing up the user's time and resources.

Once an upload job is initiated, the system provides a job ID that can be used to track the progress of the upload. Users can check the status of their upload jobs at any time, providing them with flexibility and control over their tasks.

Upon completion of the upload job, a report is generated detailing the outcome of the operation. This report includes information such as the number of labels successfully uploaded, any errors encountered, and other relevant details. This allows users to quickly identify and resolve any issues with their data.

This asynchronous mode of operation enhances the scalability of our service, enabling it to efficiently handle large data volumes. It also improves the user experience by providing a non-blocking, responsive interface.

### Database

Given our data structure and the composite primary key requirement (`record_key` + `label_type` +`label_time`), here are a few databases that could be suitable:

1. **PostgreSQL**: PostgreSQL is a powerful, open-source object-relational database system. It has strong support for JSON fields, which we need for the `label_value` field. It also supports composite primary keys. PostgreSQL is known for its robustness and its compliance with SQL standards.

2. ~~**MySQL**: MySQL is another popular open-source relational database management system. It also supports JSON fields and composite primary keys. However, its JSON support is not as advanced as PostgreSQL's.~~

3. ~~**Cassandra**: Apache Cassandra is a highly scalable, high-performance distributed database designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. It supports composite primary keys and has some support for JSON, but it's not as comprehensive as PostgreSQL or MySQL.~~

4. **MongoDB**: MongoDB is a source-available cross-platform document-oriented database program. It's classified as a NoSQL database program, and it uses JSON-like documents with optional schemas. However, MongoDB doesn't natively support composite primary keys. We would need to create a compound index on `record_key` and `label_time` fields.

5. **Couchbase**: Couchbase Server is a NoSQL document database with a distributed architecture for performance, scalability, and availability. It enables developers to build applications easier and faster by leveraging the power of SQL with the flexibility of JSON.

Given our data structure, here's how you could define a PostgreSQL table to store label data:

CREATE TABLE labels (
    record_key text, -- type_id + uint64
    label_type text,
    label_value jsonb,
    label_source text,
    status text,
    confidence_score numeric,
    label_time timestamp,
    PRIMARY KEY (record_key, label_type, label_time)
);

# Useful Links
[How To Set Up An ML Data Labeling System](https://towardsdatascience.com/how-to-set-up-an-ml-data-labeling-system-4eea9b15181f)
[TelOS Repository Master Presentation By Praveen](https://rilcloud-my.sharepoint.com/:p:/r/personal/vankayalapati_p_ril_com/_layouts/15/Doc.aspx?sourcedoc=%7B6885D71A-AF3E-48BE-B8C7-45AF30DE4785%7D&file=TelOS%20Repositories.pptx&action=edit&mobileredirect=true&DefaultItemOpen=1&login_hint=Krishna.Tiwari%40ril.com&ct=1687770118144&wdOrigin=OFFICECOM-WEB.START.EDGEWORTH&cid=bee14274-fc8d-4796-8b3b-fa346c45cea6&wdPreviousSessionSrc=HarmonyWeb&wdPreviousSession=07fa0c00-a0e9-4d99-a6a4-1761773749c9)
[Annotation Service PRD by Praveen](https://rilcloud.sharepoint.com/:w:/r/sites/TelsOs/_layouts/15/Doc.aspx?sourcedoc=%7B09C4CEB1-0D28-45F9-B5C0-2A2959233C8A%7D&file=Annotation%20Service%20PRD.docx&action=default&mobileredirect=true)

# Team
Product : Praveen
Tech Team : Supriyo, Shrimant, Krishna