# CodeZero-ML ReadMe

## Notations:

- Raw Data:
- Orange
- Internal Data:
- Yellow
- Result/Output:
- Orange
- Operation:
- Red
- Pseudo data:
- white

## Preprocessing:

### Pipeline:

### Input Data:
- post text:

    <post-id>\t<post-level>\t<text>

- post key text:

    <post-id>\t<text>

- new words:

    <word>\space<frequency>

- dictionary:

    <word>\space<frequency>

- hand-crafted words:

    <word>\space<defined-frequency>

- blacklist words:

    <word-to-remove>

### Output data:

- segmented post key text:

    <post-id>\t<segmented-text>

### Program:
- Parsing:
- Merge topic and subtopic into post-key-text
- New Word detection:
- input:
- text file and separate **characters** with space.
- usage:
- modify "word_discovery.py" on the "corpus_file" argument.
- Word Segmentation:
- usage:
- input:
- 1. post key text
- 2. dictionary
- segmented in batch (to avoid OOM)

## Keyword Extraction:

### Pipeline:

### Input Data:
- segmented post key text:

    <post-id>\t<text>

- segmented post key text(jsonl):

    <metadata to index>

- segmented post key text index:

    <binary index file>

- candidate keywords:

    <keyword>

- hand-crafted keywords:

    <keyword>

### Output data:
- keywords for each post:

    {
        <docid>: 
            {<keywords>: <score>}
    }

### Program:
- Convert to jsonl:
- convert to jsonl for the indexing program to index
- Indexing
- index into binary file to calcualte the BM25 score
- Candidate Word Retrieval:
- Generate candidate keywords by POS-tagging and BM25 score
- Keyword Extraction and score calculation:
- Extract keywords from each post by candidate keywords and the BM25 score.

### TODO: Word-to-word relation:

### TODO: Post-to-Post Recommendation (Inverted Index):


### Input data:
- keywords for each post:

    {
        <docid>: 
            [
                {<keyword_1>: <score>},
                {<keyword_2>: <score>}, ...
            ]
    }

- keyword-to-post:

    {
        <keyword>: [docid_1, docid_2, ...]
    }

- keyword-to-keyword:

    {
        <keyword>: 
        [
            {<keyword_1>: <score>}, 
            {<keyword_2>, <score>}, ...
        ]
    }

## TODO:
run-script sample:
Ex:
- python segment.py --arguments
- sample file
- 3 ~ 4 lines