# Tagging system for private applications
## Objetive
The idea is that private applications can have a tagging system.
For example an application can be tagged as "for interview" "b" and other as "interviewed" "c"
In this case "for interview" and "interviewed" show the command(process) and "b" and "c" the priority
The command actions can be reported to candidates so that they can know in which process they are.
The priority tags are only useful to recruiters. Priority can be used for sort purpose.
* **is_hidden** is an independent tag which purpose is to hide an application. There is also an option to hide all applications received from a specific candidate.
* Is hidden the same as archived application?(archived applications would those that the recruiter did an interview but the candidate was not good enough for that process)
* Since the interactions between active and hidden applications are different, I am working also in a tab approach in order make a explicit difference

## Possible commands tags:
* For interview
* Interviewed
* Need more info
* Pending (this tag is for all the new applications)
* Hidden (independent tag -> is_hidden)
## UI/UX
There is going to be a filter widget in which recruiters can select one or more commands querys and another for priority.
Also recruiters could sort the results for priority.

Also each card has to have options for the tagging part.
Commands and priority have to be independent although are saved in the same multiValued field in solr

# Indexing problem:
All changes should be indexed in NRT so that the recruiter can make filters.
Currenlty all the applicaciones are fetched from solr.
The current configuration of solr does not index changes in NRT but in 5 min or more.
The last fact makes the current solr not suited for the purpose of the tagging system.
Moreover, an open jobad keeps reciving applications and the index process of these new applications has also a delay time.
Currently we are fetching a new specific application from mongodb and combine it with the solr results of the already indexed applications.
## possible solution with solr
One solution is to change the current config of to support NRT.
With a softCommit a change is going to be only in-memory. The slower process to flush all of changes to disk is related with the hardCommit that can be in 5 min for example.
Because all the data is going to be in-memory several parameters have to be tunned such as the document cache, the query result cache, the filter cache, the merge policing, the segment merge process.
Nevertheless we need to wait at least 1 second.
This change has to be made at the level of the collection.
Softcommits can be very heavy if the index is queried while indexing(atelo runs very often).

Another solution would be that all the individual updates(tags) made in these private applications could go direct to solr (also to mongodb but without a reindex request) with the `commit` method of the SolrClient:
* `public UpdateResponse commit(String collection, boolean waitFlush, boolean waitSearcher, boolean softCommit)`
`waitFlush` and `waitSearch` would be false and softCommit true.

Again for this approach we need more cache and more RAM but for defined period of time (while the recruiters are working in a `hiring process view`).
Multiple recruiters concurrently would submit softCommits, therefore we need to test what would be the min RAM available to support softCommits before a hardCommit starts.

A better approach would to send a softCommit not one by one but in a batch fashion. For this we need to understand how the recruiters would use the new page.
Maybe at the beginning they just mark with tags all the applications.
Then they would go to a filter view (UI) based in a specific tag.
Therefore we can simulate a `working loading page` of 1-2 seconds that would be enought to softCommit a batch of updates.
### solr with child documents:
A possible desing is that the parent document is the Applicaction and the child documents would have a unique id and a multivalue field for the tags.
child documents are related with the parent document. Any update in the child means a update in the parent an other child documents which also mean a reindex of the entire document.
There is a support for in-place updates for child documents but this in-place updates work for no-index no-stored numeric fields.
#### Ref:
* https://doc.lucidworks.com/fusion-server/5.1/reference/solr-reference-guide/7.5.0/updating-parts-of-documents.html
* https://stackoverflow.com/questions/22395683/how-to-write-nested-schema-xml-in-solr

### Ref
* https://books.google.com/books?id=othpBgAAQBAJ&pg=PA34&lpg=PA34&dq=%22%3CmaxTime%3E1000%3C/maxTime%3E%22&source=bl&ots=gKP1dwEn2s&sig=ACfU3U22inZ0RSX4tabb3EqZPsyQY7ehkg&hl=en&sa=X&ved=2ahUKEwjputb4vazpAhVCI6wKHYoTBJgQ6AEwAHoECAcQAQ#v=onepage&q&f=false
* https://www.aeologic.com/blog/ultimate-solr-guide-2-near-real-time-search/
* https://stackoverflow.com/questions/50597942/achieve-soft-commit-programmatically (NRT at config API level, not that recommended)
* https://community.cloudera.com/t5/Community-Articles/Ambari-Infra-Solr-Ambari-2-x-setup-recommendations-with/ta-p/247575
* https://cwiki.apache.org/confluence/display/SOLR/NearRealtimeSearchTuning

## hbase/redis
pagination and filters are not well supported (TODO needs more research)
### Pagination in redis:
https://christophermcdowell.dev/post/pagination-with-redis/
under the hypothesis that recruiters do not have more than 100 applications at the same time. We could cache all those applications in redis and make the changes in this temporal cache view. Then that view will be sync with mongodb and then solr.
When this view would be generated?
How new applications would be added to this view?
## mongodb
mongodb can handle text search in small collections. Likewise filters.
If we define a clear separation between the database-candidate-search page and the new master-detail page (`hiring process view`), I think we can rely on mongodb for updates and filters.
Another important point is that a `hiring process view` should be related with a defined job_ad_id.
In that `hiring process view` a recruiter would be interested in all the received applications, applications from other jobs and applications from the database-candidate-search related with the current jobad (category?).
I guess the number of documents in the previous defined `hiring process view` would only be around 1000 or maybe 10000 max.
In the database-candidate-search page (which comes from solr) a recruiter could mark for which `hiring process view` a application/s best fit.
This lead to another question, how the recruiters search in the database-candidate-search page? They would go to this page without any purpose (as in twitter in a infitive scrolling) or they would go to this page with a objetive in mind (find a candidate for a specific jobad - `hiring process view`)?

In mongo we would need to create a index based in the recruiter_id and jobad_id.
And also another collection in which are going to be the recruiter_id, the job_id and all the applications_ids.
This collection can also be created in solr as a child document.
### Ref:
* http://pepa.holla.cz/wp-content/uploads/2016/07/MongoDB-in-Action-2nd-Edition.pdf
# Random thoughts
One aspect that we have to think is that is not the same filter in 4M of documents that in 10K documents.
A recruiter usually only handles no more that 10K documents.
Another point is that is not the same search in the candidate database that starts the hiring process.
The hiring process may be related with a particular job_ad. Therefore, a recruiter would like to see in the "hiring" mode view of that particular job_ad all recived applications and all other unblocked applications related somehow with that jobad.
In the candidate database search the recruiter could mark for which hiring process (job_ad) a that particular candidate/application is best suited.
In this special and constrain view we could reach a NRT update.
Another point is that solr is suited to rank results based on random input text pretty much as the current empleos page but the filters in the contraing view of the recruiter are fixed such as experience, student_level ...
How much time would last a hiring process?
# random ref
* https://tech.olx.com/improving-solr-performance-f4202d28b72d: quite interesting the process of how they improve solr performance. There is also a part about NRT.