# Click Overmind overview - Arbitrage handling
## I. Generate Arbitrage feed
```plantuml
@startuml
autonumber
actor "Arbitrage Start Trigger" as arb_trigger
participant "Traffic Acquisition System" as tas
participant "Tracing System" as ts
database "AWS S3" as s3
arb_trigger -> tas ++: Trigger Arbitrage feed generation
tas -> s3: Read Job Advert parquets placed by Job Feed Processing
tas -> tas: ...
tas -> ts: Create new feed level business process
loop for each job
tas -> ts: Create new job level business process (tfj1)
note left
See example below
endnote
end loop
tas -> tas: ...
tas -> s3 --: Write latest feed XML
@enduml
```
### Tracing System record example for a `tfj1`:
```yml
global_trace_id: {"uuid": 24070a67-1642-40c1-aee7-94585f9595ab, "schema_id": 'tfj1'}
source: traffic-acquisition-system
local_trace_id: 14c4facc-a6c9-4325-a433-05b874ba4025
created_at: 2022-11-25 11:06:43.782+0000
global_state:
job_pool_specific_description_based_id: 60468a5e37aa2328eef5d1763c13bd88eb87cae005724cb80ed6be3adbcf81a7
outgoing_cpc: 0.3499999940395355
feed_cpc: 0.25
# incoming_cpc has been renamed to feed_cpc
business_strategy: instant_monetization
expiry_check: 0
silent_change: 0
job_unique_id: 10e2ff380137406cb2a6af867c180acf
publisher_id (nullable): instant_monetization
campaign_id (nullable): instant_monetization
local_state: NULL
parent_global_trace_id: {"uuid": 4aaa11e2-682e-41f2-be96-f3b95a7d96c4, "schema_id": 'tff1'}
```
## II. Job seeker clicks
```plantuml
@startuml
autonumber
actor "Job Seeker" as js
participant "Click Overmind" as co
participant "Tracing System" as ts
participant "Central Job Advert Store" as cjas
queue incoming_clicks as incq
queue outgoing_clicks as ocq
js -> co ++: click
co -> co: Pick/generate user ID
co -> ts ++: Join job level business process (tfj1)
ts --> co --: tfj1 state data
co -> ts ++: Create new click level business process (jc1)
ts --> co --: new click level global trace id ("click_id")
co -> cjas ++: Get job by job_pool_specific_description_based_id
cjas --> co --: Current Job Advert (job_unique_id may be different)
co -> co: Merge data - apply hierarchy of sources
co -> co: Detect anomalies
co -> co: Apply URL tags (e.g. conversion tracker, etc/)
co --> js: Redirect
opt success creating click level business process
co -> incq: incoming click msg
end opt
opt redirected to partner
co -> ocq: outgoing click msg
end opt
@enduml
```
### Request data
* URL path param
* job level global trace id (e.g. `24070a67164240c1aee794585f9595abtfj1`)
* Supported URL query params (others are ignored) - **these override state data from the Tracing System**
* campaign_id - cpc_cid
* publisher_id - cpc_pid
* incoming_cpc
* utm_campaign (defaults to `utm_campaign` if not provided)
* utm_source
* utm_medium
* utm_content
* utm_term
* Read cookies
* lensa_guest_id
* (planned) lensa_access_token (with Job Seeker ID)
* (+ other client info eg. ip, user agent)
#### example url
```
https://lensa.com/cgw/inc/24070a67164240c1aee794585f9595abtfj1
?utm_source=Lensa_ARB_test
&utm_medium=cpc
&utm_campaign=ARB_internal_test
&incoming_cpc=0
&cpc_cid=001
&cpc_pid=007
&cpc-jid=10e2ff380137406cb2a6af867c180acf
&source=pandologic_feed
&utm_content=24070a67164240c1aee794585f9595abtfj1
```
### Job Advert (simplified) - received from Central Job Advert Store
```yml
unique_id: 13dcb1ee6200498380454e4163796ffd
remote_id: 551485348
job_pool_slug_id: pandologic_feed
price_cpc: ??? # Not used
is_expired: false
partner_url: https://practicelinkltd.thejobnetwork.com/Job/551485348?etd=LAFFER2UEHPRBIJH7HQABA6754LROKXW5A2I3IMYKZ2YNGXGMJNNGZAHEFPN3W5Q6BLLLQUWJN7KVTBSW72ARR57LRLRWNADMN3N4ZK7MPKQYEEWXBNKLJDGJURCMNB43ABSZAXDVMZRU3JHNO3DXMMVOA%3d%3d%3d%3d%3d%3d
classified_title:
cleaned_title: Physician - Pain Medicine
location:
city: Seymour
state: IN
display_name: Seymour, IN
company:
normalized_name: Schneck Medical Center
hash_id: 6a61be4cf3d4fcce6b7c4e55b20274449abe2978
```
### Produced incoming_clicks message
```json
// Headers
{
"event_type": "click_entered",
"is_anomaly": "true"
}
// Content
{
"anomalies": [
"duplicate"
],
"utm_tags": {
"utm_source": "Lensa_ARB_test",
"utm_medium": "cpc",
"utm_campaign": "ARB_internal_test",
"utm_term": null,
"utm_content": "24070a67164240c1aee794585f9595abtfj1"
},
"incoming_cpc": 0, // Overridden in URL query param
"publisher_id": "007", // Overridden in URL query param
"campaign_id": "001", // Overridden in URL query param
"global_trace_id": "328d175ed2294aa083f825c6a0104fa3jc1",
"job_level_business_process_name": "traffic_flow_job",
"job_pool_specific_description_based_id": "60468a5e37aa2328eef5d1763c13bd88eb87cae005724cb80ed6be3adbcf81a7",
// current job_unique_id from Central Job Advert Store
"job_unique_id": "13dcb1ee6200498380454e4163796ffd",
"cleaned_title": "Physician - Pain Medicine",
"job_pool_slug_id": "pandologic_feed",
"job_remote_id": "551485348",
"city": "Seymour",
"state": "IN",
"company_normalized_name": "Schneck Medical Center",
"company_hash": "6a61be4cf3d4fcce6b7c4e55b20274449abe2978",
"job_url": "https://practicelinkltd.thejobnetwork.com/Job/551485348?etd=LAFFER2UEHPRBIJH7HQABA6754LROKXW5A2I3IMYKZ2YNGXGMJNNGZAHEFPN3W5Q6BLLLQUWJN7KVTBSW72ARR57LRLRWNADMN3N4ZK7MPKQYEEWXBNKLJDGJURCMNB43ABSZAXDVMZRU3JHNO3DXMMVOA%3d%3d%3d%3d%3d%3d",
"is_expired": false,
"ip_address": "54.208.108.236",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.1 Safari/605.1.15",
"guest_id": "4d45cacd-69d0-4e70-9e7a-1f3b0f0cd344",
"outgoing_cpc": 0.3499999940395355,
"clicked_at": 1669630502,
"job_level_global_trace_id": "24070a67164240c1aee794585f9595abtfj1"
}
```
### Produced outgoing_clicks message
```json
// Headers
{
"event_type": "click_redirected",
"is_anomaly": "true"
}
// Content
{
"anomalies": [
"duplicate"
],
"utm_tags": {
"utm_source": "Lensa_ARB_test",
"utm_medium": "cpc",
"utm_campaign": "ARB_internal_test",
"utm_term": null,
"utm_content": "24070a67164240c1aee794585f9595abtfj1"
},
"outgoing_cpc": 0.3499999940395355,
"global_trace_id": "328d175ed2294aa083f825c6a0104fa3jc1",
"job_level_business_process_name": "traffic_flow_job",
"job_pool_specific_description_based_id": "60468a5e37aa2328eef5d1763c13bd88eb87cae005724cb80ed6be3adbcf81a7",
"job_unique_id": "13dcb1ee6200498380454e4163796ffd",
"cleaned_title": "Physician - Pain Medicine",
"job_pool_slug_id": "pandologic_feed",
"job_remote_id": "551485348",
"city": "Seymour",
"state": "IN",
"company_normalized_name": "Schneck Medical Center",
"company_hash": "6a61be4cf3d4fcce6b7c4e55b20274449abe2978",
"job_url": "https://practicelinkltd.thejobnetwork.com/Job/551485348?etd=LAFFER2UEHPRBIJH7HQABA6754LROKXW5A2I3IMYKZ2YNGXGMJNNGZAHEFPN3W5Q6BLLLQUWJN7KVTBSW72ARR57LRLRWNADMN3N4ZK7MPKQYEEWXBNKLJDGJURCMNB43ABSZAXDVMZRU3JHNO3DXMMVOA%3d%3d%3d%3d%3d%3d",
"ip_address": "54.208.108.236",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.1 Safari/605.1.15",
"guest_id": "4d45cacd-69d0-4e70-9e7a-1f3b0f0cd344",
"job_seeker_id": null,
"clicked_at": 1669630502,
"is_incoming": true
}
```
## III/A. Consume incoming_clicks for legacy system support
```plantuml
@startuml
autonumber
actor "AWS Lambda" as lambda
queue incoming_clicks as incq
participant "Legacy Click Consumers" as lcc
database incoming_click
database jobalerts
lambda -> incq: poll
lambda -> lcc ++: incoming_clicks msg
lcc -> lcc: Transform for incoming_click db
note left
See below
end note
lcc -> incoming_click: Insert incoming_click
opt expired or spike
lcc -> lcc: Transform for jobalerts db
note left
We assume there is no outgoing_clicks msg,
so this consumer must insert a jobstop-type
record into jobalerts.click_tracking.
Transforms and inserts are the same as what
is described for the outgoing_clicks consumer
further below.
endnote
lcc -> jobalerts: Insert click_tracking
lcc -> jobalerts: Insert click_tracking_incoming_click
lcc -> jobalerts: Insert click_tracking_id_trace_id_mapping
end opt
@enduml
```
### Inserted incoming_click.incoming_click
Fields with non-obvious content:
* unique_id = job_unique_id
* click_source = `jobsense` or `jobsense.widget` if we recognize the business process name, else job_level_business_process_name
* global_tracing_id = job_level_global_trace_id
* incoming_click_id = global_trace_id (click-level)
* recommendation_time = NULL (constant)
```yml
id: 244110
unique_id: 13dcb1ee6200498380454e4163796ffd
job_pool_specific_description_based_id: 60468a5e37aa2328eef5d1763c13bd88eb87cae005724cb80ed6be3adbcf81a7
global_tracing_id: 24070a67164240c1aee794585f9595abtfj1
publisher_id: 007
campaign_id: 001
click_source: traffic_flow_job
incoming_price: 0.0
outgoing_price: 0.35
is_expired: 0
cleaned_title: Physician - Pain Medicine
company_normalized_name: Schneck Medical Center
company_hash: 6a61be4cf3d4fcce6b7c4e55b20274449abe2978
city: Seymour
state: IN
ip_address: 54.208.108.236
user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.1 Safari/605.1.15
utm_tags: '{"utm_campaign": "ARB_internal_test", "utm_source": "Lensa_ARB_test", "utm_medium": "cpc", "utm_content":"24070a67164240c1aee794585f9595abtfj1", "utm_term": null}'
recommendation_time: null
incoming_click_time: 2022-11-28 10:15:02.0
incoming_click_id: 328d175ed2294aa083f825c6a0104fa3jc1
utm_source: Lensa_ARB_test
utm_medium: cpc
utm_content: 24070a67164240c1aee794585f9595abtfj1
utm_campaign: ARB_internal_test
utm_term: null
cid: 4d45cacd-69d0-4e70-9e7a-1f3b0f0cd344
```
## III/B. Consume outgoing_clicks for legacy system support
```plantuml
@startuml
autonumber
actor "AWS Lambda" as lambda
queue outgoing_clicks as ocq
participant "Legacy Click Consumers" as lcc
database jobalerts
lambda -> ocq: poll
lambda -> lcc ++: outgoing_clicks msg
lcc -> lcc: Transform for jobalerts db
note left
See below
endnote
lcc -> jobalerts: Insert click_tracking
lcc -> jobalerts: Insert click_tracking_incoming_click
lcc -> jobalerts: Insert click_tracking_id_trace_id_mapping
@enduml
```
#### outgoing_clicks msg transformations for inserting to jobalerts.click_tracking
* cid = job_seeker_id else guest_id
* utm_content =
* `spike_stopped` if spike anomaly
* `expired` if job expired
* else the actual utm_content
* type =
* `job_sense` if job_level_business_process_name is a known JobSense
* `jobstop` if spike anomaly or job expired
* else `onsite`
* Constant NULLs:
* user_src
* client_id
* jobtrack_id
* title_similarity
* geo_similarity
### Inserted jobalerts.click_tracking_id_trace_id_mapping
```yml
id: 2
click_tracking_id: 312400275
global_trace_id: 328d175ed2294aa083f825c6a0104fa3jc1
```
### Inserted jobalerts.click_tracking_incoming_click
```yml
click_tracking_id: 312400275
incoming_click_id: 328d175ed2294aa083f825c6a0104fa3jc1
```
### Inserted jobalerts.click_tracking
```yml
id: 312400275
type: onsite
src: pandologic_feed
cid: 4d45cacd-69d0-4e70-9e7a-1f3b0f0cd344
user_src: null
client_id: null
ip: 54.208.108.236
useragent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.1 Safari/605.1.15
url: https://practicelinkltd.thejobnetwork.com/Job/551485348?etd=LAFFER2UEHPRBIJH7HQABA6754LROKXW5A2I3IMYKZ2YNGXGMJNNGZAHEFPN3W5Q6BLLLQUWJN7KVTBSW72ARR57LRLRWNADMN3N4ZK7MPKQYEEWXBNKLJDGJURCMNB43ABSZAXDVMZRU3JHNO3DXMMVOA%3d%3d%3d%3d%3d%3d
jobtrack_id: null
position: Physician - Pain Medicine
location_name: Seymour, IN
jobadvert_id: pandologic_feed_551485348
created_at: 2022-11-28 10:15:02.0
utm_source: Lensa_ARB_test
utm_medium: cpc
utm_campaign: ARB_internal_test
cpc: 0.35
title_similarity: null
geo_similarity: null
utm_content: 24070a67164240c1aee794585f9595abtfj1
utm_term: null
```
## IV. Track conversion
```plantuml
@startuml
autonumber
actor Partner
participant "Conversion Tracker" as ct
database jobalerts
queue converted_clicks as ccq
Partner -> ct ++: HTTP: click level global trace id in request or cookie
ct -> jobalerts: Read click_tracking_id_trace_id_mapping
ct -> jobalerts: Insert partner_application_tracking
ct -> ccq --: Send msg
@enduml
```
### Inserted jobalerts.partner_application_tracking
```yml
id: 19872194
click_tracking_id: 312400275
created_at: 2022-11-28 14:57:10
```
### Produced converted_clicks message
```json
// Headers
{}
// Content
{
"job_click_global_trace_id": "d3569ebd080a47f6b77d3fc0d4be1d73jc1",
"created_at": 1669647430
}
```
Sign in with Wallet
Connect another wallet