---
# System prepended metadata

title: Recommendation system data pipeline

---

---
title: Recommendation system data pipeline
tag: KHTN khtn recommendation system
description:
---
Recommendation system: data pipeline
===

## features - data mapping
- User profile
    - User Age:  Profile.age
    - User Location: User.country
    - User Type: Profile.type
    - Gender: Profile.gender
- Video profile
    - Genre: VideoV2.genre
    - Video category: VideoV2.categories
    - Video duration: VideoV2.duration
    - Video total like: VideoV2.totalLike
    - Video total views: VideoV2.totalView
    - Playlist id: PlaylistItemV2.playlistID
- Watching behaviors
    - like
    - valid ping counts: Video_Load_snapshot.valid_pings_count

## Preprocessing steps
- User
    - Select users watched at least 10 (5 in code) videos
- Profile
    - _isDeleted != True
- VideoV2
    - _isDeleted != True and adminBlocked != 1
    - categories.replace(r"[\['\]]", "", regex=True)
- PlaylistItemV2
    - _isDeleted != True and adminBlocked != 1

## MongoDB
### Provided collections
- VideoV2
- User
- Profile
- Genre
- CategoryV2
- MPAA
- PlaylistItemV2
### Pubsub

