# Ontology ###### tags: `whilter` ## To Do 1. Close down on tag list with Prashsant for clips - component vs composition ## Ontology ### Clip Composition Tags * camera_pan_movement * static * left_to_right * right_to_left * top_to_bottom * bottom_to_up * diagnol_to_right_bottom * diagnol_to_left_bottom * diagnol_to_right_top * diagnol_to_left_bottom * camera_zoom_movement * static * zoom_in * zoom_out * camera_rotation * static * clockwise * anticlockwise * shot_type * close-up * wide shot * shot_location * studio * green screen * outdoor * indoor * resolution * 7680p - (8K): 7680x4320 * 2160p - (4K): 3840x2160 * 1440p - 2560x1440 * 1080p - 1920x1080 * 720p - 1280x720 * 480p - 854x480 * 360p - 640x360 * 240p - 426x240 * aspect_ratio * 16:10 * 9:16 * 3:2 * 4:3 * 5:4 * preferred_shot_position * first * second * last * last_but_one * middle * anywhere * transition - https://invideo.io/blog/easy-online-video-transition-effects/ * Standard Cut * Jump Cut * L-cuts & J-cuts * Cross Dissolve * Wipe * Film Roll-in * Cross Zoom * Light Leaks & Glitch * Pan * Whip * Object Block * Breaker Clips * Timelapse ### Content Tags (Clips) * People * Toddlers (Aged 0-3) * Kids (Aged 4-7) * Childern (Aged 8-12) * Teens (Aged 13-19) * Young (Aged 20-29) * Adults (Aged 30-60) * Seniors (Aged Above 60) * One male * One female * Two males * Two females * Couple * Family * Crowd * People with Professions * Devotees * Doctors * Nurses * Priests * Students * IT people * Police * Lawyer * Teacher * Tourists * Body Builder * People with Activities * Dancing * Drinking * Driving * Helping * Holding * Riding * Sitting * Standing * Talking * Trekking * Typing * Walking * Watching * Writing * Yoga * Excercise * Body Building * Aerial * Buildings * Cities * Energy * Heritage * Mountains * Nature * Roads * Rural * Animals * Birds * Camels * Cats * Cattle * Dogs * Lion * Monkey * Eagles * Elephants * Exotic * Fish * Insects * Pigeons * Pigs * Architecture * Apartments * Bridges * Flyovers * Government Buildings * Heritage * Houses * Lighthouses * Mansions * Modern * Offices * Old Buildings * Religious Buildings * Rural * Statues * Art and Design * Articles * Furniture * Galleries * Paintings * Sculptures * Cities * Delhi * Mumbai * Kolkota * Chennai * Bangalore * Hyderabad * Ahemdabad * Chandigarh * Jaipur * Bhopal * City Life * Buildings * Lights * Markets * Morning * Night * Night life * North * Parks & Gardens * Shops * Street Food * Traffic * Water * Movie Hall * Theater * Shopping Mall * Offices * School * Colleges * Airport * Train Station * Metro * Swimming Pool * Consumer Electronics * Mobiles * TV * Refrigerator * Laptop * AC * Energy * Wind * Dam * Nuclear * Solar * Electricity * Dance * Bharatanatyam * Kathakali * Kathak * Kuchipudi * Odissi * Sattriya * Manipuri * Mohiniyattam * Salsa * Group Dance * Zumba * Regional Other * Fashion * Jewellery * Western * Formals * Traditional * Saree * Jeans * T-Shirt * Festivities * Carnivals * Christmas * Holi * Diwali * Ganesh Chaturthi * Guru purab * Eid * Weddings * Birthdays * Food & Drinks * Bakery * North Indian * South Indian * Itallian * Chinese * Coffee * Fast food * Food Courts * Home Cook * Juices * Cocktails * Local Drinks * North Eastern * Restaurants * Snacks * Spices * Street Food * Tea * Healthcare * Clinics * Doctors * Emergency Care * Hospital Staff * Hospitals * Medical Education * Nurses * Others * Money & Finance * Banks * Currency Notes * Coins * Chequeu * Credit Cards & Machine * Musical Instruments * Keyboards * Guitar * Drums * Piano * Shehnai * Nature * Beaches * Caves * Deserts * Farms * Flowers * Forests * Grass * Himalayas * Morning * Mountains * Night * Others * Rain * Rivers and Lakes * Seas * Skies & Clouds * Snow * Space & Astronomy * Spring * Sunrise/sunset * Trees * Valleys * Waterfalls * Winter * Religion and Spirituality * Buddhism * Christianity * Hinduism * Islam * Industrial * Brewery * Fishery * Heavy Engineering * Process Engineering * Pharmaceutical * Assembly Line * Warehouse * Lathe Machine * Foundary * Furnance * Sports * Cricket * Soccer * Badminton * Hockey * Table Tenis * Gymnastic * Swimming * Chess * Rubic Cube * Carom Board * Transportation * Auto-Rickshaws * Buses * Cars * Cycles * Earth-moving Machine * Motorbikes * Others * Overtaking * Pickup Trucks * Traffic * Trains * Trucks * Ship * Boats * Aeroplance * Vacations * Backpacking * Hotels * Resorts & Retreats * Solo * Tourism Industry * Hotel rooms * Uncategorized ### Other * assets_type * public-clip (lifecycle stages) * Initial upload * Auto tagged * Lower resolution created * Water marked * Manual tagging in progress (content and composition tags) * Tagging complete * Published * In active (Not visible in search) * Retired * private-clip (lifecycle stages) * Initial upload * Lower resolution created * Ready to use * In active * template (e.g. advertisement) - (lifecycle stages) * Init * Auto annotated * Manual annotation in progress * Annotation complete * Published * In active (Not visible in search) * template-instance (lifecycle stages) * Init * Auto complete (Machine assigned clip) * Edit in progress * Saved * Published * In active (Not visible in search) * finished_asset * template_type * jewellery advertisement * resturant advertisement * saloon advertisement * gym advertisement * marriage summary 3 min * marriage summary 5 min * birthday invitation ## Examples ### Public clip ```json= { "assets_type" : "pubilc-clip", "video-url" : "http://s3.amazonaws.com/whilter-components/insta-reels/aspiration/public-clip-1.mpeg", "composition_tags" : { "camera_pan_movement" : "left_to_right", "camera_zoom_movement" : "static", "camera_rotation" : "clockwise", "shot_type" : "close-up", "shoot_location" : "indoor" }, "content_tags" : [ { "People" : ["Adults (Aged 30-60)", "Two males"], "People with Activities" : ["Dancing"] "automated_tags" : ["people, dancing"] } ], "preferred_shot_position" : "start", "resolution" : "2160p", "aspect_ratio" : "16:9", "length" : 5.23, "important-time-stamps" : [1.3, 4.5], "life-cycle-stage" : "published", "created-by" : "userid-1", "created-date" : "2022-10-30*02:47:33:899", "updated-by" : "userid-2", "updated-by" : "2022-10-30*02:47:33:899", "version" : 1 } ``` ### Private clip ```json= { "assets_type" : "private-clip", "video-url" : "http://s3.amazonaws.com/whilter-components/insta-reels/aspiration/private-clip-1.mpeg", "resolution" : "2160p", "aspect_ratio" : "16:9", "length" : 7.94, "important-time-stamps" : [3.2, 9.5], "life-cycle-stage" : "published", "created-by" : "userid-1", "created-date" : "2022-10-30*02:47:33:899", "updated-by" : "userid-2", "updated-by" : "2022-10-30*02:47:33:899", "version" : 1 } ``` ### Private image ```json= { "assets_type" : "private-image", "image-url" : "http://s3.amazonaws.com/whilter-components/insta-reels/aspiration/private-image-1.mpeg", "resolution" : "2160p", "bounding-box" : [0, 0, 720, 1080], "life-cycle-stage" : "published", "created-by" : "userid-1", "created-date" : "2022-10-30*02:47:33:899", "updated-by" : "userid-2", "updated-by" : "2022-10-30*02:47:33:899", "version" : 1 } ``` ## Template * Sum of duration of scores (end_time - start_time) need equal to duration of template * Sum of duration of clips (end_time - start_time) need equal to duration of template ```json= { "assets_type" : "pubilc-clip", "base-video-url" : "http://s3.amazonaws.com/whilter-components/insta-reels/aspiration/public-clip-1.mpeg", "duration": 45, "background-score" : [ { "sequence" : 1, "score-url" : "http://s3.amazonaws.com/whilter-music/library/score-1.mp3", "start-time" : 0.31, "end-time" : 19.39 }, { "sequence" : 2, "score-url" : "http://s3.amazonaws.com/whilter-music/library/score-2.mp3", "start-time" : 1.53, "end-time" : 25.09 } ], "components" : [ { "slot-id" : 1, "type" : "video", "composition_tags" : { "camera_pan_movement" : "left_to_right", "camera_zoom_movement" : "static", "camera_rotation" : "clockwise", "shot_type" : "close-up", "shoot_location" : "indoor" }, "content_tags" : [ { "People" : ["Adults (Aged 30-60)", "Two males"], "People with Activities" : ["Dancing"] "automated_tags" : ["people, dancing"] } ], "duration" : 12, "transition" : "standard-cut" }, { "slot-id" : 2, "type" : "image", "duration" : 4 }, { "slot-id" : 3, "type" : "video", "composition_tags" : { "camera_pan_movement" : "left_to_right", "camera_zoom_movement" : "static", "camera_rotation" : "clockwise", "shot_type" : "close-up", "shoot_location" : "indoor" }, "content_tags" : [ { "People" : ["Adults (Aged 30-60)", "Two males"], "People with Activities" : ["Dancing"] "automated_tags" : ["people, dancing"] } ], "duration" : 20, "transition" : "standard-cut" }, { "slot-id" : 4, "type" : "video", "composition_tags" : { "camera_pan_movement" : "left_to_right", "camera_zoom_movement" : "static", "camera_rotation" : "clockwise", "shot_type" : "close-up", "shoot_location" : "indoor" }, "content_tags" : [ { "People" : ["Adults (Aged 30-60)", "Two males"], "People with Activities" : ["Dancing"] "automated_tags" : ["people, dancing"] } ], "duration" : 9, "transition" : "standard-cut" } ], "overlays" : { "text-element" : [ { "text" : "20% Off", "font" : "sans-serif", "font-size" : 18, "position-x" : "100px", "position-y" : "100px", "start-time" : 1.24, "end-time" : 1.35, "transition" : "fade-in" }, { "text" : "Join us during day time", "font" : "sans-serif", "font-size" : 24, "position-x" : "100px", "position-y" : "100px", "start-time" : 13.87, "end-time" : 15.68, "transition" : "fade-in" } ], "images" : [ { "images-url" : "http://s3.amazonaws.com/whilter-templates/restaurant/template-1.mpeg", "start-time" : 6.5, "end-time" : 8.5, "position-x" : "10px", "position-y" : "10px", "height" : "200px", "width" : "200px", "transition" : "fade-in" } ] }, "created-by" : "userid-1", "created-date" : "2022-10-30*02:47:33:899", "updated-by" : "userid-2", "updated-by" : "2022-10-30*02:47:33:899", "version" : 1 } ``` ## Template Instance * Sum of duration of scores (end_time - start_time) need equal to duration of template * Sum of duration of clips (end_time - start_time) need equal to duration of template ```json= { "assets_type" : "pubilc-clip", "base-video-url" : "http://s3.amazonaws.com/whilter-components/insta-reels/aspiration/public-clip-1.mpeg", "duration": 45, "background-score" : [ { "sequence" : 1, "score-url" : "http://s3.amazonaws.com/whilter-music/library/score-1.mp3", "start-time" : 0.31, "end-time" : 19.39 }, { "sequence" : 2, "score-url" : "http://s3.amazonaws.com/whilter-music/library/score-2.mp3", "start-time" : 1.53, "end-time" : 25.09 } ], "components" : [ { "slot-id" : 1, "type" : "video", "clip-url" : "http://s3.amazonaws.com/whilter-library/restaurant/start-6.mpeg", "start-time" : 1.53, "end-time" : 13.53, "transition" : "standard-cut", "filter" : "" }, { "slot-id" : 2, "type" : "image", "clip-url" : "http://s3.amazonaws.com/whilter-library/restaurant/image-6.jpeg", "duration" : 4, "filter" : "" }, { "slot-id" : 3, "type" : "video", "clip-url" : "http://s3.amazonaws.com/whilter-library/restaurant/middle.mpeg", "start-time" : 3.02, "end-time" : 23.02, "transition" : "standard-cut", "filter" : "" }, { "slot-id" : 4, "type" : "video", "clip-url" : "http://s3.amazonaws.com/whilter-library/restaurant/end-5.mpeg", "start-time" : 4.22, "end-time" : 13.22, "transition" : "standard-cut", "filter" : "" } ], "overlays" : { "text-element" : [ { "text" : "20% Off", "font" : "sans-serif", "font-size" : 18, "position-x" : "100px", "position-y" : "100px", "start-time" : 1.24, "end-time" : 1.35, "transition" : "fade-in" }, { "text" : "Join us during day time", "font" : "sans-serif", "font-size" : 24, "position-x" : "100px", "position-y" : "100px", "start-time" : 13.87, "end-time" : 15.68, "transition" : "fade-in" } ], "images" : [ { "images-url" : "http://s3.amazonaws.com/whilter-templates/restaurant/template-1.mpeg", "start-time" : 6.5, "end-time" : 8.5, "position-x" : "10px", "position-y" : "10px", "height" : "200px", "width" : "200px", "transition" : "fade-in" } ] }, "created-by" : "userid-1", "created-date" : "2022-10-30*02:47:33:899", "updated-by" : "userid-2", "updated-by" : "2022-10-30*02:47:33:899", "version" : 1 } ``` ## Finish Assets ```json= { "assets_type" : "finished_asset", "video-url" : "http://s3.amazonaws.com/whilter-components/ads/customer-1/advertisement-1.mpeg", "customer-id" : 12345, "created-by" : "userid-1", "created-date" : "2022-10-30*02:47:33:899", "updated-by" : "userid-2", "updated-by" : "2022-10-30*02:47:33:899", "version" : 1 } ``` ## :memo: Step 1: Define ### Step 1.1: Automated Initialization > Enrich below with moe information by looking at amazon reckognition service > Input : Ads similar to **Nikka Mal Babu Ram** Ad Processing : 1. Seperate out backround score and store in s3 2. Seperate out different video clips https://docs.aws.amazon.com/rekognition/latest/dg/segments.html 3. Identify objects and activities and put it in tags (Amazon Reckognition) - https://docs.aws.amazon.com/rekognition/latest/dg/labels.html 4. Identify text and its timings https://docs.aws.amazon.com/rekognition/latest/dg/text-detection.html 5. Identify logo and its timings Ouput : Draft Template Json ```json= { "template_type" : "jewellery advertisement", "assets_type" : "template", "duration": 18.78, "template_url" : "http://s3.amazonaws.com/whilter-templates/restaurant/template-1.mpeg", "background_score_url" : "http://s3.amazonaws.com/whilter-music/library/score-1.mp3", "components" : [ { "slot_id" : 1, "component_type" : "start", "composition_tags" : "", "content_tags" : "", "camera_movement" : "static" "type" : "static", "start_time" : 0.0, "end_time" : 2.53 }, { "slot_id" : 2, "component_type" : "jewellery", "composition_tags" : "close up", "content_tags" : "necklace", "camera_pan_movement" : "left_to_right", "camera_zoom_movement" : "zoom_in", "type" : "dynamic", "start_time" : 2.53, "end_time" : 4.67 }, { "slot_id" : 3, "component_type" : "jewellery", "composition_tags" : "close up shot bottom to top", "content_tags" : "ring", "type" : "replaceble", "start_time" : 4.67, "end_time" : 8.98 }, { "slot_id" : 4, "component_url" : "http://s3.amazonaws.com/whilter-components/aspiration/component-1.mpeg", "component_type" : "jewellery", "composition_tags" : "close up shot left to right", "content_tags" : "necklace", "start_time" : 8.98, "end_time" : 14.39 }, { "slot_id" : 5, "component_url" : "http://s3.amazonaws.com/whilter-components/aspiration/component-1.mpeg", "component_type" : "end_content", "start_time" : 14.39, "end_time" : 18.78 } ], "overlays" : { "logos": [ { "logo_url": "http://s3.amazonaws.com/whilter-templates/restaurant/template-1.mpeg", "start_time": 6.5, "end_time": 8.5, "transition_in": "fade-in", "transition_out": "fade-out", "position_x": "100px", "position_y": "100px" } ], "text_element": [ { "text": "20% Off", "font": "sans_serif", "font_size": 18, "position_x": "100px", "position_y": "100px", "start_time": 1.24, "end_time": 1.35 }, { "text": "Join us during day time", "font": "sans_serif", "font_size": 24, "position_x": "100px", "position_y": "100px", "start_time": 14.39, "end_time": 18.78 } ] } } ``` ### Step 1.2: Manual Identificaton and Approve Input : Ads video and raw template json Processing : 1. Allow user to edit output of above step to fine tune template json 2. Mark static and replaceable content Ouput : Final Template Json ## :memo: Step 2: Generate Template Instance ### Step 2.1: Generate different Template Instances with ML In Step 1.1 we create a template from one ad instance (nikkamal jeweller) and get idea of placing visuals, logo, text, music in any generic ad. Now with that generic template which also improved in Step 1.2, ML will replace visuals, logo, background score as per theme and generate numerous templates (cosmetics ad, fashion brand ad, lifestyle product ad...) ## :memo: Step 3: Generate Ad ### Step 3.1: Modify Template Instance in Editor Now an actual user that might belongs to a local business, brand owner, manufacturer... will use Whilter Editor tool to use these templates instances and will modify visual clips, logo, background music and other Objects to prepare his business ad. ### Step 3.2: Generate Final Video JSON Based on modification done by user, Whilter Video editor tool will generate a JSON similar to template instance that will be to complie final and actual local business ad. ### Step 3.3: Generate Final High Quality Video Now with Media library services a final video will be complied based on instructions given in template instance json of actual business ad. And a high quality video ad will be stored on s3 and available to user for download.