GCM Video Uploading Notes

# GCM Video Uploading Notes Considering this subject is the first major feature none of us have really covered before I thought it would be a good opportunity to create an RFC for it to keep the discussion in comments on this for historical reference. ## Motivation Since the GXM properties are migrating away from Neulion we need to provide a way for GXM producers to upload raw MP4 video files so they can be transcoded on [JWPlatform](https://www.jwplayer.com/video-platform/). Neulion provided mechanisms to do this with some expensive licensing of [Signiant](https://www.signiant.com/) that was hosted internally at Neulion that was then integrated with their own custom video management dashboard. ### Current Workflow Fitsum from GXM provided us with a very minimal workflow about how this all works. > 1. When a show is done recording, the producer(s) will upload 3 files to our instance of the file transfer system Signiant. These files are: > 1. The mp4 of the show. > 2. A 16:9 jpeg image used for the playlist thumbnail. For creating the sliders, the system requires 4 additional images (desktop, mobile, tablet, and OTT). The producers create these images and upload them into Wrike. Someone from our content team would then take these images and upload them into the Endeavor system via their CMS for the specific show. > 3. CSV file with some metadata about the show. > 2. Endeavor also uses their own instance of Signiant to automatically ingest the files above and load them up to their CMS to make the content available across all devices. Sean, their infrastructure person, also noted that these raw MP4 files can range from a few GB to 10+ GB which is why they needed to use Signiant for the large file transfer. ## Background Context Currently we have 3 services in the platform we can leverage to accomplish this task. 1. Video Dashboard 2. Video Platform API 3. JWPlatform (99% certain) **NOTE:** I am assuming the use of JWPlatform at the time of writing this, if this changes then so does everything else. JWPlatform provides [usage documentation](https://developer.jwplayer.com/jw-platform/docs/developer-guide/management-api/uploading-videos/) for giving them videos to be transcoded. From the list of 4 options only 2 are viable given the size of files we plan to work with. 1. [Fetch Uploads](https://developer.jwplayer.com/jw-platform/docs/developer-guide/management-api/uploading-videos/#fetch-uploads): we give JWPlatform a public URL for them to download from 2. [Regular Style Uploads](https://developer.jwplayer.com/jw-platform/docs/developer-guide/management-api/uploading-videos/#regular-style-uploads): we request a signed S3 your from JWPlatform and then use that temporary URL to upload to In either of these 2 cases the uploading and transcoding would be performed asynchronously. Fortunately, JWPlatform provides an API call to [list all videos they have in different states](https://developer.jwplayer.com/jw-platform/reference/v1/methods/videos/list.html) using a `statuses_filter` such as created, processing, ready, updating, and/or failed. ## Detailed Design ### Fetch Uploads This implementation requires more work in terms of leveraging infrastructure to store very large files in [GCP Cloud Storage](https://cloud.google.com/storage/) which is just Google's version of Amazon S3. This would give us the required public URL to give to JWPlatform to download from. The happy path of uploading a video would be as follows: #### File Upload 1. Producer logs into the dashboard and goes to a new "Video Upload" section 2. Producer picks the raw MP4 file and clicks upload 3. Dashboard does GET to the Video Platform to request a temporary GCP Cloud Storage signed URL 4. Video Platform replies with the URL to the Dashboard 5. Dashboard uses that URL in a file upload with the raw MP4 until 6. Dashboard handles any success/failure response from the signed URL from GCP #### Create Video 1. Producer navigates to the "Video" section 2. Producer clicks "New Video" to get the form fields for the video data 3. Dashboard does a GET to the Video Platform to get a list of previously uploaded files and renders a drop down for the "Raw Video Source" 4. Producer selects the appropriate "Raw Video Source" and fills in the rest of the fields for title, desc, etc. 5. Dashboard does a POST to the Video Platform 1. Video Platform creates the video in PostgreSQL with a pending status 2. Video Platform creates a sign GCP URL for the "Raw Video Source" and does a POST to JWPlatform with the title and URL 3. JWPlatform creates a video record and response with a JWPlatformVideoID and begins to download from the GCP URL. 4. Video Platform handles the responds from JWPlatform and stores the JWPlaformVideoID in PostgreSQL 6. Video Platform responds to the Dashboard creating a video with `HTTP 202 Accepted` meaning there is background processing occurring 7. Dashboard can show in the video list the current status of videos. ### Regular Style Uploads This mostly works the same way, but it completely removes the need for storing the raw MP4 files. #### File Upload 1. Producer logs into the dashboard and goes to a new "Video Upload" section 2. Producer picks the raw MP4 file and clicks upload 3. Dashboard does GET to the Video Platform to request a temporary JWPlatform signed URL 1. Video Platform creates an empty video in PostgreSQL to generate a new VideoID with title "New Video {Date}" 2. Video Platform create a new video on JWPlatform to generate a new JWPlatformVideoID and upload URL 3. Video Platform stores the JWPlatformVideoID on the record for the VideoID and sends back the VideoID and upload URL to the dashboard 5. Dashboard uses that URL in a file upload with the raw MP4 until 6. Dashboard handles any success/failure response from the signed URL from JWPlatform #### Edit Video 1. Producer navigates to the "Video" section 2. Producer find the "New Video {Date}" or sorting by created date or status 3. Producer edits all the necessary fields for description, images, etc. 6. Video Platform responds to the Dashboard creating a video with `HTTP 202 Accepted` meaning there is background processing occurring 7. Dashboard can show in the video list the current status of videos. ### Status Update Background Job Since JWPlatform does not perform any alerts or webhooks when transcoding is completed the Video Platform needs to periodically check all pending videos in PostgreSQL against JWPlatform and update their status accordingly. Once transcoding is completed the Video Platform can request the appropriate and store video sources from the [JWPlatform Video Conversions API call](https://developer.jwplayer.com/jw-platform/reference/v1/methods/videos/conversions/list.html). In the Video Platform we can provide limits so that no video in a processing status is ever returned to apps.