# Tasks for making subtitles first-class citizens in Opencast based on #4407

- https://github.com/opencast/opencast/issues/4407

## Estimates Times

- 4h Document how subtitles work with Opencast
- 12h Update workflows and workflow operations to be able to process subtitles by default
- 12h Make sure subtitles can be uploaded in the admin interface (add event)
- 22h Upload and language handling in editor
- Allow uploading subtitle fiels in the editor
- Make sure different language tags are handled properly
- 10.5h Testing & Special case handling
- How to handle already cut (shorter) subtitles
- How to handle old subtitle formats
- 12h Make sure subtitles play properly in Paella Player
- 5h Package subtitle tools so they are easy to use and install

Sum: 77.5h

## Documentation (4h)
Create a page in the documentation detailing Opencasts general approach to subtitles. Can probably be based on https://github.com/opencast/opencast/issues/4407
- Stored as tracks, and treated as such
- List of conventionally defined tags
- language, generator, type, more?
- language-codes
- The only supported format is VTT
- Link to our Vosk docs?

## Workflows (4h)
Refit community workflows to handle subtitles per default
- Use flavors like `captions/source` and `captions/delivery` to handle all subtitles at once
- Subtitle tracks are discriminated by tags
- Which workflows?
- `schedule-and-upload`, `publish`
- Which operations?
- Likely: Publish, Save to asset manager, Editor (subtitles can be cut), ...

## Uploading subtitles
Components which allow uploading media should also allow uploading subtitles by default.

### Admin UI (12h)
Make it easy for users to upload subtitles with their videos
- Modify "Create Event" dialog to allow for uploading subtitles
- Users may upload 0-n subtitles and will need to add tags (metadata)
- Allow to hide the subtitle step if not needed (like extended metadata)
- Make it fast for a good user Xperience (bulk-upload? pre-filled tags/metadata fields)
- Modify Event Asset Upload dialog
- Change default example to upload subtitles as tracks instead of attachments
- Allow users to add tags

### Ingest
`ingest/addTrack` supports tags, nothing to do here.

### New Editor (4h)
Add subtitle upload/download functionality
- In case of the new editor, this should wait after the editor has been made to work with tags for subtitles

### Various LMS plugins (2× 10h)
Change all plugins that already allow subtitle upload
- Moodle (starting with Release 4.0-r1, as of writing not yet released)

Optional: Add subtitle upload to plugins that do not yet allow it

## Generating subtitles

Ensure that existing methods of subtitle generation keep working and make it easier than ever to use them.

### Subtitling services (8h)
Subtitles can be generated in Opencast by internal or external services. The resulting media package elements should conform to the new standards.
- Change file output to VTT if possible. Otherwise advise a conversion to VTT in the docs.
- Change docs to reflect recommended flavor changes.
- Change associated WOHs to be able to deal with tags
- Services
- Amberscript
- IBM
- Microsoft
- Google
- Vosk

## Modifying subtitles

Allow users to manually and automatically modify subtitles without having to leave the Opencast ecosystem.

### Old (Admin UI) Editor
- Cannot modify subtitles. Adding the capability is likely not worth it

### New Editor (18h)
- Check if backend needs changes
- Frontend uses flavor for identification. Change to another method (language tags, track ids)
- Probably not tags, since we have to assume that they could be missing?
- But what about the configuration then?

#### Editor Backend
- Send (maybe also receive) tag information

### Editor WOH (1h)
- Does support cutting VTT subtitles already. Likely nothing to do.

## Presenting subtitles

Display subtitles in an usable and accessible manner. Likely somewhat out of our hands.

### Paella Player Integration (for both 6 and 7) (12h)
Make sure the Opencast-Paella integration can load subtitles from tracks
- Load subtitles as tracks from main flavor `captions` by looking at the tags
- Use tags for e.g. display purposes
- Keep subtitles from attachments as fallback
- Catalogs can probably be dropped?
- Drop DFPX and SubRip support

### Paella Player
Can display subtitles. Could probably be improved, but that's optional.

### Theodul Player
Don't bother. Will eventually be removed anyway.

## Handling non-source subtitles (6h)
The common Opencast approach to tracks is to store the original tracks in a mediapckage as `*/source`, and then change the flavor when the workflow modifies them. However, subtitles may be created on the basis of modified tracks (e.g. cut tracks) to save on resources, meaning the "length" of these created subtitles will not match the length of the other tracks in `*/source`. This may be problematic.

I am keeping this as an extra topic instead of integrating it into the other topics, since for Opencast this is somewhat unconventional behaviour and requires special consideration.

### Editor Operation
Check if the editor operation will wrongly cut subtitle files if they differ in length from their associated video files
- Adapt workflow to handle such cases? E.g. skip subtitle cutting if they are specifically tagged?
- But then you would need to regenerate subtitles each time the cutting is changed

### New Editor (Frontend)
Assumes that subtitle files send by the backend are based on the video tracks in `*/source`. In actuality, they may be based on `*/generated`
- The displayed video will not fully match the displayed subtitles, if for example the subtitles were generated from a video shorter than the source.
- Other component (e.g. the timeline) might be affected as well.
- Solution #1: Send `*/generated` to the frontend, for display in the subtitle editor view only
- Creates a weird disconnect between the subtitle editor view (now based on `*/generated`) and the other menus (mostly based on `*/source`)
- Changes in the cutting view could not be reflected in the subtitle view anymore
- Cutting subtitles might not work anymore with this approach
- Solution #2: Generate `*/source` subtitles from the ones based on `*/generated`, by using the cutting data from `smil` to move the timestamps
- Requires the `smil` used to create the videos in `*/generated`
- Probably requires a new, unintuitive WOH?

## Migration (12h)
Write a script that migrates subtitles from attachments to tracks
- TODO: Which attachments? From the asset manager or from the publication?
- Might be adopter-specific?
- Add tags to the attachments based on {TODO: based on what? the subflavor? check exisiting tags?}
- Change all attachments with {flavor} to tracks
- Change subtitle track flavor to unified flavor
- TODO: Convert format to VTT
- Probably trivial for Subrip, but what about other formats?
- Can we use ffmpeg for that?
- What if conversion fails? Is a warning ("track X in mp Y failed") enough?
- Make this a workflow operation

## Misc
- The Opencast Tobira module might need to change how it gets subtitles (6h)
- The LTI Service offers an `upsertEvent` function that takes captions, needs to be fixed (4h)
- Packages for STT tools (5h)