# channel-backup
*This serves to be a reformated copy of the [GitHub Project Board](https://github.com/orgs/TinkerStorm/projects/4) to formalize and map out the individual aspects of the project.*
- Core Architecture
- Plugin Authoring
- Endpoint / Destination Adapters
- [x] [Discord](https://discord.dev) (partial support)
- [Webhook](https://discord.dev/resources/webhook)
- [Embed](https://discord.dev/resources/channel#embed-object)
- [Embed Limits](https://discord.dev/resources/channel#embed-limits)
- [ ] [Slack](https://api.slack.com) (unknown)
- [ ] Office (Teams) (unknown)
- [ ] [Guilded](https://guildedapi.com) (unknown)
- [Webhook](https://guildedapi.com/resources/webhook/#execute-webhook)
- [ ] rocket.chat (unknown)
## Supported file extensions (possibilities / planned)
Including data types, template engines and data transfer protocols.
> If a template engine has native html processing that cannot be disabled, it cannot be used here (i.e. Vue, React, Marko, Pug, etc.).
| Category | Type | Extensions | Status |
| --------- | -------------------------------------------------------- | ---------------- | ----------- |
| Images | PNG | `.png` | Implemented |
| | JPG | `.jpg` | Planned |
| | WEBP | `.webp` | Researching |
| Templates | [Nunjucks](https://github.com/mozilla/nunjucks) | `.njk` | Implemented (Partial) |
| | [Twing](https://www.npmjs.com/package/twing) | ? | Alternative |
| | [Liquidjs](https://github.com/harttle/liquidjs) | ? | Unknown |
| | [Squirrelly](https://github.com/squirrellyjs/squirrelly) | ? | Unknown |
| Content | JSON | `.json` | Implemented |
| | *Discord-flavoured* markdown | `.md` | Implemented |
| | YAML | `.yml` / `.yaml` | Implemented |
| | XML | `.xml` | Unknown |
I should note that the **Markdown** content parser is just turning the string content into an object `{ content: fileContents }`, and that **Markdown** only has support for what Discord supports (i.e. headers, footnote links, checkmarks, codeblocks by indent, etc. aren't supported; spoilers are, but you won't know until it's posted).
Mentions are also handled differently with the [allowed mentions]() change the config and intended behaviour of how a webhook can target users and roles. If disabled, mentions for these entities will become regular strings with an `@` prefixed (much like how you would write it).
A *[message reference](https://discord.dev/resources/channel#message-object-message-reference-structure)* can also be provided, but knowing of such message must be done in the file itself or by logic through a template.
## Tasks / Ideas
- [ ] Define logging strategy
- Pre-existing prefixes (upper case)
- resolve
- query
- warn
- sent/{index} where `index` is the loop index from the generated file listing
- Currently declared with format `[{upper($prefix)}] {message}` which can vary based on the progress of the sequence.
- [ ] Move away from `discord.js` to lightweight / standalone alternative.
- Library currently handles:
- Webhook ratelimit handling
- Embed colour resolution
- File attachment handling
- [x] Glob star matching
- [x] Attempt to minimise the file listing by making use of `glob.hasMagic(pattern)` instead of requiring file type.
*Feature implementations below would likely prevent this from happening at this stage of development.*
- [ ] Add option to reverse a glob result.
`action / sequence` - OneOf(reverse, first, last, etc.)
- [ ] Allow foreign files to be requested from the web.
These would be retrieved without knowing the file contents, nor how to process it.
- [ ] Use of [GitHub Learning Lab](https://lab.github.com) to provide interactive tutorials?
- [ ] GitHub Action for CI / CD use
- References to:
- [dorny/paths-filter](https://github.com/dorny/paths-filter) for job workflow filtering
- [actions/issues](https://github.com/actions/issues) for glob matching against label keys
- [ ] Plugins should be able to turn off certain aspects of the core functions on initial run through.
- i.e. [Resolve Author](#Resolve-Author) should be able to disable this to allow the forced reposting of messages *incase* the content differs from last run.
- There may be an alternative to save the entire payload to the cache, but I do not see this as a viable option if the content can change if another end-user runs the script on their own system.
- [ ] {CLI} `channel-backup create [dir]` - create a new configuration in the current (or specified) folder, create it if it doesn't exist.
- Option: `--plugin` - A list of plugins without `channel-backup-plugin-` key prefix included to be installed upon creation.
- [ ] Project Structure (unknown / planning)
- Presently deciding if this should only be coded in nodejs or allow multiple languages to be existing at the same time.
- Allowing for multiple languages of the same architecture would need a centralized specification to decide if something should be implemented for a specific language and then to decide if any others should follow.
- The idea being that it would allow anyone to write their own plugins in the language of their choice and not be limited nor hindered by their own system limitations.
- [ ] Have plugin base structures embedded into the core package *or* have it as another package.
Would be named something like `channel-backup-plugin-base`, but then this would have problems with the key matching if plugins are autoloaded in.
- [x] Repository Structure (A) - Split across 2 orgs, multi language
Core modules would be on **TinkerStorm**, repo / project *templates* would be on **RocketDragon**.
- channel-backup - Project Hub
- Home to discussions, spec issues, proposals, project-wide RFCs, etc.
- channel-backup-spec?
- channel-backup-node - NodeJS Hub
- Chance of being a mono repo using lerna package config.
- channel-backup-node-action - GitHub Action using the nodejs implementation
- channel-backup-node-plugins - Node environment plugins
- channel-backup-python / channel-backup-py - Python Project Hub
- channel-backup-python-action - GitHub Action using the python implementation
- channel-backup-{lang}
- channel-backup-{lang}-action
- channel-backup-{lang}-template
- channel-backup-{lang}-plugins
- channel-backup-clean *template* - Clean environment
- [ ] Repository Structure (B) - Split across 2 orgs, only nodejs
Same Core and Templates split as before.
- channel-backup - Project Hub / NodeJS Hub
- channel-backup-action
- channel-backup-plugins
- channel-backup-template
- [ ] Repository Structure (C\) - *either A or B, but unified into one organization (either contained within an existing one, or an entirely new one dedicated to the project itself)*.
- [ ] {CLI / GH Actions} Allow a preconstructed message from the pipeline to be processed and sent forward to a webhook.
Make use of a standard override to prevent interactions with the cache (like a 'dry run', but refuse all file system modifications).
> Would be useful for announcements on servers.
- [ ] {GH Actions} Use `steps.*.output` to provide:
- an array of sent files
- an array of resulting message IDs
- etc.
- [ ] If messages aren't found before last available message, break out of edit and delete messages until it can recover (use of partial repost queue).
[
"msg1", // ✅
"msg2", // ✅
"msg3", // ❌
"msg4" // ✅
]
2nd option would be to shift the messages over and repost what remains.
> Likely to become a problem if username / avatar differs as mentioned with the [Resolver Author](#Resolve-Author) plugin idea.
- [ ] {CLI} Post a single message in a particular workflow.
`channel-backup post <workflow> [index/file]`
Unlikely to work as intended if templates are involved (unless the cache is used correctly).
Unsure how it would work with a pattern, but it is likely that it can be applied across a range by checking `matches[0]` against the full array of files found in the particular workflow and then match that against the cache.
- [ ] Add a "Credits" header for packages used in the project or those that have contributed to the project itself (directly or indirectly).
> Use of [all contributors](https://allcontributors.org) would be useful for those that contribute directly.
- [ ] Create a "Who's using this?" page / section.
I'm aware that I might be the only one using it right now (either for myself, those that I help or those that I am contracted to). It would be interesting to see how many individuals, communities or organizations use it over time, but a set of guidelines should be formed to ensure that all listed entities comply with (i.e. terms of service).
- [x] Caching would be done by storing message IDs with the webhook used to send them.
{
"webhook-id": [
"message-id-1",
"message-id-2"
]
}
Storing by workflow name would cause multiple problems including 404s due to the messages not being posted by the particular webhook or seeing that the webhook would have been sent in another channel and then it has since been changed.
- [ ] {GH Actions} Make use of environments to declare different workflows. (Would also need to detect if any files have changed for a particular workflow by using the array of files retrieved by glob.)
- [ ] Make use of a task runner on the command line? [grunt](https://npm.im/grunt) / [gulp](https://npm.im/gulp)
- [ ] {CLI} CLI tool to handle plugins / multiple workflows?
- `channel-backup`
- `channel-backup plugins` - list / manage plugins
- `channel-backup run [key]` - run [specific (key)] workflow
- `--dry-run` - run without making requests
- `--no-cache` - post without using the cache
> consider not saving either, but would probably be counterproductive to the overall flow of statements
- `--verbose` - run without refreshing the terminal
- `--web` - use log friendly format for CI
> possibly detect if a github action environment is being used instead.
- [ ] Payload validation
One of the more common issues is when the payload gets too large for Discord to accept (namely [embed limits](https://discord.dev/resources/channel#embed-limits) and [message limits](https://discord.dev/resources/channel#create-message)).
- [ ] Query: Update Promotion - built in or plugin?
:::info
If it's a plugin, more plugin base architecture may be needed [Jekyll Plugins].
:::
- [ ] Using responsive templates
:::danger
Blocker: only certain extensions should be accepted, and the file will need to be run through the loop again.
:::
- [ ] Multi config protocol.
Proving to be difficult because when thinking about actions, the only way to get it running would be to use a matrix but then keeping the webhook urls *secret* is another concern by itself.
Having a keyed secret per webhook would be quite annoying to manage (and become an increasing concern for larger repositories).
runs-on: ubuntu-latest
matrix:
include:
- webhook: ${{ secrets.INFO_CHANNEL_WEBHOOK }}
workflow: info # if not specified use root, though may be changed
dry-run: true
verbose: true
Another possibility would be to map out the matrix of configurations actions would use and then process it in action... but then that destroys the underlying functions of the matrix itself.
- [ ] [Yeoman Generator](https://yeoman.io/authoring/)
Repo: `generator-channel-backup` / `generator-channel-backup-plugin`
Command: `yo channel-backup` / `yo channel-backup-plugin`
- [ ] Test suite? `ava`, `mocha`, `tap`, etc.
- [ ] Linter: `xo`, `eslint`, etc.
- [ ] i18n support (default to English)
- [ ] Option: `--dry-run` - If provided, no mutually destructive methods should be invoked. Useful for verbose logging.
- [ ] Option: `--verbose` - Log all minor steps, possibly include rendered data before parsing into the environment.
- [ ] If originating from CLI, attempt to retrieve webhook from config file, then look in the environment using `WEBHOOK_URL`. Unsure which to prioritise, ENV *sounds* like it would give more priority, but *config* is more accessable. Consider use of [dotenv](https://npm.im/dotenv) for windows systems.
Log it as a debug that the webhook was not found in config. Force fail the program if a *valid* webhook is not found in ENV either. (Make use of GET request to make sure it's valid.)
## Plugins
### Resolve Author
```typescript
type EntityMap = { [author: string]: string }
```
| Target | Origin |
| ------:| ------ |
| `~.avatar_url` | `~.username` |
| `~.embeds.*.author.icon_url` | `~.embeds.*.author.name` |
> - `~.key` implies it is from the root of the constructed / generated payload.
> - `~.iterable.*` or `~.iterable.*.key` implies it is an `Iterator` at the root ^, and to iterate over all the values. If another key is provided after, then also access that key during the loop.
>
> See [Object filters](https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#object-filters) for [GitHub Actions] as reference.
## Examples
### Basic content rendering
```yaml
embeds:
- title: ":date: Weekend Schedule"
color: "#F44106"
- title: Friday
color: "#10B7FD"
fields:
- name: Time
inline: true
value: |
:clock530: `17:30`
:clock6: `18:00`
:clock630: `18:30`
:clock7: `19:00`
:clock730: `19:30`
:clock11: `23:00`
- name: ​ # 200B - Zero Width Space
inline: true
value: |
:tada: Pre Jam mingle on Discord
:projector: Keynote and Theme
:brain: Brainstorming
:teacher: Pitching and Teamforming (optional)
:technologist: Jamming
:city_sunset: Closing for night
- title: Saturday
color: "#3142C5"
fields:
- name: Time
inline: true
value: |
:clock9: `09:00`
:clock12: `12:00`
:clock1: `13:00`
:clock7: `19:00`
:clock11: `23:00`
- name: ​ # 200B - Zero Width Space
inline: true
value: |
:city_sunrise: Jam Re-opening
:flying_disc: Sharing of Progress
:sandwich: Lunch time mingle
:boomerang: Sharing of Progress
:city_sunset: Closing for night
- title: Sunday
color: "#6A00DD"
fields:
- name: Time
inline: true
value: |
:clock9: `09:00`
:clock12: `12:00`
:clock4: `16:00`
:clock6: `18:00`
- name: ​ # 200B - Zero Width Space
inline: true
value: |
:city_sunrise: Jam Re-starts
:flying_disc: Sharing of Progress
:package: Start of game uploads to GGJ Site
:tada: Jam Finishes
```
Results in the following post being rendered:
### Dynamic Content
The template below was coded in [nunjucks](https://mozilla.github.io/nunjucks/) but wasn't used because the template engine wasn't fully implemented. *So I have no idea if this works.* :joy:
```nunjucks
{% set numberMap = ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine"] %}
{% macro numberToEmoji(n, shift) %}{% for digit in n.toString().padLeft("0", shift).split("") %}{{ numberMap[+digit] }}{% endfor %}{% endmacro %}
username: Falmouth Global Game Jam
embeds:
- description: | {% set shift = messages.length // 10 %}
{% for item in messages %}
{{ numberToEmojis(loop.index, shift) }} **[{{ msg.embeds[0].title }}](https://discord.com/channels/{{ msg.guild.id }}/{{ msg.channel.id }}/{{ msg.id }})**
{% endfor %}
```
## Misc. / References
- GH Actions input names and types
- load_from: enum(args, env)
- sequence: string
- webhook_url: string *from `${{ secrets.* }}`*
---
- [Jekyll Hooks](https://jekyllrb.com/docs/plugins/hooks/)
- [GitHub Actions]
[Jekyll Plugins]: https://jekyllrb.com/docs/plugins
[GitHub Actions]: https://github.com/features/actions
Sign in with Wallet
Connect another wallet