[TOC]

# Current status of tests

| Test-script |Status| info |
| -------- | -------- | -------- |
|Integration tests |:OK:| |
| Telemetry| :OK:| |
| Deplyment for indivudal users| :OK:||
| **Unit tests** |
| 2. /lib/pkp/tools/travis/prepare-tests.sh | :ok: | |
| 3. Unit-tests | [Failure in EntityDAO](https://github.com/xmlFlow/ojs/actions/runs/5923791465/job/16060093105#step:13:82) | [Error](https://github.com/pkp/pkp-lib/blob/main/tests/classes/core/EntityDAOTest.php#L40) |
| 1. load data sets | | |
| 4. Validate and lint |
| 5. Upgrade |

- :watch: Functionaliy is added step-by-step but fine-granular optimizations not done.
- :question: As this can be run in parellel to current travis, shall we already merge this to main and possibly other branches ?
-

# Improvement ideas
1. :new: Seperate github workflows for unit-tests and integration tests ?
1. :new: can we rename the content creation tests to names, what the the user do ?
- The idea is: If we can selectively run them accoringly in a user github account and branches other thang e.g. main, we may run a lot of tests
3. :new:
8. Efficient testing
- :watch: cypress waiting times remove
11. :watch: How about separating the data creation from the tests?
- (E.g. only save a database dump; do an upgrade as part of the build process).

2. :OK: Run the test start manually and maintain control within the team.
- **Not needed anymore**
- Reduces waste without requiring extensive rewriting.
- Saves time for developers
13. :OK: Contact Github to get a more generous free account.
- **written a support ticket and confirmed **
14. :OK: How about reducing the size of the test matrix?
- **not needed anymore**
- Options: MySQL/PHP8.1 and PostgreSQL/PHP8.0?
- The rest is done with cron?
- Not needed now :OK:

10. :OK: Check other CI providers (Marc cites Gitlab as the ultimate for OSS)
- [circle-ci](https://support.circleci.com/hc/en-us/articles/360049861131-When-will-free-or-open-source-credits-renew-)
- Would likely to catch malicious actors targeting runners.

## CircleCi vs. Github Actions
| Characteristics | Github actions | Circle-CI |
| -------- | -------- | -------- |
| Execution Limit (at least 48000 miuntes/month) | :ok: | 40000 |
Managed (cloud service)| :ok:|:ok:
Self-hosted option| :ok:|:ok:
Available on-premises| :ok:|:ok:
External nottifications (e.g., Slack) | :ok:|:ok:
Polyglot (multilingual)| :ok:|:ok:
Docker/Kubernetes support|:ok:|:ok:
| Documentation |bad( but community documentation offer is comprehensive) | :ok:|
| Advanced security scanning| ony for paid customers| [:ok:](https://circleci.com/blog/scheduled-security-scans/)|
| creating release notes upon deployment|:ok: :new: | :x: |
Real-time debugging of builds| :ok: :new: | :x: |
| assigning issues to repository collaborators| :ok: :new: |:x: |
Multi-step pipelines| :ok: :new:| :x:|
| labeling GitHub issues |:ok: :new: | :x:|
Internal VCS |:x: | :ok:
| Supports other repositories types |:x: | :ok:|

1. Self-hosted runners.
- complicated safety implications. :x:
5. Pay for more minutes (currently prohibitively expensive).
- may be not an option :x:

This document describes the methods and steps to move PKP software testing from travis.com to github.com. The following mandatory criteria must be satisfied for a successful transition to github actions.

# Summary
## Mandatory criteria

1. **Consistency**: The current unpredictability of Travis for certain integration tests should be addressed. :OK:
2. **multiple enviroment**: support (php versions and databases) should work. :OK:
3. **Secrets storage**: support encyrypted tokens for certain operations should exist in github actions :OK:
4. **pkp-lib integration**: the integration of running pkp-lib integrations must be supported, either through similar methods or within user repositories.
5. **Rights management**: should work for pkp organisation users across repositories :OK: (under settings -> actions -> general)
1. Actions permissions
6. **Upgrade tests**: has to implemented and creation of datasets should be supported.
7.
## Nice to have criteria
1. **Performance**: Tests should run faster than in Travis.
2. **Storage**: The memory limitations should not restrict the development.
- [Possibility to delete artiacts](https://docs.github.com/en/rest/actions/artifacts?apiVersion=2022-11-28#delete-an-artifact) :OK:
3. **File sharing**: between certain testing steps. :OK: :new:
4. **Local testing:** local or cloud-based github runner: dropped :OK:
6. **Consistency**: Tooling should be kept parallel with dev environments as much as possible
5. **Email server**

# Details
## Mandatory criteria

### 1. Performance

Standard GitHub-hosted runners

GitHub plan | Total concurrent jobs | Max concurrent macOS jobs |
| --- | ---| --- |
Free|20|5
Pro|40|5
Team|60|5
Enterprise|500|50

### 2. Storage

Product | Storage | Minutes (per month)
-- | -- | --
GitHub Free | 500 MB | 2,0
00
GitHub Pro | 1 GB | 3,000
GitHub Free for organizations | 500 MB | 2,000
GitHub Team | 2 GB | 3,000

### 3. Execution time

#### Current
| Product | Time in hours |Builds | Total time in minutes |
| -------- | -------- | -------- | -------- |
OMP| 6 hours | 8 builds | 2880 minutes |
OJS| 5 hours | 24 builds | 7200 minutes |
OPS| 3.5 hours | 7 builds | 1470 minutes |

Total per week: 11550 minutes / circa. 2000 hours

**Total per Month: 8000 hours**

# Links

- [github.com guidelines for migrating from Travis](https://docs.github.com/en/actions/migrating-to-github-actions/manual-migrations/migrating-from-travis-ci-to-github-actions)
- [Billing and adminstration](https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration)
- https://poweruser.blog/storage-housekeeping-on-github-actions-e2997b5b23d1
- https://docs.github.com/en/billing/managing-your-github-billing-settings/about-billing-on-github
- https://saucelabs.com/resources/blog/circleci-vs-github-actions-vs-gitlab-key-differences

# Glossary

**Job execution time** - Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete.

**Workflow run time** - Each workflow run is limited to 35 days. If a workflow run reaches this limit, the workflow run is cancelled. This period includes execution duration, and time spent on waiting and approval.

**API requests** - You can execute up to 1000 requests to the GitHub API in an hour across all actions within a repository. If requests are exceeded, additional API calls will fail which might cause jobs to fail.

**Concurrent jobs** - The number of concurrent jobs you can run in your account depends on your GitHub plan, as well as the type of runner used. If exceeded, any additional jobs are queued.