Software Release Plan v24
=========================

## P6 Suite Database Release v24 Checklist

### Pre-Release

- [x] Approve http://eng-server/svn/all/documentation/scope/DatabaseDesign/
- Verify database changes, validate DB update .sql files.
- [x] SL: Branch Release Candidate
- [x] Team: Test Release Candidate
- [x] Product Leads: create svn tags and start tag_* builds
- REST Stop Build And Deploy

### Release Planning
- Team: Verify and Approve

- [x] Verify these changes were made to TEST_COMPLETED_RUNS.
- Team: Review Software Versions for Release.
- [x] http://eng-server/svn/infra/trunk/ilirelease/ilirelease.json
- [x] All changes are identified including runtime dependencies.
- [x] Release steps are correctly ordered by dependencies.
- [x] All RC Software is built and in Software Build History.
- [x] Verify all End-User Software has been approved by QA.
- [x] Verify all End-User Software has been tested and approved by affected business units.
- [x] Global Data Analysis Support (GAS)
- [x] Global Data Analysis (DA)
- [x] Global Field Services (GFS)
- [x] Software Engineering (SE)
- [x] Electrical Engineering (EE)
- [x] Integrity Engineering
- [x] Discuss concerns.
- [x] All tag_* Software is built and in Software Build History.

- [x] Team: Determine Service outage requirements, risk, confidence in changes, rollback requirements.
- Quirk:AWSRM
- none
- Quirk:RQR[includes DigFeedback]
- none
- Gromit
- none
- Iliad
- none
- CloudControl (TODO: smoketest in advance)
- [x] Required for Database Releases
- RESTStop (RunManager servers)
- [x] Required for Database Releases
- TK Run Scanners
- [x] Required for Database Releases

- [x] Team: Approve latest [ilirelease.json](http://eng-server/svn/infra/trunk/ilirelease/ilirelease.json)
~~- [ ] verify none, or review code changes in CMR~~
- [x] Finalize Release Plan

### Release Prep
- [x] SR: verify all tag_* builds are in SBH.
- [ ] Install Creator Wizard, +2 friends :(
- [x] JH,SR,RM Smoke test Cloudcontrol, TK scanners
~~- [ ] RM: CloudControl~~ (TODO: smoke test in advance)
- [x] JH: TK Run Scanners
- [ ] QA: Smoke Test Database
- [6.24.0_Smoke_Test](http://eng-server/mediawiki/index.php/6.24.0_Smoke_Test)
- [x] RM (TODO: automate) Validate VERSION_DATA.sql in [DB V24](http://eng-server/svn/all/documentation/scope/DatabaseDesign/DB V24)

### Release to Test
- [x] RM:
- dev/test: "\\slcnas\Disk-1\MAGPIE\Engineering\Software Build History\ftrr\Dev_Empty"
~~- prod: "\\slcnas\Disk-1\MAGPIE\Engineering\Software Build History\ftrr\Empty"~~
- [x] SR Backup ILIUpgrade "Shortcuts"
- (SR TODO: automate separate release cadences for: dev, test, prod)
- (SR TODO: automate separate per-project updates per version, by tag: beta, stable, current)
- [x] RM Run scripts to convert TEST_COMPLETED_RUNS (SR TODO: automate)

### Release Announcement
- [x] MR Coordinate/plan any outages with any affected teams.
- [x] MR Send email to affected teams

### Pre-Release (Release day)
- [x] SR 11:00 execute Disable.ILIUpgrade.source.ps1
- Alternate: RDP > slcsql4 > taskschd.msc > Disable "ILI Upgrade"
- [x] SR 11:10 Disable http://jenkins-new:8080/job/release_reports_Production/configure
~~- [ ] SR 11:15 Backup shortcuts in case a rollback is needed: copy "\\slcnas\disk-1\Upgrade\TDWilliamson\P6_Suite" to "\\slcnas\disk-1\MAGPIE\Engineering\Software Build History\Backup_of_Upgrade_Shortcuts\Upgrade\TDWilliamson\P6_Suite.2018-11-19"~~
- [x] SL,SR 11:20 Update and commit ilirelease.json "to_release" versions
- SR Jenkins will automatically execute ILI_Release to prepare //slcnas/Disk-1/Upgrade "Shortcuts"
- [x] SR 11:35 Verify shortcuts in "\\SLCNAS\Disk-1\Upgrade" point to new versions.
- [x] SR 11:40 Announce "ILIUpgrade.Custom" is updated.
- Software team may run ILIUpgrade with custom upgrade site of "\\slcnas\Disk-1\Upgrade" to get the software pending release and use TEST_COMPLETED_RUNS for testing.

- [x] SR,SL Smoke Test of P6, BF to verify versions in ILIUpgrade.Custom

### Release (to Production)
Shane Robinson
- n/a SR 14:00 Make COMPLETED_RUNS unavailable for general use (we can't do this yet; we notify and hope users listen.)
- [x] SR 14:00 Start backup of Completed_Runs database (already done nightly, but we do one in the afternoon to allow DA to work in the AM, and minimize downtime for services that aren't restarted until evening to coincide with ILIUpgrade) (45-60min)
- 14:30 Skype Meeting
- [x] SR 14:50 Verify COMPLETED_RUNS database backup was successful
---
Rex McDonald
- [x] RM 15:15 Run scripts to convert COMPLETED_RUNS .
- Scripts should be ready to go. Read any notes in the spreadsheet.
- [x] RM Database Parity Check:
- [x] Run dbal generation / rebuild gen_dbal
- [x] RM 15:20 Use dbal to confirm that everything in completed_runs is as it needs to be for P6. Generate from trunk and compare to RC. (20min)
- [x] RM 15:20 Ensure current database schema is used by FTRR Nightly Process
- [x] RM 15:45 On the sFTP site make a copy of the current database so if an old database is needed it will be there
- Verify: http://jenkins-new:8080/job/create_firebird_database_RC/1269/consoleFull
- Built from correct branch:
- Checking out http://eng-server/svn/all/branches/Release%20Candidate/6.24.0/create_firebird_database at revision '2019-07-01T02:01:16.369 -0600'
- [x] RM 15:50 Build http://jenkins-new:8080/job/FieldTechNightly/ (Jenkins) so we can confirm (next step) that nightly will work correctly. (20 min - 1 hours)
- [x] RM 15:55 Confirm updated version number in new empty.fdb on both the FTP site and ILIUpgrade
- [x] RM CloudControl rebuild with latest DB version (dependent on create_firebird_database_RC)
- http://jenkins-new:8080/job/CloudControl_RC_current/configure
- [x] RM Run ILI2AWS script with DB version.
---
Redeploy AWS CloudControl
- [x] RM 16:00 Update AWS with latest DB version. (CloudControl, s3)
- [x] Software
- [x] Configuration (s3) https://s3.console.aws.amazon.com/s3/object/tdw-cloud-control/ili2aws/template_info.xml?region=us-west-1&tab=overview
- [x] SR 16:00 Update CloudControl (ec2_stage) on TK machine
- [x] Verify credential profile or xml is in place on TK machine
- [x] Smoke Test AWS CloudControl (unable to test prior to release) (RM TODO: Environment abstraction so we can release and test early.)

### Release Evaluation
- Does anyone see any issues or have concerns? Voice them!
- Best option for our business users at this point:
- go-forward, or rollback?
- Impact of each?
- Software Engineering Team Approval
~~- Project Owner for every affected project~~ (ideal)
- [x] MR Software Engineering Release Manager
- Software Quality Assurance Testing Team Approval
- LS,AH,JA,SR(Team) Run ILIUpgrade and Smoke Test P6, BF, IR, and all updated applications as needed.
- LS Empty.fdb Smoke test (RM,LS TODO: doc how to switch to dev build of Empty.fdb from sftp to ftrr.)
- [x] http://eng-server/mediawiki/index.php/6.24.0_Smoke_Test
- Exception: PODS and CustomLabelImporter are not marked as pass; belief is that Louie tested these before he left. Remaining team members don't know how to test these.
- Software Release Manager Approval
- Present status of release, all issues, and any concerns to release manager for final decision by 17:00 (ideally).
- Later approval can complicate the release process, but mostly only for Shane, so it's probably fine.
- Any rollback decision is going to be a late night.
- Software Release Manager Approval Decision
- [x] :+1: Go-Forward? (SR)
- [ ] :-1: Rollback

#### Release: Rollback
- [] SR, Brent Restore Completed_Runs database from pre-release backup (or from nightly backup, replaying transaction journals on top of it)
- [] SR Revert ilirelease.json changes
- [] SR Verify that ILI_Release rolls back all changes to "Release"

#### Release: Go-Forward
- [x] SR 16:50 /!\ Enable http://jenkins-new:8080/job/release_reports_Production/configure auto@17:00
- [x] SR 17:20 /!\ Enable ILIUpgrade.source (slcsql4) before auto@1730

- [x] SR 18:00 Redeploy and restart AwsRM (Quirk) Tomcat services
- [x] prod-quirk r86063
- [x] dev-quirk r87378 to r87499
- [x] SR 18:00 Redeploy and restart RunManager.war Tomcat service
- Challenges: Difficulty Redeploying RunManager.war Service (Typically at SWIFB1, SAVILI101) due to network connectivity issues.
- Workarounds: Redeploy manually. - Preventative: Improve RunManagerMetrics error handling and use a more standard .war redeploy method.
- [x] SLCRST101
- [x] SLCRST301
- [x] SWIFB1
- [x] SAVILI101
- [x] VCRFB4
- [x] HOUFB1
- [x] TULFB1
- [x] EDMFIR101
- [x] SR 18:00 Redeploy and restart TK Services globally
- Challenges: Difficulty Redeploying TK Services (Typically at least SWIRUN101, SWIRUN102, TULFB1) due to improper volume mounts and locking.
- Workarounds: Move services to C:, force-stop applications. - Preventative: Replace machine using Infrastructure-as-Code, Configuration Management.
- [x] SLCTK1
- [x] Retry Redeploy: DentScannerService
- [x] SLCTK2
- [x] Retry Redeploy: S3RunUploader (Could not delete old service dir)
- [x] SLCTK3
- [x] SWIFB1
- [x] SWIRUN101
- [x] Retry Redeploy: WallThicknessScannerService (Could not delete old service dir)
- [x] SWIRUN102
- [x] VCRILI101
- [x] HOUFB1
- [x] TULFB1
- [x] EDMFIR101
- [x] SR (SR WIP) Verify healthy state of all RM and TK services.
- [x] EDMFIR101 TK services
- ~~[ ] SR ~18:00 Notify GAS and GFS when RunManager and TK Services are online~~
- [x] SR ~18:00 Verify ILIUpgrade.source (slcsql4)
- ~~[ ] SR ~ Push other sync sites~~
- [x] SR ~19:00 Verify ILIUpgrade SLC mirror site upgraded (and any others that have completed)
- [ ] SR ~18:45 Notification of Scheduled ILI Software Upgrade completion

#### Release Incidents and Notes
- List any incidents or notes relevant for release report
1) http://jenkins-new:8080/job/FieldTechNightly_Dev/ not updated.
2) http://jenkins-new:8080/job/FieldTechNightly_Dev/4/console error:
Insert Row - Failed violation of FOREIGN KEY constraint "FK_INVENTORY_PART" on table "INVENTORY"
Foreign key reference target does not exist
Problematic key value is ("PART_FK" = '{1ffab6c1-07dd-4a1f-b2fb-7964e2b30214}')
3) RM updated code
4) PODS and CustomLabelImporter were not marked as tested, but believe Louie tested them.
5) SR Cannot modify "\\slcnas\Disk-1\MAGPIE\Engineering\Software Build History\run_manager\current" due to security permissions.
SR Updated ILIUpgrade reference from "current" to "run_manager\1.0.182.87301"
6) SR RunManagerMetrics outdated.
SR Workaround: Manual redeploys.

### Post-Release (to be completed by 10:00 the following day)
- [x] SR (SR Future TODO: automate) Run scripts from eng-server and savili101 to verify versions in all ILIUpgrade mirror sites.
- [ ] JH (SR Future TODO: automate) Smoke test scanners TK Services
- [ ] JH (SR Future TODO: automate) Smoke test database updating from Run Manager TK Services
- [ ] SR (SR WIP: ilireleasereport) Release Report
- [ ] LS Empty.fdb Smoke test

### Release Review
- [ ] (Team) Release Review (via blameless postmortem process)
- [ ] (Team) Include Release Review in Sprint Review "\\slcnas\Disk-1\MAGPIE\Engineering\Software Documents\Sprint Reviews"