###### tags: `Jenkins X` # Jenkins X office hours notes ## November 8, 2022 Participants: - Christoffer - Mårten - Terry Agenda: - Hacktoberfest - [Changelog propagation](https://github.com/jenkins-x/enhancements/tree/propagate_changelogs/proposals/propagate-changelogs) Mårten suggested to start working on propagating changelogs. One notable result of this should be that we get an accumulated changelog in the version stream changelog. - reusePullRequests - for jx-promote and jx-updatebot, pull requests are reused (Looking up chart name label in PR, when new version, the commits will be replaced) - kpt live apply can be used in boot job, manifests in kpt package applied to cluster. It will keep list of deployed resources and will actually remove stuff when removed (which is not necessarily the case with current tools) - OCI registry support for helm charts. Implementation in boot job - is it a hack or not? - Supply chain security - Terry is eager to hear the status on this, as it may be a nice reference project for others to follow, White paper on Supply chain security wanted! - CDEvents - an early 0.0.1 Release is starting to stabilize, we should get involved. they have a golang api. Should be simple to integrate. - 2022 MlOps roadmap. - Should refresh the MLOps capabilities in JX. - Current offerings of MLOps not following EU AI regulations. Handling personal data requires very strict processes. Third party audit. jx treats ml as software, going through the same quality process. Should tidy up the MLops parts of Jenkins X. Scan through [the MLOps roadmap](https://github.com/cdfoundation/sig-mlops/blob/main/roadmap/2022/MLOpsRoadmap2022.md) Action Items: - Blog on GSoC work - JX-Ui and Supply chain security - CDEvents - get started - MLOps - check it out ## October 25, 2022 Cancelled (kubecon/CD Summit) ## October 11, 2022 Participants: - Ankit - Andy - Rajat - Terry ## September 27, 2022 Participants: - Ankit - Andy - Osama - Vansh Agenda - GSoC update - https://github.com/tektoncd/chains/pull/436 -> Tekton chains pipelinerun attestations - Jx ui chart is in versionstream - LTS version - AWS credits - Testing/Infrastructure SIG: https://github.com/jenkins-x/jx/issues/8386 ## September 13, 2022 Participants - Andy Nelson - Ankit - Dave C - Marten - Osama - Rajat Gupta - <Add me> Agenda: ## August 30, 2022 ### Participants - Ankit - Andy Nelson - Bachu Paul - Christoffer Vig - Rajat Gupta - <add me> ### Agenda: - Two new members in jenkins x org (Rajat and Osama from GSoC) - GSoC update - New UI - Sveltekit upgrade to almost latest version (next.449) - Npm run build works - Dockerfile is created - Next: run it in jx infra cluster and make it available in version stream - Supply chain security - Partially working release pipeline for version stream to unblock new releases - ToDo: fix jx-verify and jx-test ### Action Items: - Write a blog about terraform-operator (Ankit) - Fix: DAG for failed pipelines (Rajat) - CD-mini-summit (chris) ## August 16, 2022 Participants: Andy Arvind Bachu Christoffer Rajat Gupta Dave C Utkarsh Agenda: Remote cluster bug update. GSoC: Update from Rajat: Creating DAG graphs using D3 to show pipeline steps. For instance - Click on a step can restart it Golang upgrade to 1.18 - in progress, a little belated Jenkins jx-docs website, font size and hierarchy of code base should be changed, very messy, Docsy should be used from scratch, using a blank template. Should recreate the whole docs when we find a new name. Utkarsh: interested to test Azure terraform Arvind: Learning helm charts. The new UI needs a helm chart! You may contribute here. Bachu: Want to contribute to upgrade External Secrets August 2, 2022 Participants: Osama Magdy Aied Rajat Gupta Mårten Svantesson Ankit Mohapatra Achernar Eridani Andy Nelson Dave C Bachu Paul Agenda: GSoC status update: Rajat: first UI Demo (stopping a pipeline from the UI) Osama: SBOM status update ## July 19 2022 Participants: Aju Ankit Arvind Christoffer Dave C Osama Magdy Rajat Gupta Terry Agenda: HackMD - Ankit to take a look Youtube channel - Christoffer to get maintainers admin access AWS open source credits - no news yet. Ankit to send an email tonight ESO migration from KES: https://github.com/jenkins-x/jx/issues/8298 Remove/close old issues(2018-2020), there are 8 in jenkins/jx Fix issues with nested gitlab repository witty changelog (slack and jx pipeline start/stop work in progress): https://github.com/jenkins-x/jx/issues/8303 and https://github.com/jenkins-x/jx/issues/8304 . Tekton 0.32.4 - released chart - changed versioning https://github.com/cdfoundation/tekton-helm-chart Ankit tests on new and upgrade clusters Tekton chains helm chart: https://github.com/chainguard-dev/tekton-helm-charts Arrange charts chat with chainguard (https://github.com/chainguard-dev/tekton-helm-charts) CDF TOC, Terry - create reference architecture for Continous Delivery Welcome to new members - Aju and Arvind :) Tip to install k3s https://jenkins-x.io/v3/admin/platforms/k3s/ Go upgrade to 1.18 happening soon issue Release process: LTS versions have not been released for a long time, due to lack of dev resources How handle version upgrades, beta releases, upgrade path. Should take care to use proper versioning, breaking changes require major version upgrade Should try to keep two versions (?) updated Release notes should have proper documentation on how to upgrade Predictable release cycles. End-to-end tests for each commit to master Terry: check out old archives, issues, etc.white papers, enhancement proposals repo. this was discussed earlier Build a new website for Jenkins X https://github.com/jenkins-x/jx-docs/issues/3627 DEMO New JX-UI - https://github.com/jenkins-x/jx-ui Knapsack Pro https://knapsackpro.com/features#supported-ci-providers TOC meeting July 6 2022 Participants Ankit Christoffer Vig Bachu Paul Rajat Gupta Fatih Degirmenci Osama Magdy Mårten Svantesson Agenda: Increase the adoption of Jenkins X Reach out to Microsoft/Google/Amazon about docs, how Jenkins X integrates with cloud providers Needs visibility - talks and blog posts Lists of conferences and such where we can contribute Increasing the number of contributors Internships- Need list of internships & projects Like outreachy, GSoC, GSoD, LFX etc. HackToberFest (Oktober) - issues must be created , and CDF must be informed GSoC status Integration with other CDF projects Would be nice to have a contact person in as many CDF sigs as we can to understand the direction of other projects … Document who from Jenkins X is in what CDF-Sigs Ankit: Supply Chain Sig Christoffer: Best practices SIG Your name here: Add issues for the things we want to do Document TOC in website Migrate notes to HackMD (CDF has account) New name. 35 voted yes, 6 voted no. So, we need to move the notes to the community repo, with links for the survey and results. Create election process - in the future Head of TOC . Mårten was elected Substitute for the Holidays? Head of finance : Christoffer Fund raising? Use opencollective to handle our funds? *Tekton! We should cooperate more with them as it is a very central component in our pipeline :) Debugging Pipelines - most central for us Tekton v1 Coming soon ShipWright - image building tools - could replace kaniko with this ? Shipwright uses Tekton - custom tasks? Tekton Chains - Supply chain Action item: Get Access to Youtube channel for Jenkins X Migrate to HackMD Create events in Bevy - sync with CDF OpenCollective - where is the funds coming to ## July 5, 2022 Participants Ankit Archi Christoffer Dave C Fayez Mårten Rajat Gupta Agenda GSoC update - Rajat & Osama, started working Milestone MVP 1 for UI July 19 Create event in Jenkins X https://www.bevy.com/conference chapter and that should sync to CDF public calender (From fatih) Jenkins X Technical Oversight Committee (TOC) - Open to everybody! Details will be posted 4PM UTC July 6 2022 Archi: want to fix web site! Create a polished web site in Figma Fix mobile view :) Ankit added “Concepts” section to jx-docs Apidocs should be auto-generated Jenkins X domains issue reported to Jenkins security issue https://github.com/jenkins-x/jx/issues/8283 -> go upgrade 1.18 Rajat: suggests to start a Cloud native CI/CD meetup - Check with CDfoundation ## June 21, 2022 Participants: Ankit Christoffer Rajat Gupta Marten Agenda [Automate] issue management - first let’s close old issue (marten already closed a lot of them!) Roadmap - add issues to public roadmap and announce in slack channel Survey results - for Jenkins X name change: 40 answered, 34 voted yes for change, 6 voted no. TOC Ongoing work status Add Jenkins X meeting to CDF calendar June 9, 2022 (Contributor Summit) Participants: Ankit Mårten Terry Ted Brian Hays Kara Gerd Osama Rajat Intro words, summit overview State of Jenkins X CDF Incubating project Four (4) maintainers Ankit Mohapatra Christoffer Vig Mårten Svantesson Tom Hobson Two (2) students from Google Summer of Code (GSoC) https://summerofcode.withgoogle.com/ Osama Magdy Rajat Gupta Public adopters (and many more): https://jenkins-x.io/#users Infrastructure: GCP for build/test New name for Jenkins X Survey to suggest names, Second survey to select a name New logo for Jenkins X (after new name is approved) Discuss with CDF on the specifics of the contest Winner to get tshirts with new name Technical Oversight Committee (TOC) The TOC should Coordinate with CDF TOC, and pick a representative to the CDF TOC Pick maintainers from regular contributors Handle costs and budgets - Do we need a finance team for this in the future? Elect chairman and referee for the TOC (use bot for random selection each time?) Set up LF open collective if someone wants to donate to Jenkins X maintainers Should have Maintainers + representatives from the end user community Biweekly meeting, anyone can join Start the process of becoming a graduated project (no ETA, but hopefully by end of the next year): toc/PROJECT_LIFECYCLE.md at master · cdfoundation/toc · GitHub Structure to Jenkins X office hours Public bug scrub for 10 minutes, if you cant be in the office, remember to update the office hours agenda with a github link and ur name. End users to vote on issues which gets prioritized in the project board Create a Public Roadmap with some timeline so end users know what to expect During office hours, spend 10 mins updating the board. Release process Should we have LTS? It’s hard to maintain 2 variants of JX instead focus more on testing improvements? End users need stability! Better release notes. Look at tekton/terraform (or any other open source project) for inspiration! The problem is with auto bumps not creating good release notes! How about manually pushing a tag, and creating better release notes? Atm end users are blindly updating Jenkins X May be make pre-release automated, but make release manual (gives us chance to update release notes) Integrations with other CD Foundation projects/SIG Ankit - Supply chain security SIG Maintainers - CDF TOC meetings (rotation policy to better manage time) Other maintainers to pick one SIG of their choice Internal SIG updates. SIG UI Other Sigs to be formed once the community becomes bigger Documentation Testing (Plumbing) Security Outreach Current GSoc New Jenkins X UI Supply chain security Future: LFX (if we are selected) https://lfx.linuxfoundation.org/ OutReachy https://www.outreachy.org/ Process improvement TOC to maintain a list of ideas (with prospective mentors) which can be used for internships TOC to maintain a list of internship programs with their deadlines Template for each internship project (starting with GSoC) Maintainers to take turn in organizing the internship Project mentors to write a short blog on the challenges they faced when running the internship Future of Jenkins X Testing improvements UI Supply chain security Cloud and CD Events Improve gitops Should we use a third party for Continuous Delivery or improve our existing codebase? Needs a Key enhancement proposal - (JXEP?) Multi-tenancy (install Jenkins X in existing cluster/multiple instances) Implement best practices ## May 24, 2022 Participants: Ankit Christoffer Vig Rajat Sanyam Mårten Agenda: GSoC External secrets - should upgrade to new version Pusher wave -ditto https://deps.dev to check for vulnerabilites etc. https://deps.dev/go/github.com%2Fjenkins-x%2Fgo-scm ## May 10, 2022 Participants: Ankit Marten Rajat Sayam Agenda Improving tests Current state Gke-gsm Tests run only on new cluster Desired state Short term Gke-gsm Gke-vault Test on newly created cluster Kind tests to run atleast boot jobs on multiple kubernetes versions (Compatibility chart) Long term E2e kind tests running quickstart/import and promotion Test on existing cluster (to test real world upgrade) Create a new cluster off the base branch, and then apply the changes from the PR AWS ASM AWS Vault Azure Vault Azure AKV Other SCM providers (we only test github) Turn on updatebot for certmanager (we are on 1.7.2 now, and 1.8.X is the latest) Tekton upgrade plans(?) We are on 0.29.0, let’s slowly upgrade (chart and codebase) Kubernetes 1.23 and 1.24 support CodeQL alerts in almost all repos Thanks to rajat, aabhaas and osama TOC formation - requirement for graduation Action items Kind tests Tekton upgrade ## April 26, 2022 Participants: Tom Marten Archie Rajat Agenda Trivy db tasks in catalog! https://github.com/jenkins-x/jx3-pipeline-catalog/pull/956 Jenkins X infra re-created deleted jx-git-operator namespace and import resources into state as state is stored locally - still need to move infra state to google storage bucket and may be use atlantis for internal infra! Move to pre-emptible nodes for jx3-version e2e tests (will save 60-90%) Kubernetes 1.22 update Catalog PR merged: https://github.com/jenkins-x/jx3-pipeline-catalog/pull/984 Blog post created: https://jenkins-x.io/blog/2022/04/22/kubernetes-1.22-tekton/ Version stream and tekton chart to be updated soon We will update our tests to use version 1.22, and verify that things are working. Nginx chart upgrade pending We also need to update the jx infra cluster to 1.22 (after validating that the changes work with 1.21 - they should be backwards compatible) Golang 1.17 rollout Catalog PR merged Most packages are now using go 1.17.9 (includes fixes for the 2 CVEs found in 1.17.X) Instructions on how to update go version , so we can split the work next time: https://github.com/jenkins-x/jx-docs/pull/3581 Arm support (darwin, linux and windows) Catalog PR merged (update to go-releaser version - we were using an ancient version) Most binaries (plugins and jx cli) now have arm binaries Actions: Setup blog post for TrivyDB - R&T Jenkins-x way to manage all service pipelines (or some with a filter) at once - Everyone Compatability matrix for plugins / helm charts / docker images for ARM - R & T Lighthouse multi-architecture????? Kaniko doesn’t support, should we review other tooling - M Kubcon Valencia -- Should we come up with a meetup plan? - Anyone going to kubecon ## April 12, 2022 Participants: Ankit Christoffer Dave C E K Rajat Sanyam Ted Youssef Agenda: Introduction from Rajat and Sanyam, Gsoc Jenkins X status from Ankit: The infrastructure is broken! We need help Go update 1.17 in the versionstream. All images are still using 1.15 Security updates from dependency alerts Cert-manager upgrades. Anyone still on the versionstream version of cert-manager? Update-bot should be used to update all? Charts. Dependabot - perhaps use it later Problems with applying all helm chart dependencies each time - builds take too long Should apply only the changed chart (like ArgoCD :) Slow update cycle of dependencies Move all bash scripts/makefile to go (perhaps) Youssef - can we install Jenkins X only CI without the CD part? Use ArgoCD? Dave C - should we break JX into parts? Ankit - scope of JX is huge Youssef - make jenkins-x-argocd project templates etc? Supply chain security SIG - should be built-in - openSSF . Could we swap out Tekton and use Jenkins X instead? Cloud events - can we integrate with other tools with lighter coupling ## March 28, 2022 Participants: Ankit Christoffer Aabhas Osama Excel Anurag Kumar Ted Agenda: GSoC contributors quick intro (if any) Name, interested project A: Google Season of docs application was submitted, now we wait Results to be announced on April 14 Blog post: Project proposal for Google Season of Docs 2022 📄 | Jenkins X - Cloud Native CI/CD Built On Kubernetes 4 volunteers A: kubernetes 1.22 support status JX3 version PR for tekton fails Needs jx3-pipeline-catalog PR most certainly Jx3-pipeline catalog tests are ancient! JX infra broken (James R to fix soon) A: Upgrade tf providers for different cloud providers AWS: Ankit Azure: Tom GCP: James (?) C: Jenkins X Most Valuable Contributor Award 2022 trophy https://github.com/jenkins-x/jx-community/issues/16 Is there a proposal template for GSoC? Vulnerability scanning March 15, 2022 Participants: Ankit Tom Ted Chris Damian Marten Rajat Agenda: A: JX has been accepted into gsoc 2022! Yay! A: Merged tekton PR - will try it locally A: Access (Talks with James R) T: TrivyDB D: Multi Tenancy T: Useful Community Notes A: GSOD Actions: If anyone wants to help mentor, please bring it forward to Ankit, Chris or Tom A&T&C: work on the TOC, Roadmap to becoming a contributor etc Use the crane db tag for pipeline catalog (Trivydb) Review LTS Document release and LTS process T - Documentation on air-gapping cluster T - Email announcements monthly (mailchimp) T - Review how many repos / orgs we need and have (purge some) A - How are bigger projects handling a massive amount of repos? How can we restructure repos E - Review calling softwares (google meet, slack, bevy) A - review diagramming tools to make it more change managed and easier to update T&M - Review container building tools as some issues have been generated with our upgrades Kubecon/CDCon - review attendees and arrange a meetup for those T: Demo trivy on the next office hours ## March 1, 2022 Participants: Ankit Christoffer Damian Dave C Rajat Youssef Agenda: A: Submitted gsoc 2022 application, we should know by March 7, if we got selected or not: https://jenkins-x.io/blog/2022/02/20/gsoc2022-ideas/ A: Project board: https://github.com/orgs/jenkins-x/projects/13 A: Announcements in jx docs website, instead of slack channel (hard to add new channels for external projects) Damien: Multi-tenant jx ## February 15, 2022: Participants: Ankit Christoffer Dave C Damian Terry Tom Mårten Youssef Rajat Agenda: C: Survey results A: Status update on crd api switching to v1 from v1beta1 - completed A: Google summer of code topics from the community A/M/Y: Anything on tekton upgrade? A: Plan to support a range of versions and run tests to verify atleast boot jobs work across different kubernetes versions (may be follow the release calender of EKS: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar )? Actions: A: Prepare project board Everyone: Split survey results into issues in project board Speak to James about merging Tekton Look at re-adding pipeline debugging to Tekton upstream Can we contribute to the Tekton helm charts and bring them up to scratch so we don’t have to maintain our own fork Everyone: start maintaining a budget plan for future infrastructural needs A, C, Tom: Mentoring for GSOC Tom: Proposal for GSOC quickstarts project Tom: Docs page for maintaining and managing cloud providers (having leads for cloud providers) Everyone: When we’re making proposals in future, our cloud agnostic nature is very beneficial to the cdf and they’re interested in best practices, adding this to our proposals should increase our likelihood to obtain more funding Everyone: Get involved in cdf best practices sig work and show how to best use jenkins-x platform for end to end delivery ## February 1, 2022: Participants: Christoffer Ankit Mårten Dave Juan G Youssef Agenda: C: Results of the Jenkins X survey A: tekton chart upgrade to a version which supports kubernetes (get James opinion) A: What are the other issues with JX 1.22? A: How can we do things better so that we dont get in the same situation again (similar things happened with kubernetes 1.16) D: Access to logs? Publicly viewable logs will help new contributors. D: How is the JX infrastructure created? Who has access to it? M: It would be nice to add changelogs to the jx version repo, so it’s easier to generate blog posts about monthly JX updates A: First pass at adding logos Y: Making boot jobs faster (take 30 odd mins to run today) - seems like the helm templating/Download step is the slowest. Action Items: M: look at caching of helm charts as part of the bigger project towards making boot jobs faster (reworking pipelines to use cache to speed up boot jobs, risky (similar to lighthouse)) A: Ask James about tekton chart upgrade (ankit to send DM, CRDs now working but Tekton outstanding) A/M: Access to mail chimp and twitter handle? (mailchimp is cloudbeeds, so unlikely to get hands on it. Twitter is James so can work with him on that) A/M: A new slack channel only for announcements? (speak to kubernetes slack moderators) A/M: Look at adding pluto to the pipelines to catch API issues earlier. (ankit planning to start this week) ## January 18, 2022: Participants: Ankit Tom Mårten Christoffer Youssef Ted Agenda: Action Items A: kubernetes 1.22 support A: We are not compliant with gitops definition (https://opengitops.dev/) - missing reconciliation - I can change things manually in the cluster, and jx wont correct it till I make a new push to the cluster git operator. A: Monthly update blog posts A: Company listings in website. A: Review CDF graduation requirements A: new CDF zoom sub account for meeting: https://github.com/cdfoundation/foundation/issues/356 C: Want to contribute to improving onboarding experience. What are the most pressing issues? Notes: K8s 1.22 Kubenetes 1.22 requires some additions to jx api. Youssef was working with ankit on getting the upgrades completed. Tekton however does not support 1.22 with our current helm chart issues. Breakpoints no longer working in tekton, work has been completed to readd it, but that has paused. Drift detection Helmfile already does a sort of drift detection, but it doesn’t detect drift in charts deployed outside of helmfile. Ankit suggested a kubectl diff, but we’ll investigate this further. Monthly update blog posts It will help keep the blog up to date and draw in new users if they can see blog posts being done. Company listings in website Users of jenkins-x can add themselves to the main website to see who is using jenkins-x and help us to show how many happy users we have and provide confidence to new users. Review CDF Graduation Requirements We should review the criteria and that will give us a better baseline of support and enable us to do more in the future. As we’ve lost cloudbees support we’ll want to get another supporting body. CDF has weekly/bi-weekly meetings that at least one of us should attend. Action Items: Ankit to complete jx-api changes for k8s 1.22 Marten to look at tekton charts / work out where james Ankit to investigate kubectl diff for drift detection Ankit to join cdf meetings ## January 4, 2022: Participants: Ankit (Dexai Robotics) Tom (MQUBE) Christoffer (CoWork) Ted Agenda: Tom: Spot Instances on Azure! Tom: Kaniko issues on azure Ankit: Fix awesome-go requirements (https://github.com/avelino/awesome-go/pull/3971) ! Ankit: https://opengitops.dev/ (mentioned on slack by christoffer vig) Ankit: What is the release cadence for LTS versions? Do we do monthly releases (for example gitlab does that) starting from february or do we do zreleases every 3 months. Ankit: Ask the community about what is the biggest pain point (suggested by Tom) and may be make an issue/roadmap (https://github.com/aws/containers-roadmap/projects/1 for example), so jx users know what to expect when (rough timeline) Ankit: The documentation needs to be more up to date with what works, and what does not. For example: bitbucket and kubernetes versions supported by Jenkins X Ankit: Is it possible to get a security audit of the entire codebase by an external vendor? Can we do it through CDF? Or are there security companies who do audits for open source projects? Before we do that, we should fix all gosec issues though. Notes: Azure Spot Instances Terraform-jx-azure now supports spot instances, there’s jx docs for setting up spot nodes but we still need to look at the jx-boot jobs to run on there. Kaniko The commit hash version we can update the docs to make sure they’re using the latest kaniko for now. Kaniko 1.8 should be released soon, we’ll run a test then push it out to the pipeline catalog. How can we make sure this doesn’t happen again. The Kaniko team have asked about integration testing so that’s probably going to be the best place to put it. Gitops Chris raised that gitops should have a few points All configuration should be stored within Git It should be a pull based model (jenkins-x currently does a lot of pushing) LTS We’re all working on this in our own time, so we need to make sure we’re being as efficient as possible and we’re aware that things are currently broken. The current plan is to rerelease LTS in 3 months, then we’ll move over to 1 month schedules after that. User Feedback We have the project board within the projects section, but we need a decent way to communicate with users, is this just a post into jenkins-x-users, is it a user voice or github issue solution. We should propose this within jenkins-x users. Permissioning There’s a gap currently with jenkins-x and how to integrate with large orgs. Jxv2 supported this, but v3 doesn’t have a lot of integration. Ted has created some documentation around this and we’re hoping to push this to the docs site to show that we are working on it, but we all think this would be a useful addition to jenkins-x v3. Documentation Documentation is always welcome and will very rarely be turned away. There’s a lot of v2 docs that aren’t relevant anymore or don’t have a relevant alert for v2 being deprecated and unsupported. Unfortunately there isn’t the bandwidth there to be able to support v2 and all of the maintainers are using v3. We should remove the semantic check for jx-docs as it’s a barrier for entry that’s not really needed. Supported tooling (K8s versions, SCM providers, CD providers, Cloud providers) Give people a good set of pages to be able to tell if their toolset is working with k8s Maintainers How do we foster more contributions from the community, now there is nobody being paid to work on jenkins x we need to make sure there is a low barrier for entry for maintainers. We can get more people on the office hours if we move it an hour earlier. Action Items: A: Azure OSS credits for BDD tests Join OpenGitops group Review v2 and depreciation notices T: Submit permissioning integration Speak to james about removing semantic check for jx-docs as it’s a barrier for entry that is now defunct Draft a supported tooling page CDF speak about security audit Move Office hours back an hour