B-06163: Research running Selenium on Jenkins

B-06163: Research running Selenium on Jenkins === ###### tags: `B-06163` 'Selenium' 'Jenkins' :::info - **Location:** Main Conference Room - **Date:** June 18, 2019 11:00 AM (MDT) - **Participants:** - Shane Hirsekorn - Chrisanna Gordon - Nate Beaver - Tyler Seibert - Steve Szettella - Pietro ::: :loop: Implementation Stages to Consider: -- ### Running selenium test code in Jenkins - [ ] Run the UI and execute python tests seperately: - Running the UI by booting a container using NPM - Notes: - Given the current architecture, Jenkins checks out from `BitBucket` (for webui code) and `Subversion` (for jaws-trunk/integration) code in order to run the tests in the pipeline. - Running NPM creates a process that listens on a port, shutting it down is not as straightforward as simply running it in the background and shutting it's PID. - This is hard to handle at the Jenkins level since there are poor/limited workarounds handling process started by their pipeline. The working solution is to shut processes being listened in a particular port. - My current example toy code runs the UI and Automation Tests in parallel in Jenkins so that the Jenkins `pipeline.log` doesn't scramble the output. - Problems: - Port is hardcoded to '3333' or whatever port is specified for a particular test, this means that two builds cannot be run in parallel or they will run into a port conflict. This isn't a problem in first examples but could potentially be solved with `Docker Compose` encountered later. - Capturing the PID properly to terminate this process. There are two options: - Scripting the solution in Jenkins to handle possible port collisions - Adding a `npm run serve:stop` script in the `package.json` scripts could fix the falsely failed build problem in Jenkins. - Both options need some Jenkins/Pipeline logic to pool when the server is up, tests are done, and termination is possible using a shell script command like `curl` or `wget/grep`. Whatever logic the pipeline uses a `npm:stop` script, pid killing or terminating a docker container must happen in Jenkins. - It's important to stop the NPM gracefully since otherwise it will flag the build as failed falsely. Therefore it is important that the server's configuration is changed to handle a shut down signal and return a non-zero exit code. - [ ] Executing the selenium tests - Notes: - The tests require python3 and use the PyUnit test framework, and various python external modules (such as selenium webdriver). This raises the problem of ensuring Jenkins recognizes all of these (currently solved with a Jenkins python plug in, `ShinningPanda`). - Alternaitve: Installing it in the CLI. ### Running headless configurations - [ ] **Running selenium tests headless** - There are many frameworks that can run selenium tests. Currently we do it utilizing the `unittest` framework in python - There exist frameworks outside the standard libary such as `seleniumbase` that use the `selenium web-driver` and `pytest` libraries to provide wrapper around selenium for more features like snapshots. It also can create .xml test suite reports for Jenkins to simply display in the build. - [ ] **Running Chrome/Firefox headless** - Chrome/Firefox are the only two browsers that had I could find which could be easily configured to run and turn off graphical displays through option configurations. - [ ] **Running a xvfb server with a pipe that displays to `dev/null` in other words, performs graphical operations without showing any screen output.** - Plug ins: This particular test/server framework is supported by an up-to date plug in in Jenkins and the python`seleniumbase` framework. - How it works with our current JAWS architecture - [ ] *Paid services like: Saucelabs, BrowserStack, Cloudbees, Google Platform would possibly a wider variety of browsers type and version/tests.* - Seems like some people use them, some don't, they cost money and arguably could just be avoided with a bit of effort ### Dockerizing the tests - [ ] **Launch Chrome in a Docker Container** - Pointing python tests into a docker container running a particular browser. - [ ] **Lunch any browser in a Docker Container** - There are 12 available hub images available from a public Selenium Docker Hub repository. - https://hub.docker.com/u/selenium - [ ] **Additions to Docker Configuration.** - Using a `Dockerfile` - It is possible to use the `Docker Slaves` plug in in Jenkins to run the Dynamic Jenkins slave with the Docker Slave plugin. - [ ] **Using Docker Compose:** - Creating a`docker-compose.yaml` would allow us to scale various drivers (chrome/firefox/ie), applications and tests into a simple Jenkins step: - `#sh docker-compose up --build` - It is possible to write a script to ensure web drivers are up or any other component is finished before the other executes. - It is also possible to easily execute commands inside the `docker-compose.yaml` (and build those images) simplifying `Jenkinsfile` scripts. - This would entile composing the entire build, thus dockerizing the `browsers` - When testing is finished, the step: `docker-compose down` can be used to stop and delete running containers. - Containers can be isolated and each component's status can be checked for a `ready` state. ### Reporting the results of selenium tests - [ ] **Reporting/Recording output in the build screen.** - Using the `xUnit` plug in which is already downloaded in our Jenkins(local)/webui server any .xml will be displayed in both the Jenkins client and Blue Ocean's `test` tab. - Regardless of the python test framework used (i.e: `unittest`, `nose` or `pytest`) it is easy to configure a JUnit xml output in a build folder which is compatible with the xUnit plug in. - `pip install unittest-xml-reporting` is needed for the current unittest framework used, xml reporting is built in with `pytest`. :closed_book: Possible Stories: -- *Format:* ==Order== (1 - 5) / Story / **Estimate** (1, 2, 3, 5, 8, 13 ...) ### Developing - [ ] ==2== Create a `python3.x` Docker with all the libraries and modules needed. - [ ] ==2.5== Add our code to our python docker (testable). **1** - [ ] ==2.5== Make the test project ready for Docker using a `dockerfile` based on a `python3.x` image. (Ensure test files are copied to image folder, made into a working directory and potentially install test project libraries and all desired python modules are executable) **2** - [ ] ==4== Creating Docker images for webserver, tests and web-browser. **5** - [ ] ==4.5==Executing selenium tests inside Docker container **2** - [ ] ==4.5== Connecting web browser driver to executable tests **3** - [ ] ==5== Pointing tests to webui `npm server:container` **3** - [ ] ==7== Use a `user defined network` and mount a `volume` into a container with the test runner - [ ] ==8== Write a docker-compose file for our intergation tests architecture. - [ ] ==9== Integrate Docker Compose steps into Jenkins ### Research Involved - [ ] ==1 [Spike Story]== Learn about the design practices and creation of Dockerfiles **?** - [ ] ==3== Research the implementation details behind a Docker Compose and customizing dependencies to `webui` webserver, `jaws-trunk` implementation tests and any necessary docker web browser **3** - [ ] ==6== Scaling Docker container build set up using Docker compose and creating `docker-compose.yaml` files. **3**