--- robots: noindex, nofollow --- # Workshop with Ops - 02.12.2020 :movie_camera: [Link to video (inside Insee)](https://minio.stable.innovation.insee.eu/groupe-908/atelier-exist-2020-12-03.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=1RLZFNYFOEDES3IY3P1W%2F20201203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201203T164849Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiIxUkxaRk5ZRk9FREVTM0lZM1AxVyIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJhdWQiOlsia3ViZSIsIm9ueXhpYSJdLCJhdXRoX3RpbWUiOjE2MDcwMTQwMTQsImF6cCI6Im9ueXhpYSIsImVtYWlsIjoibWljaGFlbC5jYW5hbGVzQGluc2VlLmZyIiwiZXhwIjoiMTYwNzA1NzIxNSIsImZhbWlseV9uYW1lIjoiQ2FuYWxlcyIsImdpdGxhYl9ncm91cCI6WyIxMTk3OlNORElMIiwiNjI0OmFwcGxpY2F0aW9ucy1zbmRpLWxpbGxlIiwiNDU1OmdlbmVyaWMiLCI5MDg6R01QQyIsIjYyMjpvZmF0cyIsIjgyMDpvcmJlb24iLCIxMjYxOlJHSFZQMCIsIjQ0Njpjb2x0cmFuZSIsIjkwOTpTTkRJTC1DaGVmcyBkZSBncm91cGVzIiwiMTk2OmNvbW11bmF1dGUtamF2YS1zaW5sIiwiNjYxOkluZm9ybWF0aXNhdGlvbiBkZXMgZW5xdWV0ZXMiLCI1MTE6R1Qtc3VwcG9ydC1leHBlcnRpc2UtU05ESUwiLCI5Nzg6c2ljb3JlIiwiMTI1OTpJbnRlZ3JhdGlvblNORElMIl0sImdpdmVuX25hbWUiOiJNaWNoYWVsIiwiaWF0IjoxNjA3MDE0MDE1LCJpc3MiOiJodHRwczovL2F1dGguc3RhYmxlLmlubm92YXRpb24uaW5zZWUuZXUvYXV0aC9yZWFsbXMvYWdlbnRzLWluc2VlLWludGVybmUiLCJqdGkiOiIxNzk3MzZlOS0wZmY3LTQ1NzUtOGZjYi04YzMxNTZkOGY4ZjUiLCJuYW1lIjoiTWljaGFlbCBDYW5hbGVzIiwibm9uY2UiOiIwMmI4YzExNC1iZmVhLTQ4MWYtOGNkZC1kOGQxN2QzNTRkZWYiLCJwb2xpY3kiOiIiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJmaGRkaGEiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib255eGlhLXVzZXIiXX0sInNjb3BlIjoib3BlbmlkIGdpdGxhYi1ncm91cHMiLCJzZXNzaW9uX3N0YXRlIjoiOWM1M2YwYjAtZjFhYS00N2VkLTg0M2UtMzE1NTkyNjY3N2I4Iiwic3ViIjoiZjoxODA2YjRjMS01Nzk4LTRiOTAtYTUxMi04NDMzYzUwYTUyNDk6RkhEREhBIiwidHlwIjoiQmVhcmVyIn0.u5SYeHGFlKAqsori-FbckcHP4U8BwEAmL830qS5egJVd-ib4mgDMm5jmGu-knU7BqMBSCEfW6HkVlhFeglFTkg&X-Amz-Signature=8c30abeb2208a2ef5dff61aacecbddafd6a06d35cba0007dd9ac381ec480b897&X-Amz-Security-Token=eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiIxUkxaRk5ZRk9FREVTM0lZM1AxVyIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJhdWQiOlsia3ViZSIsIm9ueXhpYSJdLCJhdXRoX3RpbWUiOjE2MDcwMTQwMTQsImF6cCI6Im9ueXhpYSIsImVtYWlsIjoibWljaGFlbC5jYW5hbGVzQGluc2VlLmZyIiwiZXhwIjoiMTYwNzA1NzIxNSIsImZhbWlseV9uYW1lIjoiQ2FuYWxlcyIsImdpdGxhYl9ncm91cCI6WyIxMTk3OlNORElMIiwiNjI0OmFwcGxpY2F0aW9ucy1zbmRpLWxpbGxlIiwiNDU1OmdlbmVyaWMiLCI5MDg6R01QQyIsIjYyMjpvZmF0cyIsIjgyMDpvcmJlb24iLCIxMjYxOlJHSFZQMCIsIjQ0Njpjb2x0cmFuZSIsIjkwOTpTTkRJTC1DaGVmcyBkZSBncm91cGVzIiwiMTk2OmNvbW11bmF1dGUtamF2YS1zaW5sIiwiNjYxOkluZm9ybWF0aXNhdGlvbiBkZXMgZW5xdWV0ZXMiLCI1MTE6R1Qtc3VwcG9ydC1leHBlcnRpc2UtU05ESUwiLCI5Nzg6c2ljb3JlIiwiMTI1OTpJbnRlZ3JhdGlvblNORElMIl0sImdpdmVuX25hbWUiOiJNaWNoYWVsIiwiaWF0IjoxNjA3MDE0MDE1LCJpc3MiOiJodHRwczovL2F1dGguc3RhYmxlLmlubm92YXRpb24uaW5zZWUuZXUvYXV0aC9yZWFsbXMvYWdlbnRzLWluc2VlLWludGVybmUiLCJqdGkiOiIxNzk3MzZlOS0wZmY3LTQ1NzUtOGZjYi04YzMxNTZkOGY4ZjUiLCJuYW1lIjoiTWljaGFlbCBDYW5hbGVzIiwibm9uY2UiOiIwMmI4YzExNC1iZmVhLTQ4MWYtOGNkZC1kOGQxN2QzNTRkZWYiLCJwb2xpY3kiOiIiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJmaGRkaGEiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib255eGlhLXVzZXIiXX0sInNjb3BlIjoib3BlbmlkIGdpdGxhYi1ncm91cHMiLCJzZXNzaW9uX3N0YXRlIjoiOWM1M2YwYjAtZjFhYS00N2VkLTg0M2UtMzE1NTkyNjY3N2I4Iiwic3ViIjoiZjoxODA2YjRjMS01Nzk4LTRiOTAtYTUxMi04NDMzYzUwYTUyNDk6RkhEREhBIiwidHlwIjoiQmVhcmVyIn0.u5SYeHGFlKAqsori-FbckcHP4U8BwEAmL830qS5egJVd-ib4mgDMm5jmGu-knU7BqMBSCEfW6HkVlhFeglFTkg) [TOC] ## Program of the workshop - observation on live of the production database behavior via the Monex tool - observation of the metrics collected over a longer time --- ## Monex monitoring => issue in production > Gharib says that sometimes going to the dashboard may crash the database it appeared 3 or 4 times in the past ... > => Lars never heard of that issue > => Perhaps due to memory given to ExistDB > machine memory (cf. last workshop) *Idea : Reproduce on non production system with the same memory (+give logs to ExistSolutions)* *Lars on Monex Use* - For admin perspective : we don t rely on Monex but can be useful for a first look. - Important for developpers : profiling and indexes use. Cedric : introduction to Monex => see first workshop sum-up ## Long term metrics (centreon) **Memory usage** - ExistDb constantly using 16GB. - After reboot it instantly uses 16GB that may be a problem. - We should reboot from time to time but not everyday ... Lars => long term metrics : load average // CPU // swap dont't show problem of load **Around the db crash on 2nd october** - When reboot are we sure that Db was unavalaible for users ? Not sure. - Reindexation caused huge loads *Recommandations :* - Wrapper shutdown to 600 secondes (if not java wrapper can hard kill the DB which causes reindexation) - Insee needs to set up stress test and intégration tests to decrease the RAM consumption to go down to 8 or 12 gb ... - @Insee can send the GC logs files to Exist Solution to see how often the GC is triggered (last week + around real crash => nov 2019) + exist.log files or the complete log folders. The decrease of RAM was not done by Insee until now because of the RestXQ issue which caused memory leak. This problem is about to be resolved with the abandonment of RestXQ. ## Integration and stress test ?? > *Way Exist Solution could help INSEE on integration and stress tests?* Beginning with POST and GET, use monex to monitor and generate JMeter tests. ExistDB could offer a naked setup (example of integration test) to integrate to our project for testing purpose. To have test to run every night. Lars proposes to have a worshop for integration and generate load tests. Recommendation is to use a dedicated server for this integration tests with use of Jenkins. Integration tests for daily use, for most important changes (ex version update) stress tests are recommanded. Question @Insee : possible use of Jenkins for integration test ? > [name=Benjamin Bour] Few month ago, Insee has decided to migrate all ci cd jobs from Jenkins to Gitlab CI. It won't be possible to use Jenkins for integration test. > [name=Benjamin Bour] Our internal software factory Gitlab uses Gitlab Runners with one executor shell (including jdk8, jdk11, maven for java project and node, npm for js project) and one executor docker (to build custom image or pull image from Docker Hub and run containers inside job). Currently we have the possibility to deploy in Mesos Marathon (not in production, just in sandbox mode). We plan to implement Kubernetes next year (for now still in sandbox mode as well, not in production). Gitlab CI/CD can be used ? Lars will discuss ! ## Good practices for developpements - at Insee now : devs are directly working with Exide ... we don't have not a lot of good pratices , no automated tests, no performance tests... Lars proposal : ***next year a workshop*** with developpers : how to work with ExistDB (use of Visual Studio Code, automate push to dev server, use of Docker ...) Post meeting question @Insee : Use of Docker on desktop ? > [name=Benjamin Bour] This subject is in the pipeline at Insee :) but not for the moment. Lars: Use of Oxygen is ok just for XSLT debugging or large XML. Some use it for XQuery, it can be linked to ExistDB. Lars will send proposal for next workshop CI/CD, stress test. Logs of garbage collector around a crash (nov 2019 ?) and the last week + exist.log files for the 2 DBs. Reviewed by - [x] Michael Canales - [x] Benjamin Bour - [x] Guylène Tandeau de Marsac - [ ] xxx - [ ] xxx