# How is it like to bring DevOps to a software factory on central america

Hi Everyone, I’ve created this talk based on histories from the communtiy and my experince colaborating with companies from a just me size to be part of one of the biggest companies in central america, I've also colected some stories from folks working on Banking, Telco and Goverment here but first what is Central America?

Central America is a geographic region between North and South America, conformed by 7 countries, Guatemala, Belice, El Salvador, Honduras, Nicaragua, Costa Rica and Panama with a sumarized GDP just bellow Colombia and English Proficiency low as most of the countries on latin america affecting the access to information and education.

A Software factory atleast here at Central America is a company or a department that his main role is create and sell software, it includes the analysis, specification, design, implemenentation, deployment, testing and usually the opeation of software products.

Cloud, Agile, Devops, SoftSkills

So why to talk about this, here most of my colleges use to read and hear about big large scalle northern companies success histories implementing DevOps, usually with a large mature software department, top cloud technology and of course a good budget.

But what about small companies over here, is it possible with our mature level? could we change how we all have been doing stuff since ever? where to start? should I learn about dockers? kubernetes? microservices?

And yeah, Before 2020 the agile methodology adoption in the region was still a thing, dont say cloud native tech, the software factories inside large financial, retail and service companies had the fame of be old school ones dont say goverment, with a resistance to change, with a slow development process over dusty servers running dusty apps that it is working please don't touch it, dont look at it, don't even think about because it can stop working.

I remember being talking with members of the community arguing that cloud and those dockers looks pretty but nobody uses that here, there was no opportunities for Cloud Engineers, microservices or kubernetes.

But then in a matter of weeks everything changed, companies arround the world was in a hurry to move their business to remote and online first and that was of course a challenge having in mind that most of those companies had never built a propeer e-commerce, the necessity for engineers increased and as remote was so poppular the world look at us and a strong human capital flight started, I know a lot of folks here are now working with international companies, good pay and good culture they say.

Meanwhile local banks and retailers had to duplicate their capacity on online and mobile platforms, and teams and technology was no prepared for that of course the necessity for a digital transformation was evident, sadly the necessity for a cultural transformation was not that evident, teams inside those companies had learn on record time how to improve perfomance, implement automatization, scalabillity, observavlity and improve their english because of course you need to read documentation, watch youtube videos and deal with cloud provider support, because there is usually no spanish l2 or l3 support available.

A lot of companies acquired expensive pieces of cloud that says devops all overthere, tons of container layers was built, multiple kubernetes instances deployed, ci here, cd there and bualaaaa "DevOps"! they said.. for a while. Once something needs to be updated or migrated old problems come across again, and for some reason by then the one that build the stuff is usually not longer in the company or the support was too expensive.

Experienced IT engineers are usually moving to companies outside the region, here right now if you are a developer, familiar with the cloud and with a descent english you are very likely to be working on a foreign company or atleast with a lot of linkedin mesagges, and I would say thats why dev teams are mostly aged beteen 23 and 35 years old with and an average of 4 years of experience

Believe me its getting hard to hire talent and experience overhere, but it has oppened oportunities as some companies are starting to increase their salaries or create internships that could help them to train jr develpoers on technology and methodology but the company needs to make that investment and takes time.

There are tech communities that helps by creating spaces to share about technology, and talk about our experiences, but the community culture is something that is not spread just on hads of a few, adictionally most of the movement and community meetings stopped on late 2020 even CNCF communites are dessapearing and now there is only 20% percent of the communities that existed before pandemics.

On mid 2021 I was giving an online talk at the Kubernetes Comunity on Guatemala, one of the assistants contacted me, said that the company he was working on just started with a digital transformation and required help to adopt and automate, there was some stuff already running on cloud and brace yourself because proyects are comming, offering me a position to help with cloud adoption as DevOps.

By then I was working for a small company on Netherlands as Cloud Engineer, I had plans to move there but the proccess was on hold because of pandemics so I had to decide, and I choosed go for the new challenge, two months later I was there about to know how is it like to bring devops to a software factory on central america.

I was involved on a lot of meetings they pressented me as the DevOps that I was going to help teams with new standards, security and cloud adoption, please ask the devops if they need help to deploy to the cloud

As you might thing I had tons of other meetings sitting there with devs to understand their workloads, building containers toguether and explaining more about what I do and what devops prettends, it was clear most of they knew it had something to do with automation and cloud but not sure about what else.

On my first month there I was invited to a knowledge transfer meeting, one of the engineers with more experience on some critical products was leaving so there was a meeting to understand his work, important processes, tasks and attributions, you know. I will never forget that when the team pressented me and explained that I was there to help with DevOps adoptaion he said "Oh, welcome and good look with that" with that tone of "welcome to the jungle you are gonna die", during the next 2 hours he moved arround a number of documents on a shared drive with information and details about tasks and projects he had done over his 5 years on the company, it was evident that the number of attributions, the lack of propper documentation and the ambiguous role definition limmited him to move and evolve inside the company.

It's common to find complex components built by and arround an engineer or an small team of engineers, usually becasue bad or unilateral descitions comming from bussiness side and engineers has the mindset that things has to be done, nomatter, usually affecting perfomance, quality and documentation.

Some weeks later I was there conversating with a developer, he asked about my experience working with an european team and why did I left, implying that he wouldnt leave a job like that to move there, I had to explain that we where innovating, adopting new methology that for most on the region was just a good to have, challenge was interesting, bring devops to a multicultural team in central america, the company was in the digital transformation phase and there was multiple oportunities to learn and prove it can be done!

Of course it has been possible no because of me, no because of him, it was because the entire department has been committed to archive it.

Toguether with the cloud adoption and agile methodologies the software factory was refactoring the QA Department and creating QA Framework for automated testing, I lean a lot helping that team to adapt kubernetes and VM loads, it also required a transformation on the QA team as most of the QA Analysts has been focused on manual testing and was lacking of technical knowledge, experience with programmming languages was added as a required skill and the existing team started a some training to leverage their skills.

During the implementation and adoption a new elite team emerged and started to help teams with the adoption as well, some of them had never interacted with a linux terminal and now they configure nginx, deploy containers and move arround with no problems, documentation was crucial as they are now helping our servers team to get in touch with platforms and perform configuration tasks on produciton servers.

We have also started with trainings accross the organization, explaining the problems we want to attend and why devops is important, and as we wanted to capt everybody attention I made a talk about DohOps with the simpsons struging with the same problems we had and his journey to devops, with an strong inception of DevOps = Culture.

### Technolgoy and methodology

Not just automation...
Not just measure...
Its about communication and collaboration
Its about get hands dirty.

One of the critial things we had to archieve was improve the user story definition, the adoption of Gherkin make the scope of each feature very specific helping develpors and Team Leaders to have a better understanding of the project status.

We moved from
"Implement table filters and sort"
to
Feature: Implement table filters
Scenario: Filter stores by country
When the country filter input changes
Then Stores on table must be filtered by the country input
Scenario: Sort stores by name
When the name column header is clicked
Then: Stores on table must be sorted by name and ascending

Now the company moved from a 10% of workloads on cloud on 2019 to 80% on Q2 2020, the time it takes to create, deploy and test a single feature moved from 2 weeks to 1 day, not just because the process is better but because the feature definition improved and teams evolved

We are still working on the methodology adoption and imporovement of our processes and while we had a good progress with the adoption now we are conciliatig our activities with security team, lets say someone forget to involve them on many meetings.

But we did noticed that the team improved his trust on new metodologies, his trust on team mates and more important his trust on themselves, know they know that its not just ok to fail, its important to fail because we can learn and do it better the next time.

And I want to finish with a message to all my folks: Confien en ustedes y asegurense de poder confiar en sus equipos, estamos parados frente a una de las mayores transformaciones tecnologicas en nuestra region y se necesitan y necesitaran personas como ustedes que ayuden a lograrlo, no es facil, tomara tiempo pero creanme que tenemos lo que se necesita, solo necesitamos creerlo nosotros primero.

Gracias.