Adoption Journey Document

[mikecali]

Now that GitOps is getting a lot of traction, we got a lot of questions from different parties how to start their own GitOps journey.
I think this warrants a dedicated document of what are the things to consider before even thinking of adopting GitOps to avoid disaster or bad taste.

Something I have discussed with my colleagues are GitOps pre-requisites and we ended up with this list.

1. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risk)
2. Automation Maturity - Infrastructure as Code at the minimum
3. Executive sponsor
4. Champions / CoP - a team that can ‘go first’
5. Standardisation (can’t have 20 x Git sources of truth)
6. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
7. Matrix view of crawl / walk / run (ie start w infra, then apps, then network)
8. K8s has to be on the roadmap - how do we get ready for that
9. Start Small!

Happy to expound this and contribute if needed.

[lloydchang]

A suggestion is to invert your list so it begins with "Start Small!" towards bigger "Communication Strategy":

1. Start Small!
2. K8s has to be on the roadmap - how do we get ready for that
3. Matrix view of crawl / walk / run (ie start w infra, then apps, then network)
4. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
5. Standardisation (can’t have 20 x Git sources of truth)
6. Champions / CoP - a team that can ‘go first’
7. Executive sponsor
8. Automation Maturity - Infrastructure as Code at the minimum
9. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risk)

WDYT? @mikecali @sunil390 @OpsM0nkey @scottrigby

[mikecali]

I agree.
Looks good to me.

[lloydchang]

@mikecali @sunil390 @OpsM0nkey @scottrigby I recommend adding "Independent deployability" and "Rapid application deployment" to the adoption journey document as numbers 2 and 3.

---

Hence the list becomes:

1. Start Small!
2. Independent deployability
3. Rapid application deployment
4. K8s has to be on the roadmap - how do we get ready for that
5. Matrix view of crawl / walk / run (ie start w infra, then apps, then network)
6. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
7. Standardisation (can’t have 20 x Git sources of truth)
8. Champions / CoP - a team that can ‘go first’
9. Executive sponsor
10. Automation Maturity - Infrastructure as Code at the minimum
11. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risk)

---

As references:

(2014) from https://martinfowler.com/articles/microservices.html

independently deployable by fully automated deployment machinery

(2014) from https://martinfowler.com/bliki/MicroservicePrerequisites.html

Rapid application deployment

(2014) from https://blog.cleancoder.com/uncle-bob/2014/09/19/MicroServicesAndJars.html

You don’t have to coordinate with some huge deployment effort

(2019) from https://blogs.oracle.com/javamagazine/post/design-change-tolerant-software-with-cloud-native-patterns

Finally, something I’ve already hinted at is creating smaller units of deployment. Rather than a deployment encompassing a huge portion of your ecommerce system—for example, the catalog, search engine, image service, recommendation engine, shopping cart, and payment-processing module all in one—deployments should have a far smaller scope. You can easily imagine that a new release of the image service poses far less risk to the business than something that involves payment processing. Proper componentization of your applications—or as many would call it today, a microservices-based architecture—is directly linked to the operability of digital solutions.

(2019) from https://www.oracle.com/oce/dc/assets/CONT9DD07156DE4043BF8BA54229540143FC/native/107876-ebook-cloud-native-patterns-oracle.pdf

Your goal is to release frequently, and monolithic software doesn’t allow this; too many interdependent pieces require time-consuming and complex coordination. In recent years, it’s been soundly proven that software made up of smaller, more loosely coupled and independently deployable and releasable components (often called microservices) enables a more agile release model.

(2019) from https://www.oreilly.com/library/view/monolith-to-microservices/9781492047834/ch01.html

Independent deployability is the idea that we can make a change to a microservice and deploy it into a production environment without having to utilize any other services. More importantly, it’s not just that we can do this; it’s that this is actually how you manage deployments in your system. It’s a discipline you practice for the bulk of your releases. This is a simple idea that is nonetheless complex in execution.

Tip: If there is only one thing you take out of this book, it should be this: ensure you embrace the concept of independent deployability of your microservices. Get into the habit of releasing changes to a single microservice into production without having to deploy anything else. From this, many good things will follow.

To guarantee independent deployability, we need to ensure our services are loosely coupled—in other words, we need to be able to change one service without having to change anything else. This means we need explicit, well-defined, and stable contracts between services. Some implementation choices make this difficult—the sharing of databases, for example, is especially problematic. The desire for loosely coupled services with stable interfaces guides our thinking about how we find service boundaries in the first place.

(2020) from https://www.infoq.com/news/2020/05/monolith-decomposition-newman/

focus on the outcome, not the technology, and to always remember the goal is independent deployability

(2020) from https://www.theregister.com/2020/03/04/microservices_last_resort/

I should be able to deploy that shipping service into a production environment without having to change the rest of the system. This helps to ship software more quickly and it helps our teams work in a more autonomous fashion.

(2020) from https://gotopia.tech/bookclub/episodes/moving-to-microservices-with-sam-newman-and-martin-fowler

So, you need a reason to push for microservices. What would be your top three reasons that you think are valid reasons that people should be thinking of going down that path?

Yes. Number one is independent replayability and specifically, I would say taken to the extreme would be zero downtime independent deployability. So, the ability to make a change to a piece of functionality, deploy that functionality, and do so in a way that the rest of the system doesn't need to change, so that limits the scope of each release, but also implement that deployment in such a way that I get zero downtime deployment. So, that's really useful if you think about, you know, a SaaS based business where you can't afford downtime, and that's really, really key. So, I think that's kind of the big clear winner.

(2021) from https://www.oreilly.com/library/view/design-patterns-for/9781492090700/ch01.html

For the purposes of this book, we take a bottom-up approach to defining cloud native. We look at all the characteristics of cloud native applications, across the board, by going through every stage in the life cycle of a cloud native application—including design, development, packaging, deployment, and governance. Based on those characteristics, we’ve come up with the following definition:

Cloud native is building software applications as a collection of independent, loosely coupled, business-capability-oriented services (microservices) that can run on dynamic environments (public, private, hybrid, multicloud) in an automated, scalable, resilient, manageable, and observable way.

When your time kindly permits, perhaps you can read and/or listen to the above snippets then dive deeper into the links. Thank you!

[mikecali]

@lloydchang The idea of the list I shared is to put a recommendation around what an organization needs to consider or lined up before getting into GitOps adoption.

My opinion is that the 2 points you added might be encapsulated under this: Matrix view of crawl/walk / run (ie start with infra, then apps, then network)?

[lloydchang]

Thank you @mikecali

"(ie start w infra, then apps, then network)"

Two out of the three examples, "start w infra" "and "then network", aren't in the scope of GitOps' matrix view of crawl / walk / run. They are in the scope of K8s.

By incorporating your opinion, I'm recommending that the new list looks like:

---

1. Start Small!
2. K8s has to be on the roadmap - how do we get ready for that (i.e. start w infra, then network)
3. Matrix view of crawl / walk / run (i.e. Independent deployability, Rapid application deployment)
4. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
5. Standardisation (can’t have 20 x Git sources of truth)
6. Champions / CoP - a team that can ‘go first’
7. Executive sponsor
8. Automation Maturity - Infrastructure as Code at the minimum
9. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risk)

---

WDYT?

[lloydchang]

@mikecali As for:

5. Standardisation (can’t have 20 x Git sources of truth)

I recommend the Standardisation example be changed to:

5. Standardisation (i.e. Choose a standard from the options in Argo CD and Flux CD to structure your repositories)

For example:

Argo CD https://argo-cd.readthedocs.io/en/stable/user-guide/best_practices/#separating-config-vs-source-code-repositories

Option A. Separating Config Vs. Source Code Repositories

Flux CD https://fluxcd.io/docs/guides/repository-structure/

Ways of structuring your repositories
Option B. Monorepo
Option C. Repo per environment
Option D. Repo per team,
Option E. Repo per app

WDYT?

---

FWIW, the following are trivia:

• Git is flexible, and the 20 x Git repositories can be combined via git subtree, or git submodule, or Repo, or depot_tools

• One can have 20 x Git sources of truth because Git is distributed and decentralized by design, and a Git commit is immutable with a unique SHA.

Two objects colliding accidentally is exceedingly unlikely. If you had five million programmers each generating one commit per second, your chances of generating a single accidental collision before the Sun turns into a red giant and engulfs the Earth is about 50%.
https://github.blog/2017-03-20-sha-1-collision-detection-on-github-com/

[OpsM0nkey]

Could I also recommend that we add something about considering the right skill set at the start of the journey? GitOps is the primary enabler of a software driven operating model, so the ability to think like a software engineer (or at least understand SDLC principles) is vital - i.e branching, pull requests etc...
I think it's almost assumed knowledge, but there are tons on traditional infrastructure people out there there that can't comprehend the power of the pull request just yet...

[lloydchang]

Good point @OpsM0nkey Related, not all software engineers use pull requests (PRs). PR isn't inherit in Git at all. PR is a feature in code review tools such as GitHub, GitLab, Bitbucket, Gitea.

[lloydchang]

Depending on the Git implementation in use for GitOps, there may need to be a walkthrough. For example:

Basic Gerrit Walkthrough — For GitHub Users
Note: This document aims to provide a concise description of the core principles of code review in Gerrit for people that were previously using Pull Requests on Github or similar concepts. Nothing in this document is meant to state that one or the other might be better, but only aims to help new users understand Gerrit more readily. We use Github as the point of comparison since it seems to be the most popular service.
https://gerrit-review.googlesource.com/Documentation/intro-gerrit-walkthrough-github.html

[OpsM0nkey]

Good point @OpsM0nkey Related, not all software engineers use pull requests (PRs). PR isn't inherit in Git at all. PR is a feature in code review tools such as GitHub, GitLab, Bitbucket, Gitea.

It's a very good point - PRs aren't a feature of Git, but rather of the Git based platforms. But they're not just for code review (albeit, it's an important aspect) - they're a notification that a change is incoming and a discussion forum for the intended change. IMO, they're an important part of what makes GitOps so effective for scalable operations. When utilised properly, a PR takes away the need to maintain certain traditional ITIL change management practices (such as CAB) - ie the Pull Request is the Change Request, and all of its links, discussions and approvals provide the necessary assurances that a change is good to go.
That's not to take away from the main point of this thread, other than to say that understanding of software development practices (like branching, PRs) is an important consideration when adopting GitOps.

[mikecali]

I totally agree. Should we actually have a separate section called "GIT Best Practices"
I actually started thinking about it and here is the first stub of what I thought we can add.

GIT Best practice

Don’t commit code as an unrecognized author. Git needs to be a source of truth and allow auditing of all actions.
Ensure all secrets are stored encrypted. Use 3rd party secrets Management tools
Don't commit huge files or binaries into the repo. Consider using binary repository managers.
Use git ignore when appropriate and make it clear.
Clean up stale branches and Repos
Create a Branch naming convention that's scalable
Enable security alerts and cleanup old users
Scan for vulnerabilities and secret commits (Example AWS Keys)```

[OpsM0nkey]

@mikecali I like it 🙂

[mikecali]

@lloydchang @OpsM0nkey

I think this is really good now? I am happy with this list. Should we move this forward?

1. Start Small!
2. K8s has to be on the roadmap - how do we get ready for that (i.e. start w infra, then network)
3. Matrix view of crawl / walk / run (i.e. Independent deployability, Rapid application deployment)
4. Enablement strategy - Git enablement, Ansible enablement, DevSecOps Enablement
5. Standardisation (can’t have 20 x Git sources of truth)
6. Champions / CoP - a team that can ‘go first’
7. Executive sponsor
8. Automation Maturity - Infrastructure as Code at the minimum
9. Communication Strategy (why are we doing this, how does it fit our context, how will we manage risks

With the addition of Git Best Practices maybe? But I am not sure if this fits in the adoption journey document.

1. Don’t commit code as an unrecognized author. Git needs to be a source of truth and allow auditing of all actions.
2. Ensure all secrets are stored encrypted. Use 3rd party secrets Management tools
3. Don't commit huge files or binaries into the repo. Consider using binary repository managers.
4. Use git ignore when appropriate and make it clear.
5. Clean up stale branches and Repos
6. Create a Branch naming convention that's scalable
7. Enable security alerts and cleanup old users
8. Scan for vulnerabilities and secret commits (Example AWS Keys)

[lloydchang]

@mikecali

Can

Standardisation (can’t have 20 x Git sources of truth)

change to:

Standardisation (i.e. Choose a standard from the options in Argo CD and Flux CD to structure your repositories)

for the reason at #19 (comment) ?

Thank you!