devops – Agitma

Category Archives: Devops

03 Dec 2019

Terminology matters: Agile/Scrum vs CD/CI vs DevOps

Chances are you were triggered by the enumeration of buzz words in this blog’s title. Without effort you could probably squeeze in a few more, like Lean or another flavor of ‘Continuous x’. A lot of managers and teams struggle explaining the company’s meaning regarding these terms. Striking is the absence of a clear and concise understanding of the vocabulary. DevOps is translated into Business involvement, CD/CI is equated to Scrum and Agile is basically a term covering everything. The point is, you need to make sure everyone around you understands what you mean and how the terms relate to your organization’s context. During sessions, I use the diagram above to explain the different angles that Agile / Scrum – Continuous Delivery – DevOps have. In this blog, I will dive in a little deeper and share my thoughts on the terminology and how I explain the nuances and differences.

Agile/Scrum and Continuous Delivery

Agile/Scrum focuses on the ability of teams and organizations to absorb the array of changes that happen while making software. The setup of the Scrum rituals ensures that transparency is present at all times, for the team itself but also for the stakeholders. When functioning well, the Product Owner helps the stakeholders in guiding the product development in the right direction. Adapting change and changing course now and then is part of the normal process. The main aim of Scrum is to deliver as much value as possible working in a sustainable pace [1]. The framework says nothing about how this should be achieved from a technical perspective. Not what tools to use or what platform is appropriate, nor does it mention test automation, infra structure as code or deployment pipelines.

The power of short cycled feedback (highly valued within Scrum) is strengthened enormously when teams can focus on developing the properties of the systems (functional and non-functional) instead of spending time on manual tasks. Manual tasks can include everything from testing code by hand, setting up new environments or executing tedious deployment scripts following a step by step manual. This is the area where Continuous Delivery steps in. Continuous Delivery empowers teams by giving them the tools and methods to automate their process as much as possible and minimize manual labor. Not only does it save time, it also decreases the error rate by eliminating the human factor [2].

Simply stated: by combining Scrum and Continuous Delivery, teams can create a new feature, run automated tests and deploy the new software to environments where validation by business can be done. All covered by a framework supporting transparency, inspection and empiricism at all times.

Continuous Delivery and DevOps

Continuous Delivery is largely based on the technology that enables teams to automate as much of the development process as possible. Traditionally, a handover moment between development and operations was in place when software was finished and accepted. The deployment to the production environment and the responsibility for the quality of service once running was in hands of the operation teams. In DevOps, teams become responsible for the entire chain all the way to production [3]. Technically, the tools and actual deployment on production are not so different than those used on other stages. Having the explicit mandates and authorizations marks the distinction, these include access to production monitoring and logging information.

Often the biggest chasm to cross in a journey towards DevOps is organizing the rights and responsibilities for newly formed DevOps-teams (see “DevOps understanding the Evolution” to have a brief chronological overview of DevOps). This becomes more evident when discussing quality of service. Who is allowed to see how a system is running, how to deal with production data and what information has been logged? Especially large organizations with a history of separated departments and teams for operations and development are discussion prone. If you find yourself in a mandate-discussion going in circles, what can help is the next simple question: Who would be responsible if we only had one team? Clearly, it is only logical that this one team should be in the lead. Barricades often dissolve after discussing why it should be different just because you have multiple teams.

A well-aged mantra reads as follows: ‘if you want to have a stable environment, don’t change it’. This contrasts with the nowadays popular one: ‘fail fast and fail often’. The technology now present – based on Continuous Delivery – enables teams to develop and deploy new software with no or hardly any downtime. Combined with DevOps, teams can take ownership for the entire life cycle of the product. Responsibility only ends when a system is decommissioned.

DevOps and Agile/Scrum

While DevOps ensures that teams are able to take technical and organizational ownership for the quality of service of the products they develop and maintain, the term says nothing about the team dynamics that should be in place to support this responsibility. The other way around is also true, Agile/Scrum makes no mention of different environments or how to address major production incidents. Of course, it is possible to have for example Scrum without DevOps. However, this means you will have a delay in your feedback loop and miss valuable information about the production environment. More importantly, teams that have production responsibility have more insights in the behavior – intended or not – of their system.

Naturally, DevOps requires some additional agreements in a team’s Scrum process. How is monitoring approached (not technical but process wise) or how much time can be spent on improving ‘the run’. By discussing the time spend on monitoring and analyzing the software in production, a more in-depth analysis can be made based on value. These Lean influences from DevOps provide the structure to improve operational excellence as well. For example: ownership and code-logging hygiene go quickly hand in hand when a team has to fix its own production problems. Decision making and prioritization by Product Owners is more coherent in DevOps. The total costs of running a system become more transparent and are influenced more directly by the Product Owner’s team. See “Product Owners in DevOps” to read more on backlog prioritization in DevOps.

Combining the flow and value stream ideas from DevOps with the sustainable pace and delivery of Done increments from Scrum is powerful. Feedback from clients or the business on the latest software can continuously be weighted up against other aspects of running the product.

Conclusion

While it is theoretically possible for teams to use exactly one of the ‘buzz terms’ without the others, it is not common. What I usually see in organizations is a mix of all three. That makes sense, the terms and related concepts focus on team responsibility and share the desire for short feedback loops, build-in quality and transparency towards stakeholders and teams internally. They re-enforce each other and offer opportunities to improve specific parts of an organization’s way of working. That is one of the reasons why you should be able to describe in a concise way what you mean when discussing improvements to your development process.

Cheers,
– Sjors Meekels

Disclaimer: the description above is not complete and it surely is not one hundred percent perfect. It is not meant to be. It is a personal simplification to address the consistency of a set of popular tech-buzz words.

References & recommended reading:

[1] SCRUM – a Smart Travel Companion, Gunther Verheyen
[2] Continuous Delivery, Jez Humble & David Farley
[3] The DevOps Handbook, Gene Kim, Jez Humble i.a.
[4] DevOps: understanding the Evolution: https://www.agitma.nl/devops-understanding-the-evolution
[5] Product Owners in DevOps: https://www.agitma.nl/product-owners-in-devops-what-dominates-your-backlog-urgent-or-important-matters
[6] https://www.linkedin.com/pulse/differences-between-continuous-integration-delivery-versteijnen Pieter Versteijnen (last checked 19-11-2019)

Download blog as PDF

This blog is available here:

Filled Under: agile,continuous delivery,devops,kanban,lean,scrum,software development Posted on: 3 December 2019

19 Nov 2019

DevOps in beweging – Heunks

DevOps in beweging van Jan Heunks is een Nederlandstalig verzamelwerk dat DevOps uit de doeken doet. Raakvlakken met onder meer Agile, Lean en Continuous Delivery komen uitgebreid aan de orde. Voor een allround beschouwing van DevOps, inclusief plaatsing in de huidige organisatiestructuren, is dit boek een uitstekend startpunt.

Jan Heunks laat zijn ervaring met Lean en Lean IT terugkomen in zijn boek ‘DevOps in beweging’. Alleen Lean noemen naast natuurlijk DevOps doet het boek te kort. Het is een overzichtswerk waarbij een veelheid aan zowel technische- als managementontwikkelingen beschouwd door een DevOps-bril.

‘DevOps in beweging’ is opgebouwd uit drie delen, met per deel twee hoofdstukken. Ieder hoofdstuk wordt afgesloten met een korte conclusie en de delen als geheel met een aparte samenvatting. Het boek heeft een soft cover en pagina grootte A4, dit maakt het wellicht wat minder geschikt om zo mee te nemen maar wel prettig leesbaar. Het is jammer dat niet alle figuren even scherp in het boek zijn opgenomen, storend is dit echter niet.

In Deel I legt de auteur uit wat de toegevoegde waarde is van DevOps. Onderdeel hiervan is een korte geschiedenis en de belangrijkste invloeden die geleid hebben tot de huidige denkbeelden omtrent DevOps. Natuurlijk komen de invloeden van Agile software development en Continuous Delivery aan bod. Bij het uitdiepen van de waarde voor de keten en organisatie legt hij de verbanden met Lean (IT & Startup) en beheerprocessen zoals ITIL.

Deel II betreft het theoretische kader van DevOps. De auteur borduurt net als in Deel I voort op Lean & Agile en op bekende werken als The DevOps handbook, The Phoenix Project en Continuous Delivery. De hoofdstukken 3 en 4 vormen zo een stevig gecombineerd kader van relevante management aspecten (o.a. Theory of Constraints, IT-servicemanagement) en technologische principes (o.a. Infra structure as code, Simian Army). Sprekend voor de huidige uitdagingen bij het uitleggen en bediscussiëren van DevOps is het ontbreken van één heldere & geaccepteerde definitie. Paragraaf 4.2 illustreert dit door de gegeven voorbeelden en vergelijking.

Het laatste deel richt zich op de integratie van DevOps binnen organisaties. Helder is dat investeringen gedaan in bestaande processen en structuren vooral niet direct overboord gegooid moeten worden. Nee, DevOps kan prima een logisch opvolgende evolutie zijn van de huidige organisatiestructuur. Wat niet ontbreekt bij het beschrijven van de aanpak binnen organisaties is een DevOps niveau-indeling voor teams (hfd 6). Dergelijke formats, vaak maturity models genoemd, zijn al gemeengoed bij veel organisaties die certificering hebben of gestructureerd verbeteringen willen doorvoeren.

Het is indrukwekkend om te zien hoeveel bestaande vakliteratuur, modellen en theorieën zijn meegenomen in de beschrijving van DevOps. Het geeft hiermee lezers aanknopingspunten vanuit bekende referentiekaders en achtergronden. Dit helpt bij het vaststellen wat DevOps toevoegt aan de eigen organisatie en waar huidige knelpunten kunnen zitten. De veelheid kan echter ook overweldigend zijn, zeker wanneer je alle figuren en tabellen wilt begrijpen of onderling verbinden. Met name de verschillen tussen de theorieën en figuren kunnen het lastig maken, termen worden anders geïnterpreteerd of uitgelegd en het is onmogelijk om alles in één overkoepelend raamwerk te plaatsen. Dat laatste is echter ook geen doel van de schrijver.

Conclusie

Al met al, is het een stevig boekwerk wat een goed beeld neerzet van DevOps. Het is volledig Nederlands, op het Engelse vakjargon na natuurlijk, wat voor sommige lezers de drempel net zal verlagen. Een goede combinatie van management aspecten en technologische invloeden maken dit boek tot een prima startpunt voor het uitdiepen van DevOps voor de eigen organisatie.

Over deze recensie
Deze boekrecensie is tevens verschenen op www.managementboek.nl.

Cheers,
– Sjors Meekels

Download blog as PDF

This blog is available here:

Filled Under: agile,book review,devops,lean,software development Posted on: 19 November 2019

07 Jun 2019

DevOps: understanding the evolution

Many organizations find themselves somehow confronted with the need to change the way their teams are organized and operating. Echoing in the hallways is this idea of DevOps. It sounds promising and has a lot of common ground with Scrum, which organizations are already familiar with. Moreover, it is drenched in Agile thinking. This blog is the first of a few concerning DevOps. In this blog I’ll describe an organizational evolution which frequently ‘’happens” to IT organizations in their journey towards DevOps. Which phase is your organization in and what is there to gain and overcome?

Definitions

Before jumping into the evolutionary path, let’s spend a minute chewing on three DevOps definitions.

DASA: “DevOps is a cultural and operational model that fosters collaboration to enable high performance IT to achieve business goals.” [1]

Humble: “a cross-disciplinary community of practice dedicated to the study of building, evolving and operating rapidly-changing resilient systems at scale.” [2]

Len Bass et al. “a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production, while ensuring high quality” [3]

As you can see, the definitions have some overlap but aim their focus differently. The term DevOps has not one clear and uniform definition that is shared in the community. Therefore, it is a good practice to find some common ground before starting any discussion on DevOps in your own organization. For now, it is good enough to have a feeling of what a definition could look like.

1. Pre-DevOps: Scrum teams

Scrum is still the most widely applied framework which is used to organize software development teams. There is no need to differentiate in level of maturity or quality by teams using this framework. In this phase, teams just focus on getting their next Increment of Done software ready and possible even release it. In the team all capabilities to get the software Done should be present, for example designing, programming, testing & even appropriate documenting. However, these are all capabilities and not roles and they are preferably not tight to only one person. All members are simply called developers and there is a chasm between teams and production environment.

COMMON CHALLENGES: managing external dependencies, including limiting waiting times and hand-overs to other departments for deployments.

2. Pre-DevOps: CD/CI

Since the Continuous Delivery (& Integration) revolution was ignited by Jez and Farley in 2010, a lot more attention is paid to automate and integrate, test execution, deployments on pipelines and feedback loops. The availability of tools that could execute and monitor these activities via – more or less – user friendly interfaces helped getting momentum. In this phase teams are able to deploy much faster and more reliable to different stages. The long-winded deployment manuals become something of a quickly forgotten past, instead the teams are empowered with new tools.

COMMON CHALLENGES: setting up reliable pipelines, changing existing software with CD in mind, depending on the team that supports the pipeline tooling and operator who execute the deployments to production.

3. DevOps round I: Functional Operations

Coming from the previous phases, teams have a rhythm of delivering working software. They should start to live by the mantra “You build it, you run it”. Teams become more interested in the health and status of the production environment. That is an excellent reason for reaching out to Functional Operations to obtain this – more business – side of the software. New questions arise. How is the software used? Which features are the most valuable ones? What is the logging or monitoring of the production environment telling us? The usual approach to enable this organizational need is by extending the existing delivery teams with people who have this Operational role. Often these people come from a more centrally organized department. In doing so, the teams receive more responsibility in the shape of team members who are authorized to see certain data on production and who have in-depth knowledge of how the software is being used. This Functional Ops-role is embodied by an Ops-Engineer who joins the teams.

COMMON CHALLENGES: embedding the Functional Operators in the team (and not creating a split team), getting the required rights available for more team members, getting all the information from production to the team e.g. logging and monitoring information.

4. DevOps round II: Technical Operations

Once organizations have more than a few development teams, there are usually also some supporting teams around. The latter teams are usually specialized in the infrastructural layers of the software solutions: development machines, testing environments, disk management and operating systems are all good candidates. If such a central team or department is in place, this team is often responsible for the deployments of new software into production. But, the newly empowered teams in phase 2 or 3 have no such need anymore. More importantly, the teams themselves have the technological means to get their new Increment deployed on any stage or server that is required. The next question usually follows very soon: “Who is now allowed to perform the deployment to the production environment?”. This task had been one of the most eminent examples of the segregation between Dev & Ops, but is in this phase integrated into one.

COMMON CHALLENGES: again, embedding new people in the team, especially when mindset between Dev & Ops are colliding.

5. Post-DevOps: BizPerfSecDevOps?

In order to make teams more and more self-supporting, they need other capabilities, either technical or more business oriented. Depending on the nature of the business and – again – the size of the organization, any of the following experts’ roles could be eligible for the team: DB-admin, security expert, platform engineer, performance engineer, UI/Interaction designer, SEO specialist, business expert.

COMMON CHALLENGES: which extra responsibilities are helping teams to create more value versus which capabilities should remain separate?

Conclusion: How to evolve?

From an organization view point any of the phases described above is just one of the almost infinite amounts of possibilities to get organized. Take a step back and see how your organization is just a collection of people, with skills, tools, tasks, responsibilities and mandates organized in a particular way. We aim to come up with the exact organization that is able to create the most value. In the evolution above there is an ongoing shift in roles and responsibilities to and from teams. The context of the team is very much leading in getting the optimum setup. Longstanding organizational setups and existing boundaries are often hard to get by.

One way of thinking more out of the box is by raising the question: “What if we only had two teams?” or even better “What if we only had one?” This forces you to elaborate on who is allowed to do what and why. This question is applicable in all phases, yet becomes paramount in the last one. Last but not least: don’t get stonewalled by the existing organizational structures and boundaries.

In the next blog – “Terminology matters: Agile/Scrum vs CD/CI vs DevOps” – I will focus on the above used terminology in depth.

Cheers,
– Sjors Meekels

Notes:
1. The aforementioned common challenges are by no means complete.
2. Furthermore, being in a next phase does not exempt you from keeping attention to practices of previous phases (e.g. Scrum).

References & recommended reading:

[1] DASA: DevOps Fundamentals_Glossary_English
[2] https://theagileadmin.com/what-is-devops/
[3] Bass, Len; Weber, Ingo; Zhu, Liming (2015). DevOps: A Software Architect’s Perspective.
[4] Scrum a Pocked Guide, Gunther Verheyen
[5] Continuous Delivery, Jez Humble, David Farley
[6] The DevOps Handbook, Jez Humble, Patrick Debois a.o
[7] DevOps for Developers, Michael Hüttermann
[8] The Phoenix project, Gene Kim, Kevin Behr a.o.

Download blog as PDF

This blog is available here:

Filled Under: agile,devops,scrum,software development Posted on: 7 June 2019

14 Apr 2017

Product Owners in DevOps: What dominates your Backlog, Urgent or Important matters?

Product Owners have a great responsibility in prioritizing the work on the Backlogs. Their role can hardly be underestimated. How about some support from the past? Decades ago Eisenhower made a quote on important and urgent matters. His idea has transferred into a matrix used to prioritize personal actions. Can Eisenhower’s priority matrix help Product Owners in prioritizing backlog items? I think so. In this post I will combine Eisenhower’s ideas with those of Agile product backlog management, and a little wild life.

Eisenhower’s matrix
In short, Eisenhower’s matrix focuses on four distinctive quadrants created by the two viewpoints – urgency and importance. The goal is to determine someone’s priorities [1][2].

Urgent matters: require immediate attention. If possible you should act on these matters now. Important matters: these relate to long term objectives and need constant focus. When combined this leads to the following overview and actions:

Now let’s transpose these quadrants to backlog management in an Agile environment. All work on the Backlog can be qualified according to Eisenhower’s quadrants. But what is the optimum setup for your Sprint Backlog? Is there any? Based on the model, I will discuss a set of situations which are all taken from projects and teams I have worked with over the years. In these situations, teams are working on Product Backlog Items that primarily originate from one of the quadrants. To make teams easily recognizable, they all have a different nickname which sticks out.

1. Orange Firefighters, brave and busy

If your teams are busy working on the urgent important issues or new work of this category is entered each running sprint, it seems like your devops-teams are firefighting. This is only a good thing when a) there is an actual fire and b) – more importantly – they can extinguish the fire. If this pattern is more the rule than the exception, changes are something is not OK. This could relate to your production environment, either the applications, the infra, people handling it, or all. Try to look for the root causes and fix those, via the regular process of course. Instead of focusing on optimizing the fire-hoses and firemen. Although your teams are working on the top priority work items, there are risks and down sights. For example, every Product Backlog item which is refined or taken in the sprint and pushed out by the orange firefighters, has been a waste of time. Moreover, the company’s short term future may get hurt if this situation occurs for too long a time.

PROS: working on top priority items, CONS: waste in preparation time, pressure within teams, ATTENTION: should not persist for long, focus on fixing root causes!

2. Blue Coyotes, opportunistic

Sometimes teams are working on urgent but not important issues sprint after sprint. These teams are the so-called coyotes. This scenario also raises an interesting question. Are those issues really urgent? If you and your team tend to say no, there should be a dialogue. Re-discuss these issues with your stakeholders and try to balance them out with your company’s important stories. In practice it is more convenient to work on topics that are urgent to our stakeholders now. However, don’t go for convenient, but invest in discussions and long-term goals.

Okay, so are you done when the answer is yes? Well, sorry, no. If this situation is continues for a long period of time, it could be a sign that you are understaffed. Not working on the important issues, could be a serious threat to business continuity on the long term. Perhaps you can consider to appoint a temporary team that can eliminate the ‘pile’ of urgent stuff, which is in the way of working on the important stories. However, one thing should be in place before you scale up: a clear vision on what is actually important. This may sound like a no-brainer, but more than a few Product Owners are struggling in identifying what is really contributing to the company’s future.

PROS: satisfying to stakeholders, CONS: not working on the long run, ATTENTION: are backlog items really that urgent or is there too much stuff in the way?

3. Green Tortoises, steady but a bit boring

The third scenario reflects the mindset of the tortoise. Namely, where the majority of the work directly relates to important backlog items. Good for you, you keep the company or department in business. However, you might be missing out on immediate revenues from low hanging fruit. Why not optimize your SEO rank now and then, or eliminate that bug 80% of your users are complaining about? These small improvements won’t harm the long-term objective and they do give you a change to help users, increase the short-term cash-flow and satisfy a few stakeholders along the way. Like their nickname, these projects or teams can be a bit boring and move slowly towards a certain ‘old age’ or goal. In their journey, they could lose the connection with the rest of the organization when results are not made visible in the meantime. Besides that, if these projects entail migration without new functionalities, also the teams can get bored. They may benefit from the diversion of small improvement stories.

PROS: working on the mission of the company, CONS: in long running projects stakeholders can get detached, ATTENTION: room for quick wins along the way?

4. Well… Black Dodo’s

If you have teams working on the black stuff, something is off. Perhaps you are way overstaffed, or backlog management is in utterly disorder. Avoid this situation at all costs. When nothing changes soon these teams will become extinct…

PROS: none, CONS: waste of time and energy for all, ATTENTION: immediate action required!

Balancing act, rhythm and focus

Do you feel like you are getting mixed signals now? You are right. This is the arena where great Product Owners can stand out of the mediocre crowd. If they are able to balance the Product Backlog with the stakeholders and find a rhythm together with their team(s) to ensure enough focus, you have a winning combination. Relevant questions to be answered are: Who is your most important stakeholder? And how are competing stakeholders managed? What is most important for the customer and the future of the company?

A predicable heartbeat helps the team and PO in delivering on regular intervals. Yet, it also promotes alignment in the rest of the organization. The characteristics play an profound role in the optimum heartbeat. For example, if you are part of a true DevOps team, you know ad-hoc work (important and urgent issues) will emerge and that is part of your sprint routine. In another case you might have a more component oriented team, building stuff for other teams.

If you only have one development team available, the balancing act of shifting priorities becomes more precarious. The team can feel comfortable working on multiple topics in one sprint and it is not forbidden to do so, as long as there is enough clarity and stories are really Done at sprint’s end. However, most teams benefit from focus during the sprint, so when possible, craft your Sprint goal on one or two categories only. You could for example spend one sprint on urgent matters and the next two or three on the important trajectory. In general, true orange stories are always executed asap. When multiple teams are involved you can have more flexibility and for example rotate duties between teams every few sprints.

Last thoughts

So, Eisenhower can help in identifying certain wanted or unwanted patterns in your current Product Backlog. What it cannot do is provide the ‘correct’ distribution. That task is still meant for the Product Owners and their teams. I am curious, what kind of wildlife represent your teams currently and how is your Backlog setup in terms of Eisenhower’s quadrants?

Cheers,
– Sjors Meekels