The book Accelerate is the single book I’d recommend to anyone who wants to understand some tangible and satisfying truths about succeeding in this space. Producers especially pay attention.

Thank you for the clip! The entire episode is so densely packed with info that this is a fresh, welcome reminder.

Nicole and her team at DORA did great work validating effective software practices. The challenge I encounter is that 90% of senior software leaders have probably never heard of her or DORA. They are far too engrained in traditional practices cultivated over decades to consider any changes.

Metrics are inherently doomed to fail as we in the end work with people, and people are *biased*, *lazy*, *jealous*, and will irremediably work towards improve the metrics for their own sake, instead of the team's or the manager's.

Randy uses the words "cause" a number of times. However from what I remember of the Accelerate book, they were very clear to _not_ claim causation. They did use the term "predict", ie "DF predicts high productivity", but that isn't quite the same thing. Did I miss something, or is there some other research that really proves causation? One other thing I noticed about the research presented in the Accelerate book is that almost everything was based on _self reported_ data. DF and LTC were self-reported - and so was the "productivity". The research therefore demonstrated a clear correlation between those *claiming* high DF, low DTC, etc. and those *claiming* high productivity. However was there ever any independent verification of those productivity claims?

Hi Dave! Quick question, when doing continuous delivery, when you're releasing changes every hour, should you still check your changes on production? Or is checking locally before deploy enough?

@ContinuousDelivery

3 ай бұрын

CD doesn't demand that you release your changes every hour, but rather that your changes are deemed "releasable" at least once per day, but ideally ever hour. However, having said that if you are releasing that frequently, then it makes most sense to automate any verification, including checking that things are working in production. Companies like Amazon and Netflix go as far as defining what "success" means for each feature, automating how they will check for "success" and if the change is not deemed to be successful in prod, will then automatically roll back the change. For the systems that I worked on, we didn't go that far, but we did include general "health-checks" in the system that confirmed in production, post deployment that everything was working as expected. Basically these were sort of smoke-tests to check that all the pieces were correctly in place and that they could talk to each other. All automated, as I said.

@danielwilkowski5899

3 ай бұрын

@@ContinuousDelivery does that mean that I should run my tests after deployment? I created my tests using tdd, I have about 2000 of them, and they take about 20 seconds to run. What to do when they fail, should I automatically rollback the deploy?

@ToMeq87

3 ай бұрын

@@danielwilkowski5899 Ideally you should hook up your tests to the build, so that a successful build means the tests completed successfully too. In such setup you only consider deploying your stuff in QA/production that passes your test suite. Therefore, there’s no need to roll anything back, you rather avoid putting out crappy versions. So, you should run your tests before deployment, post the build phase, that’s for the unit/integration tests. Post deployment you can also run a bunch of more elaborate tests, like functional or end-to-end.

@danielwilkowski5899

3 ай бұрын

@@ToMeq87 Obviously I'm doing that - i run tests locally, and then during the deploy in the pipeline. My question is different - should you run some kind of additional checks/tests on the production machine after the deploy. Not instead of - additionally. And if yes, what happens when those fail?

@ToMeq87

3 ай бұрын

In the projects I’m in we do indeed execute additional checks. That’s because the data we work with only exists in production, making QA testing nearly impossible/irrelevant - we do it, but don’t trust the results all that much. We also can’t just copy the data to QA, because of it being pulled through all kinds of integrations. Having the above in mind, if those additional post deployment checks fail, we roll back to the previous working version to avoid a surge in support case count. What are your thoughts on doing automated post deployment verification?

What do you think about DORA in the lights of Goodhart‘s law?

@johnb883

3 ай бұрын

The difference between Lean flow metrics and, say, counting lines of code or story points, is that you absolutely do want teams to change their behaviour in response to these metrics! Optimising flow metrics is typically desirable because there are cumulative benefits. Controlling work in progress counts reduces context switching and cycle times, enabling faster iterations and thus faster learning, as well as reducing burnout. Teams that iterate faster get more practiced at releasing to production and responding to feedback from production, helping them to better understand and to improve their product. It's an ongoing virtuous cycle. Of course, if you have tyrannical/incompetent management then you may not see benefits and the system might be gamed. But that's true of any system under poor management! However, if management can cultivate sufficient psychological safety then teams 100% should change their behaviour in response to Lean metrics. (That's why one of the DORA capabilities is 'Visual Management': making these metrics visible to everyone will help keep them front of mind, so that team members can take actions in response to relevant leading metrics: dora.dev/devops-capabilities/process/visual-management/)

@RFalhar

3 ай бұрын

DOTA is science. Goodhart's law is one guy's assertion that people like. Unless you have scientificly-backed reason why we should bother with it, it should be safe to ingore it.

@KiddyCut

3 ай бұрын

The idea of a good metric is that if you try to game it, you actually improve. Try to improve your deployment frequency. How to game it? Deploy more often. Now you actually deploy more often. This was the goal. The only way I see this gameable is to deploy the same state of the software again and again. Easy fix. Forbid to deploy the same version. The same is true for the other parts of the metrics. I think it is still important to have this as a metric for the team, so they can measure their own maturity process. If upper management dictates it is still better to measure DORA instead of lines of code or story points outcome, but already the Change Failure Rate might be a bit easier to game. "Was this really a failure that needed remidition?"

@stevecarter8810

3 ай бұрын

​@@KiddyCutyes that's what I like about the Dora metrics: if you bring them together then gaming then makes you act in good ways. If you game for deployment frequency or lead time, you risk breaking change fail percent or mttr

@simonk1844

3 ай бұрын

I think there are several ways in which incentives for improving metrics can backfire. It seems to me that Goodhart's law does apply, and that the DORA metrics are useful indicators, but should not be goals. Just skipping documentation will increase deployment frequency - and indeed, delivery of useful features. Less work means faster work initially. A lack of documentation will however *decrease* DF eventually, and probably hurt CFR as well - but when will that kick in? Quite likely, in about 2 years after the devs stop writing internal documentation - or when several team members leave, whichever comes first. Not thinking too deeply about security also results in code and features deployed faster. The results of that won't be apparent until some undetermined time in the future - ie is hard to capture in a metric, and certainly won't show up in CFR early enough to be a useful counter-balance. The definition for LTC has never been clear/consistent to me. One interpretation is the interval from merge into trunk (integration completion) to receiving real user requests (production) - ie LTC includes only: build-pipeline execution, some subset of automated tests, deployment to a production environment, and the start of rerouting of user requests. This metric is almost pure "ops", and software development practices are hardly involved at all. That doesn't make it useless for an organisation, but does make it pretty much useless as an indicator of development productivity. An alternative is to include various parts of the development phase: code review for a single patch? Maybe measure LTC over a _user story_ rather than a single patch? Maybe include requirements gathering and user testing? If LTC is an incentivised "goal", and the "pure ops" interpretation is taken, then there is potential incentive for devs to reduce the amount of unit testing. If LTC is being defined in other ways, then I'm concerned that there are a whole lot of other useful tasks that devs will be pushed to omit in order to optimise their LTC.

The problem is not down to folk that are new to software delivery, it is the folk that have been in it for some time in many cases.

@stevecarter8810

3 ай бұрын

Both have their own challenges. The old guard are skeptical and hard to get to change their practices; the young are naive and neglect diligence in the name of going fast.

@seamusconnolly9710

3 ай бұрын

@@stevecarter8810 Just an observation

@stevecarter8810

3 ай бұрын

@@acmhfmggru my challenge is to balance value delivery and sustainability. We have clusters of people who love an emergency and some heroics. They dismiss all metrics including teams telling them they are full and won't be able to integrate the changes by the due date (this is desktop software). This is why I'm on the metrics side, because I want managers to understand that their serial exceptionalism (jumping from emergency to emergency, burning engineering diligence in the name of the current emergency) sacrifices our actual system health. To do that you have to make the system visible to them.

Craftspersonship?

SRE = Site Reliability Engineering

“Speak truth to power” - Tell that to Elon Musk who fired engineers for telling the truth, but I guess we should consider Elon to be new to the industry.

_Continuous delivery_ of oxygen for 8 days - speak truth to power - they marched people to doors coughing with pneumonia - they paid bosses £100K furlough

Thanks for the video. If KZread lets you edit video titles… there's a missing 'r' en.wikipedia.org/wiki/Nicole_Forsgren