As software engineering teams rush to integrate Large Language Models and complex data pipelines, it is tempting to believe the “old” rules no longer apply. However, the complexity of AI-driven systems makes foundational DevOps practices more critical than ever. Whether you are shipping distributed systems, a microservice or machine learning models, these 20 principles remain the bedrock of high-velocity, high-quality delivery.

1. “Everything breaks all the time.”

  • The Reality: The naive assumption of perfect system reliability leads to brittle architectures that collapse under pressure.
  • The Fix: Design for failure. Assume hardware, software, and networks will fail and build resiliency mechanisms (like circuit breakers) to handle these inevitable outages gracefully.
  • Reference: Werner Vogels, “A Conversation with Werner Vogels”

2. “If it hurts, do it more often.”

  • The Reality: Teams often delay painful tasks like integration, testing, and deployment until the end of a project, creating massive, high-risk batches of work.
  • The Fix: Bring the pain forward. By performing these tasks continuously, you reduce the batch size, making complex merges and deployments manageable and routine.
  • Reference: Martin Fowler, “Frequency Reduces Difficulty”

3. “You build it, you run it.”

  • The Reality: Traditional silos between development and operations create “walls” where developers throw code over to ops, lacking visibility into how it performs in the real world.
  • The Fix: Shift to cross-functional teams where developers take accountability for the deployment and ongoing operation of their software, fostering a culture of shared responsibility.
  • Reference: Werner Vogels, “A Conversation with Werner Vogels”

4. “Done means released.”

  • The Reality: Features are often marked “done” when coding is finished, but undeployed code is essentially inventory waste that delivers zero business value.
  • The Fix: Redefine “done” to mean functionality that has been showcased or released in a production-like environment. Until users can use it, it isn’t done.
  • Reference: Jez Humble and David Farley, Continuous Delivery

5. “Production is the only environment where code can repay its debt.”

  • The Reality: Engineering effort is an investment that remains in the red (debt) as long as the code sits in staging or development.
  • The Fix: Focus on shrinking the path to production. Prioritize continuous deployment to start generating value and feedback immediately.
  • Reference: This is a widely circulated community principle with no single canonical source. Its ideas are well-represented in The DevOps Handbook.

6. “There is no place like production.”

  • The Reality: Lab and staging environments are essentially “works of art” that can never fully replicate the diversity, data volume, and chaos of real-world user traffic.
  • The Fix: Rely on safe testing in production (e.g., canary releases, feature flags) to gather noisy, real-world data that lab environments simply cannot generate.
  • Reference: This is a piece of folk wisdom from the software development and operations community with no single canonical source.

7. “100% is the wrong reliability target for basically everything.”

  • The Reality: Striving for 100% uptime is prohibitively expensive and stifles innovation because every change is viewed as a threat to stability.
  • The Fix: Set a realistic “error budget.” Accept that failure is normal and use the remaining budget to push features faster.
  • Reference: Site Reliability Engineering (Google)

8. “Data is a silent killer.”

  • The Reality: Unlike code crashes which are loud and immediate, bad data can silently corrupt reports and ML models for months before being detected.
  • The Fix: Implement “Observability Driven Development” (DODD) to monitor data quality and lineage as aggressively as you monitor infrastructure uptime.
  • Reference: Matthias J. Sax, “Data is a Silent Killer”

9. “‘The network is reliable’ is a fallacy.”

  • The Reality: Developers new to distributed systems (and distributed ML training) often assume network packets will always arrive, leading to systems that hang indefinitely during outages.
  • The Fix: Architect systems to expect latency, timeouts, and dropped packets. Use asynchronous communication and queues to decouple components.
  • Reference: The Fallacies of Distributed Computing

10. “All non-trivial abstractions, to some degree, are leaky.”

  • The Reality: We rely on abstractions (like Cloud APIs or ML frameworks) to simplify complexity, but when they break, the underlying complexity leaks through, often baffling those who don’t understand the layer below.
  • The Fix: Don’t just learn the tool; understand what it abstracts. Competency requires knowing how to fix the system when the “magic” stops working.
  • Reference: Joel Spolsky, “The Law of Leaky Abstractions”

11. “Architecture is leadership.”

  • The Reality: Architecture is often treated as a static document or a command-and-control function, which becomes a bottleneck.
  • The Fix: Architects should act as mentors, providing common building blocks and a technical vision that empowers teams to make local decisions aligned with global goals.
  • Reference: Gregor Hohpe, “The Architect Elevator”

12. “Architecture is the what, why, and when; tools are the how.”

  • The Reality: Teams often suffer from “shiny object syndrome,” selecting popular tools (like the latest vector database) before understanding the architectural need.
  • The Fix: Define the architecture first. Choose tools only after you understand the problem you are solving, not the other way around.
  • Reference: This is a widely circulated community principle with no single canonical source.

13. “Cease dependence on mass inspection to achieve quality.”

  • The Reality: Relying on a massive manual testing phase at the end of development is too late to fix quality issues effectively.
  • The Fix: Build quality in from the start. Integrate automated testing (unit, component, acceptance) into the daily workflow so defects are caught immediately.
  • Reference: W. Edwards Deming, “14 Points for Management”

14. “Testing is not a phase.”

  • The Reality: Treating testing as a separate phase after coding creates feedback loops that are too slow, leading to “integration hell”.
  • The Fix: Make testing a continuous, cross-functional activity that happens alongside development, not after it.
  • Reference: Jez Humble and David Farley, Continuous Delivery Principles

15. “If any part of the pipeline fails, stop the line.”

  • The Reality: Ignoring broken builds to “keep moving” results in a pile-up of defects that makes the software undeployable.
  • The Fix: Treat a broken deployment pipeline as a high-priority emergency. Stop all new work until the pipeline is green again to ensure the software is always in a releasable state.
  • Reference: The Andon Cord (Toyota Production System)

16. “A postmortem without subsequent action is indistinguishable from no postmortem.”

  • The Reality: Companies often write incident reports that simply gather dust, ensuring the same outages happen again.
  • The Fix: Ensure every postmortem results in prioritized, tracked action items. Focus on fixing the process, not blaming the people.
  • Reference: Google SRE Book, “Postmortem Culture”

17. “Keep absolutely everything in version control.”

  • The Reality: When configuration, scripts, or documentation are stored outside version control, environments become “works of art” that are impossible to reproduce.
  • The Fix: Store every artifact—code, test data, database scripts, and infrastructure configuration—in a single repository to ensure traceability and reproducibility.
  • Reference: Jez Humble and David Farley, Continuous Delivery

18. “Infrastructure should be autonomic.”

  • The Reality: Manually maintaining servers leads to “configuration drift,” where no two environments are alike and deployments become unpredictable.
  • The Fix: Use automated configuration management to ensure infrastructure corrects itself to the desired state. It should be cheaper to replace a server than to repair it.
  • Reference: Chad Fowler, “Immutable Infrastructure”

19. “Automate almost everything.”

  • The Reality: Manual processes are boring, repetitive, and error-prone. If a machine can do it, a human shouldn’t.
  • The Fix: Automate your build, deploy, test, and release processes. If you have to do a task more than twice, write a script for it.
  • Reference: Jez Humble and David Farley, Continuous Delivery

20. “Continuous deployment reduces the risk of any particular release.”

  • The Reality: Releasing infrequently builds up a massive “change delta,” increasing the probability that something will go wrong.
  • The Fix: Release every single good change to production. Smaller releases mean smaller risks and instant feedback.
  • Reference: Timothy Fitz, “Continuous Deployment at IMVU”

Conclusion

While the tools and technologies of software delivery continue to evolve, these immutable truths of DevOps remain constant. By embracing these principles, engineering teams can navigate the complexities of modern software development with confidence, delivering value faster while maintaining reliability and quality in an increasingly AI-driven world.

<
Previous Post
Feature Complete: The Art of System Synthesis
>
Blog Archive
Archive of all previous blog posts