Taming the change chaos: Why change tracking matters in software

For developers and operations teams, change can feel like a double-edged sword. It's the engine of progress, but uncontrolled changes can quickly turn into a debugging nightmare. Identifying the root cause of issues after a flurry of updates can be time-consuming and frustrating.

This is where change tracking steps in. It acts as your secret weapon, providing a clear picture of how modifications impact your software systems. By shedding light on "who, what, when, and why" behind changes, you can quickly pinpoint the source of problems and ensure your software remains stable and reliable.

Introducing New Relic Change tracking

New Relic offers a robust change tracking solution that seamlessly integrates with your existing development workflows. Imagine having a detailed logbook that automatically records all your deployments, code updates, and infrastructure changes. This empowers you to:

• Correlate changes to performance impact: See how recent changes affect key metrics like response times, error rates, and user experience. This allows you to identify if a new deployment introduced any regressions.
• Simplify troubleshooting: Quickly pinpoint the exact change that might be causing an issue. No more sifting through endless logs or guessing what might have gone wrong.
• Boost collaboration: Foster clear communication within your team by providing a centralized view of all changes made to your system.

Want to start tracking your changes? Visit the New Relic change tracking product page for detailed information and resources. When you use the change tracking feature, you can see how recent changes affect end users as well as other micro services dependent on the application.

You can also analyze the impact of changes by comparing with previous release versions.

New Relic change tracking empowers you to embrace change with confidence, ensuring a smooth and efficient development process. But how can you take this a step further and proactively reduce the risk of negative impacts from changes?

Deployment strategies for minimized risk

New Relic change tracking integrates seamlessly with popular deployment strategies, allowing you to introduce changes in a controlled manner. Here are some key strategies to consider:

• Canary deployments: Roll out a new version of your application to a small, controlled group of users first. Monitor performance and user feedback with New Relic change tracking to identify and address any issues before rolling out the update to everyone.
• Blue-green deployments: Maintain two identical production environments (blue and green). Deploy the new version to the green environment and then switch traffic over if everything seems stable with New Relic monitoring.
• Feature flag deployments: Gradually enable new features for a percentage of users using feature flags. This allows you to test and monitor the impact of the feature before making it broadly available.

Rolling updates: Update your application in a rolling fashion, swapping out instances with the new version one at a time. New Relic change tracking helps track the progress and ensures a smooth transition.

Advanced tracking for deeper insights

New Relic change tracking goes beyond simply recording deployments. It offers advanced capabilities like capturing environment variables and build versions, providing a richer context for your change history. Let's delve deeper now into how leveraging these capabilities can unlock a new level of understanding:

• Environment variables: Unveiling configuration context
• Build versions: Pinpointing code-level impact

By enriching the entities with release information, you can analyze performance impact of an ongoing deployment and make faster data-driven decisions if a deployment needs to be rolled out or rolled back.

New Relic tags are a great way to enrich your application with metadata information, which not only provides ease of use, but deep insights in making decisions and managing work flows. For a given application, which is integrated with any of the New Relic APM agent, let’s define release-related environment variables.

• environment: Environment where the application is running; for example, production, staging, or testing.
• release_name: User-friendly name given to the release; for example, macOS High Sierra, Big Sur, etc.
• release_version: Version of the application code that is deployed; for example, 1.0.1.
• release_commit: Entire or partial or first n characters of the application commit SHA; for example, abcd123.
• release_type: Type of deployment strategy you follow; for example, blue-green, canary, etc.

You can configure the above tags in the below ways:

1. Environment variables. These are host-level environment variables. This works best when using Kubernetes and Docker for deploying your application.
   • NEW_RELIC_LABELS="environment:prod;release_name:pakodi;release_version:v2;release_commit:2;release_type:CANARY;"
2. Agent config. The APM agent’s local configuration for each instance, like newrelic.yml, newrelic.config, etc.
   • Configuration options are somewhat agent-specific, but generally use the “labels” key. 

Tags applied from APM agent configuration can result in multiple values for a given tag key on a given entity, because each deployed instance of the agent could be configured to send a different value.

That’s all the configuration that’s required. 

Now, let’s build a simple release tracker dashboard to leverage this metadata combined with performance details captured by an APM agent to derive deeper insights. The dashboard contains three tabs:

Compare Versions: In this tab, for the selected environment and application in the dashboard filters, DevOps engineers can compare golden metrics performance between releases. This comparative data empowers DevOps engineers to make data-driven decisions:

• Green light for rollout: If the metrics for the new version in the canary group remain within acceptable thresholds compared to the baseline (older version), it indicates a stable rollout. DevOps engineers can then proceed with confidence to roll out the new version to the broader user base.
• Identify and address issues: If performance metrics for the new version in the canary group show regressions compared to the older version, this signals potential problems. DevOps engineers can quickly investigate these issues within the contained canary group before they impact a wider audience. This allows for faster troubleshooting and adjustments before a broader rollout.

• Compare Transactions: In this tab, for the selected environment and application in the dashboard filters, engineering can compare golden metrics of each transaction and understand if there were any issues with new features or changes made to an existing transaction. This allows for faster troubleshooting. Developers can quickly respond to the issues raised and provide a quick fix to avoid release disruptions and roll out changes with confidence.

• Release Overview: This tab provides an overview of release information across all applications and environments in this account. This tab is especially useful for release managers.

By harnessing the power of the environment, release, and build variables, you can transform change tracking from a simple logbook into a strategic tool for informed decision making.