Diagnose a KPI Drop with a Data Contract Check

Who This Helps

Hey there, Junior Analyst. You just saw a key metric drop 15% overnight. Your stakeholder is asking why, and you need a clear answer fast. This is for anyone who wants to move from panic to pinpoint accuracy. It's a core skill from the Data Reliability Leadership course.

Mini Case

Sam's weekly active user report showed a sudden 12% dip. Instead of diving into every query, Sam first checked the 'userlogin' data contract. Turns out, a backend service change had stopped sending 'sessionend' events for 48 hours. The contract flagged the missing data pattern. Diagnosis time: 25 minutes. Panic avoided.

Do This Now (5 Steps)

1. Pause the panic. Take three deep breaths. Your job isn't to fix everything instantly; it's to find the right starting point.
2. Grab your metric's data contract. If you don't have one written, jot down the three core things: the source table, the freshness rule (e.g., updates daily by 9 AM), and the key field used for calculation.
3. Check the source. Go directly to the raw data source named in your contract. Look at the last 3 days. Is the data there? Is it fresh? Do row counts look normal?
4. Check the calculation. Open your analysis code or tool. Compare yesterday's logic with last week's. Was a filter added? Was a column renamed? Even tiny changes can have big effects.
5. Isolate one variable. Your mission is to answer: Is this a business change or a data break? Your contract helps you rule out the data break first. Think of it like checking if the oven is on before blaming the recipe.

Avoid These Traps

• Don't start by building five new charts. You'll waste time visualizing a problem you don't understand yet.
• Don't assume it's 'just noise.' A 10% shift needs a reason, even if the reason is a data issue.
• Don't skip talking to the data engineer or source system owner. A quick 2-minute message like 'Hey, seeing a gap in table X after 4 PM yesterday, any known deploys?' can save you 2 hours.
• Don't forget to document your steps as you go. What did you check? What did you rule out? This turns your solo diagnosis into a team asset for next time.

Your Win by Friday

By using this contract-first approach, you'll ship a diagnosis that says: 'The KPI drop is due to a 48-hour gap in source data from System Y, not a change in user behavior. The data team is restoring it. Recommend holding any strategy changes until Friday's data is in.' That's clean, clear, and builds massive trust. You go from being the person with the confusing charts to the person with the answer. Now that's a good Friday feeling.