Metric Definition and Governance

Why Metrics Conflict

Every organization with more than one analyst eventually encounters the same crisis: two dashboards show different numbers for the same metric, and each analyst insists their calculation is correct. The meeting to resolve the conflict consumes hours, erodes trust in data, and ends with no clear resolution. This is not a technical problem. It is a governance problem.

Metrics conflict for four reasons:

1. Different definitions. "Active user" means different things to different teams. The product team defines it as "logged in within the last 30 days." The marketing team defines it as "performed a key action within the last 7 days." Both are reasonable definitions, but they produce different numbers. Neither team is wrong; they are measuring different things and using the same name.

2. Different calculation logic. Even with the same definition, the SQL can differ. Does "monthly recurring revenue" include trials? Are refunds subtracted? Is the calculation based on the billing date or the recognition date? Each choice produces a different number. Without a documented, canonical calculation, each analyst makes their own choices.

3. Different data sources. One analyst queries the production database. Another queries the data warehouse. The production database has real-time data; the warehouse has data that is 24 hours old. At 9 AM, the two sources show different numbers for "daily active users." Both are correct for their respective snapshots.

4. Different time zones. "Daily active users" for March 15 means different things depending on whether the day boundary is midnight UTC, midnight Pacific, or midnight in the user's local time zone. A global product with users in 40 countries has no single obvious answer, so each analyst picks one.

The cost of conflicting metrics is not just the meeting where you resolve the conflict. It is the cumulative erosion of trust. When stakeholders cannot trust that the number on the dashboard is the number they will be evaluated against, they stop using the data. They revert to gut feel, spreadsheets, and arguments. The analyst's most important contribution to organizational decision-making is not any single analysis. It is a shared, trusted set of metrics that everyone uses.

Writing a Metric Definition

A metric definition is a precise, unambiguous specification that allows any competent analyst to calculate the same number from the same data. A good definition has seven components:

1. Name and unique identifier. Use a consistent naming convention and assign a unique ID. MET-001: Monthly Active Users (MAU). The unique ID prevents confusion when metrics are referenced in code, documentation, and conversations.

2. Plain-language description. What does this metric measure, in terms a stakeholder can understand? "The number of distinct users who logged in at least once during a calendar month." This is the definition you communicate to non-technical audiences.

3. Technical definition. The precise calculation logic. "Count of distinct user_id values from the login_events table where event_timestamp falls within the calendar month in UTC, excluding accounts flagged as test or internal in the users table." This is the definition you give to analysts and engineers.

4. Data source. Which table or system is the canonical source? "login_events table in the production warehouse, updated hourly." If multiple sources exist, designate one as canonical and document why.

5. Grain. One row per what? "One row per user per month." The grain determines how the metric can be aggregated and segmented. A metric defined at the user-month grain can be rolled up to user-year, but not to user-day without re-querying the source.

6. Dimensions. What can this metric be segmented by? "MAU can be segmented by acquisition channel, device type, country, and signup cohort. It cannot be segmented by user name because the grain is user-month, not user-name-month." Documenting dimensions prevents analysts from attempting impossible segmentations.

7. Owner and version. Who is responsible for this definition, and what version is current? "Owner: Jane Doe, Analytics. Version: 3.2, last updated 2026-01-15." The owner is the person stakeholders contact with questions. The version allows you to track changes over time.

Here is a complete example:

MET-0042: Weekly Active Users (WAU)

Description: The number of distinct users who performed at least one
key action during a calendar week (Monday 00:00 UTC to Sunday 23:59 UTC).

Technical definition:
  SELECT COUNT(DISTINCT user_id)
  FROM user_actions
  WHERE action_type IN ('create_project', 'edit_document', 'invite_member')
    AND event_timestamp >= '2026-03-09 00:00:00 UTC'
    AND event_timestamp < '2026-03-16 00:00:00 UTC'
    AND user_id NOT IN (SELECT user_id FROM internal_accounts)

Source: user_actions table in the production warehouse
Grain: One row per user per week
Dimensions: acquisition_channel, device_type, country, signup_cohort
Owner: Analytics Team
Version: 2.1 (2026-02-01)
Changelog:
  2.1 - Added 'invite_member' to key actions
  2.0 - Changed from login-based to action-based definition
  1.0 - Initial definition (login-based)

The changelog is critical. When a stakeholder asks "why did WAU drop last week?" the first check is whether the definition changed. If version 2.1 added a new key action, the metric is not comparable to periods before February 1st.

The Semantic Layer

A semantic layer is a centralized repository of metric definitions that sits between the raw data and the people who consume it. It is the technical enforcement of metric governance. Without a semantic layer, metric definitions live in individual analysts' heads, in outdated Confluence pages, and in the SQL of dashboards that no one remembers building.

The semantic layer has three functions:

1. Single source of truth. Every metric is defined once, in one place. When an analyst builds a dashboard, they reference the semantic layer's definition instead of writing their own SQL. This ensures that two dashboards showing "Monthly Active Users" are actually calculating the same thing.

2. Consistent calculation. The semantic layer stores the calculation logic as code. When the definition changes, you update the semantic layer, and every dashboard that references it automatically uses the new logic. Without this, changing a metric definition requires finding and updating every dashboard, query, and spreadsheet that uses it — a process that is error-prone and often incomplete.

3. Discoverability. A well-designed semantic layer is searchable. An analyst can search for "revenue" and find all revenue-related metrics, their definitions, their owners, and their downstream dependencies. This prevents the creation of duplicate metrics and helps new analysts understand what already exists.

Modern semantic layers are implemented in tools like dbt (data build tool), Looker's LookML, or dedicated metric platforms. The implementation matters less than the discipline. A semantic layer maintained in a shared spreadsheet is better than no semantic layer at all, as long as it is consistently used and updated.

A minimal semantic layer in a spreadsheet has these columns:

The definition_sql column is the most important. It is the executable specification. Any analyst can copy that SQL, modify the date range, and get the correct number. When the definition changes, the SQL is updated, the version is incremented, and the changelog is appended.

The semantic layer also enables impact analysis. Before changing a metric definition, you can query which dashboards, reports, and downstream metrics depend on it. This prevents the common disaster of changing a definition silently and breaking five dashboards that other teams rely on.

Change Management for Metrics

Metric definitions change. Businesses evolve, new products launch, and old metrics become irrelevant. The problem is not change; it is unmanaged change. A governance process for metric changes has five steps:

1. Proposal. Anyone can propose a metric change, but the proposal must include: the current definition, the proposed definition, the reason for the change, the expected impact on historical values, and the stakeholders who will be affected. "We should exclude bot traffic from DAU because it inflates the metric by 3%" is a valid proposal. "DAU is wrong" is not.

2. Impact assessment. Before approving a change, assess the blast radius. Which dashboards use this metric? Which reports? Which downstream metrics are calculated from it? Will historical values be recalculated, or will the change apply only going forward? If historical values are recalculated, how will stakeholders who have already seen the old numbers be notified? The impact assessment prevents the "silent change" disaster where a metric moves and no one knows why.

3. Approval. The metric owner approves or rejects the change. For high-impact metrics (revenue, user growth, retention), approval should include the stakeholders who rely on the metric. The owner's job is to balance the accuracy improvement against the disruption of changing a widely-used metric. Sometimes the right answer is to create a new metric (DAU excluding bots) rather than changing the existing one (DAU), so historical comparisons remain valid.

4. Implementation. The change is implemented in the semantic layer, the definition is updated, the version is incremented, and the changelog is appended. If historical values are recalculated, the recalculation is documented with the date range affected and the before/after values for key periods.

5. Communication. Stakeholders are notified of the change, the reason, the effective date, and the impact on their reports. The notification should include a clear statement of what has not changed, so stakeholders do not assume everything is different. "DAU now excludes bot traffic. Historical values have been recalculated back to January 2025. The definition of 'active user' has not changed."

The changelog is the audit trail that makes this process work. Every change is recorded with: date, author, reason, old definition, new definition, and impact. When a stakeholder asks "why did this metric change?" six months later, the changelog provides the answer.

Governance fails when it is too heavy. If changing a metric definition requires a two-week approval process, analysts will bypass the process and create their own definitions. The process should be proportional: low-impact metrics can be changed with minimal approval; high-impact metrics require stakeholder sign-off. The goal is not to prevent change; it to make change visible and reversible.