Dashboard QA and Review

Data Accuracy Validation

Data accuracy validation is the process of verifying that the numbers displayed in a dashboard match the numbers in the source systems. This is the foundation of trust. A dashboard with inaccurate data is worse than no dashboard at all — it leads to wrong decisions and erodes confidence in data-driven processes.

Why Data Accuracy Matters

Stakeholders make decisions based on dashboard data. If the sales total on the dashboard is $10M but the actual sales total is $9.5M, the stakeholder may approve a budget based on incorrect information. The cost of inaccurate data is not just the wrong decision — it is the loss of trust in the dashboard and in the data team.

Data accuracy validation is not a one-time activity. It must be performed every time the dashboard is updated, the data source changes, or the calculations are modified. A robust QA process catches errors before they reach stakeholders.

Reconciliation Techniques

Reconciliation is the process of comparing dashboard metrics against an independent source. The independent source can be a SQL query, a report from the source system, or a manual calculation.

1. Total reconciliation. Compare the grand total from the dashboard against the grand total from the source system. If the totals match, the data is likely accurate. If they do not match, investigate the discrepancy.

2. Row-level reconciliation. Compare individual rows from the dashboard against the source system. This catches errors that cancel out in the total (e.g., one row is overstated and another is understated by the same amount).

3. Sample reconciliation. Compare a random sample of rows from the dashboard against the source system. This is faster than full reconciliation and catches most errors.

4. Cross-metric reconciliation. Compare related metrics against each other. For example, if the dashboard shows total sales and total orders, verify that the average order value (total sales / total orders) matches the source system.

Common Sources of Data Inaccuracy

1. Filter errors. A filter that excludes or includes the wrong rows. For example, a date filter that uses the wrong timezone or a region filter that misspells a region name.

2. Join errors. A join that duplicates rows or excludes rows. For example, a left join that becomes an inner join due to a filter, or a many-to-many join that creates a Cartesian product.

3. Aggregation errors. An aggregation that uses the wrong function. For example, using AVG instead of SUM, or using COUNT instead of COUNTD.

4. Calculation errors. A calculated field or DAX measure that uses the wrong formula. For example, dividing by the wrong denominator or using the wrong date range.

5. Data type errors. A field that is stored as the wrong data type. For example, a date stored as a string, or a number stored as text.

6. Null handling errors. A calculation that does not handle null values correctly. For example, a division that does not check for zero, or a concatenation that does not handle null strings.

Practical Example: Reconciling a Sales Dashboard

Suppose you have a sales dashboard that shows total sales by region. The dashboard shows $10M in total sales. You need to verify this against the source system.

1. Run a SQL query against the source database: SELECT SUM(Revenue) FROM Sales WHERE OrderDate >= '2024-01-01' AND OrderDate < '2025-01-01'. The result is $9.8M.

2. Investigate the discrepancy. The dashboard shows $10M but the source shows $9.8M. The difference is $200K.

3. Check the filters. The dashboard uses a date filter of "2024" but the source query uses a date range of January 1, 2024 to December 31, 2024. The dashboard may be including orders from January 1, 2025 that have a 2024 order date in a different timezone.

4. Check the joins. The dashboard joins the Sales table to the Products table. If the join duplicates rows (e.g., because a product has multiple categories), the total will be overstated.

5. Check the aggregation. The dashboard uses SUM(Revenue) but the source query uses SUM(Revenue). The aggregation function is the same, so this is not the issue.

6. Identify the root cause. The join to the Products table duplicates rows because some products have multiple categories. The fix is to use a distinct product ID in the join or to aggregate the Products table before joining.

7. Verify the fix. After fixing the join, the dashboard shows $9.8M, which matches the source.

Best Practices for Data Accuracy

1. Reconcile totals before publishing. Always compare the dashboard grand total against the source system before releasing the dashboard.

2. Document data lineage. Track where each metric comes from — the source table, the source column, the transformation, and the calculation.

3. Automate reconciliation. Build a reconciliation dashboard that automatically compares dashboard metrics against source metrics and flags discrepancies.

4. Test with known values. If you know the expected result for a specific filter combination, verify that the dashboard produces that result.

5. Involve stakeholders in validation. Ask stakeholders to verify that the numbers match their expectations. They know the business and can catch errors that technical QA misses.

Interaction Testing

Interaction testing is the process of verifying that all interactive features work correctly. This includes cross-filtering, drill-through, parameters, tooltips, and navigation. Interaction testing is essential because broken interactions frustrate users and undermine trust in the dashboard.

Cross-Filtering Tests

Cross-filtering is the most common interaction. To test it:

1. Click a data point on a visual and verify that all other visuals update correctly.

2. Click a legend item and verify that the visual updates to show only the selected item.

3. Select multiple items (using Ctrl+Click or Shift+Click) and verify that the visual updates to show all selected items.

4. Clear the selection and verify that the visual returns to its default state.

5. Test with filters applied. Apply a slicer filter and then click a data point. Verify that the cross-filtering respects the slicer filter.

Drill-Through Tests

Drill-through is the primary navigation pattern for exploring detail. To test it:

1. Right-click a data point and select "Drill Through." Verify that the drill-through page opens.

2. Verify the filters. The drill-through page should show only the data related to the selected data point.

3. Test the back button. Verify that the back button returns to the summary page.

4. Test from multiple data points. Drill through from different data points and verify that the filters are correct each time.

5. Test with different users. Verify that the drill-through works for users with different permissions.

Parameter Tests

Parameters enable what-if analysis. To test them:

1. Change the parameter value and verify that all visuals update correctly.

2. Test the minimum and maximum values. Set the parameter to its minimum and maximum values and verify that the dashboard handles them correctly.

3. Test the default value. Reset the parameter to its default value and verify that the dashboard returns to its default state.

4. Test parameter interactions. If multiple parameters interact (e.g., a discount rate and a target sales), verify that changing one parameter does not break the other.

Tooltip Tests

Tooltips provide additional context when users hover over data points. To test them:

1. Hover over a data point and verify that the tooltip appears.

2. Verify the tooltip content. The tooltip should show the correct values for the hovered data point.

3. Test tooltip interactions. If the tooltip contains interactive elements (e.g., a mini-chart), verify that they work correctly.

4. Test on mobile. Tooltips work differently on mobile devices (tap instead of hover). Test the tooltip on a mobile device.

Navigation Tests

Navigation includes page transitions, bookmarks, and buttons. To test it:

1. Click a navigation button and verify that it takes the user to the correct page.

2. Test bookmarks. If the dashboard uses bookmarks to save filter states, verify that they work correctly.

3. Test deep links. If the dashboard supports deep links (URLs that open the dashboard with specific filters), verify that they work correctly.

4. Test the back button. Verify that the browser back button works correctly within the dashboard.

Performance Under Interaction

Interactions generate additional queries. A dashboard that loads quickly in its default state may become slow when users interact with it. To test performance under interaction:

1. Measure the time for each interaction (click, drill-through, parameter change).

2. Test with realistic data volumes. A dashboard that is fast with 1,000 rows may be slow with 1,000,000 rows.

3. Test with multiple users. A dashboard that is fast for one user may be slow for 100 concurrent users.

4. Identify slow interactions. If an interaction takes more than 3 seconds, investigate and optimize.

Practical Example: Testing a Sales Dashboard

Suppose you have a sales dashboard with cross-filtering, drill-through, and a discount rate parameter.

1. Cross-filtering test. Click the "West" region on the bar chart. Verify that the line chart, map, and table update to show only West region data. Click "East" and verify the update. Clear the selection and verify the default state.

2. Drill-through test. Right-click the "West" region and select "Drill Through." Verify that the drill-through page shows only West region orders. Click the back button and verify that you return to the summary page.

3. Parameter test. Set the discount rate to 10% and verify that the discounted revenue updates. Set it to 50% and verify the update. Reset to 0% and verify the default state.

4. Performance test. Measure the time for each interaction. If the drill-through takes more than 3 seconds, investigate the query.

5. Mobile test. Open the dashboard on a phone. Verify that cross-filtering works (tap instead of hover) and that the layout is readable.

Common Pitfalls

1. Testing only the default state. The default state is the easiest to test but the least likely to reveal bugs. Test with different filters, selections, and parameter values.

2. Testing only on desktop. Mobile devices have different interaction patterns and screen sizes. Always test on mobile.

3. Testing only with one user. Multiple concurrent users can reveal performance and security issues that a single user cannot.

4. Not documenting test results. Document what you tested, what passed, and what failed. This creates a record for future reference.

5. Testing after every change. A change to one visual can break interactions with other visuals. Test after every change, not just before release.

Stakeholder Feedback and Iteration

Stakeholder feedback is the primary mechanism for improving a dashboard. The people who use the dashboard every day know what works and what does not. Gathering and acting on their feedback is the difference between a dashboard that is used and a dashboard that is ignored.

Gathering Feedback

Feedback can be gathered through multiple channels:

1. Structured interviews. Sit down with stakeholders and ask specific questions about the dashboard. What do they use it for? What do they like? What do they dislike? What is missing?

2. Surveys. Send a survey to all dashboard users. Ask them to rate the dashboard on accuracy, performance, usability, and relevance. Include open-ended questions for qualitative feedback.

3. Usage analytics. Track how users interact with the dashboard. Which visuals do they use most? Which filters do they apply? Where do they drop off?

4. Support tickets. Track support tickets related to the dashboard. Recurring issues indicate problems that need to be fixed.

5. Observation. Watch stakeholders use the dashboard. Observe where they struggle, what they ignore, and what they export to Excel.

Prioritizing Feedback

Not all feedback is equal. Some issues are critical (data is wrong), some are important (a feature is missing), and some are nice-to-have (a color change). Prioritize feedback based on:

1. Impact. How many users are affected? A bug that affects all users is higher priority than a bug that affects one user.

2. Severity. How severe is the issue? A data accuracy bug is higher priority than a cosmetic issue.

3. Effort. How much work is required to fix the issue? A quick fix may be higher priority than a complex feature, even if the feature has higher impact.

4. Strategic alignment. Does the feedback align with the dashboard's goals? A feature that supports the dashboard's primary use case is higher priority than a feature that supports a secondary use case.

Iterative Improvement

Dashboard development is iterative. The first version will not be perfect. The goal is to release a minimum viable dashboard, gather feedback, and improve it in cycles.

1. Release a minimum viable dashboard. Include the most important metrics and interactions. Do not try to include everything in the first version.

2. Gather feedback. Use the channels described above to collect feedback from stakeholders.

3. Prioritize feedback. Identify the most impactful changes.

4. Implement changes. Make the changes in a development environment.

5. Test changes. Verify that the changes work correctly and do not break existing functionality.

6. Release changes. Deploy the changes to production.

7. Repeat. Continue the cycle of feedback and improvement.

Managing Stakeholder Expectations

Stakeholders may request features that are not feasible, not aligned with the dashboard's goals, or not the highest priority. Managing expectations is a critical skill.

1. Be transparent. Explain why a feature is not feasible or not the highest priority. Stakeholders are more likely to accept a "no" if they understand the reasoning.

2. Offer alternatives. If you cannot build the requested feature, offer an alternative that addresses the underlying need.

3. Set timelines. If a feature is planned but not yet built, give a realistic timeline for when it will be available.

4. Communicate progress. Regularly update stakeholders on the status of their feedback. Silence creates frustration.

5. Say no when necessary. Not every request can be fulfilled. Saying no to low-priority requests allows you to focus on high-impact work.

Practical Example: Iterating on a Sales Dashboard

Suppose you have released a sales dashboard and gathered feedback from stakeholders.

1. Feedback: "The dashboard is slow." This is a high-impact, high-severity issue. You investigate and find that the data source query is slow. You create an extract and optimize the queries. The dashboard loads 80% faster.

2. Feedback: "I want to see sales by sales rep." This is a medium-impact request. You add a sales rep dimension to the dashboard and a bar chart showing sales by rep.

3. Feedback: "The colors are hard to distinguish." This is a low-severity issue but affects accessibility. You change the color palette to a colorblind-friendly palette.

4. Feedback: "Can I export the data to Excel?" This is a medium-impact request. You enable the "Export data" feature in Power BI or the "Download" option in Tableau.

5. Feedback: "I want a forecast." This is a high-effort request. You explain that a forecast requires statistical modeling and is not feasible in the current sprint. You offer to add it to the roadmap for the next quarter.

Common Pitfalls

1. Gathering feedback but not acting on it. If stakeholders provide feedback and see no changes, they will stop providing feedback. Always act on feedback or explain why you cannot.

2. Trying to please everyone. Not all feedback can be fulfilled. Prioritize based on impact, severity, and strategic alignment.

3. Releasing without stakeholder input. A dashboard that is technically correct but does not meet stakeholder needs is a failure. Involve stakeholders throughout the development process.

4. Ignoring negative feedback. Negative feedback is the most valuable feedback. It tells you what is broken and what needs to be fixed.

5. Not documenting feedback. Document all feedback, the decisions made, and the rationale. This creates a record for future reference and prevents the same issues from recurring.

Documentation and Governance

Documentation and governance are the processes of recording what the dashboard does, where the data comes from, and who is responsible for maintaining it. These processes are essential for long-term sustainability. Without documentation, a dashboard becomes a black box that only the original author understands. Without governance, a dashboard becomes stale and unreliable.

Metric Definitions

Every metric on the dashboard should have a clear definition. The definition should include:

1. Name. The name of the metric as it appears on the dashboard.

2. Description. A plain-language description of what the metric measures.

3. Formula. The calculation used to compute the metric. For example, "Total Sales = SUM(Sales[Revenue]) where OrderDate is in the current year."

4. Source. The data source and table where the metric comes from.

5. Owner. The person responsible for the metric's accuracy.

6. Refresh frequency. How often the metric is updated.

7. Known limitations. Any caveats or edge cases that affect the metric's accuracy.

Data Lineage

Data lineage is the record of where data comes from and how it is transformed. It includes:

1. Source systems. The databases, APIs, or files where the data originates.

2. Extraction process. How the data is extracted from the source system (e.g., SQL query, API call).

3. Transformation process. How the data is transformed (e.g., joins, filters, aggregations).

4. Loading process. How the data is loaded into the dashboard (e.g., extract, live connection).

5. Calculation process. How metrics are calculated (e.g., DAX measures, Tableau calculated fields).

Data lineage is essential for debugging. When a metric is incorrect, data lineage helps you trace the error back to its source.

Dashboard Documentation

Dashboard documentation should include:

1. Purpose. What questions the dashboard answers.

2. Audience. Who the dashboard is for.

3. Data sources. Where the data comes from.

4. Refresh schedule. How often the data is updated.

5. Known issues. Any known problems or limitations.

6. Change log. A record of changes made to the dashboard.

7. Contact information. Who to contact with questions or issues.

Governance Processes

Governance is the set of processes that ensure the dashboard remains accurate, relevant, and secure over time. Key governance processes include:

1. Change management. A process for requesting, approving, and implementing changes to the dashboard.

2. Access control. A process for granting and revoking access to the dashboard.

3. Quality assurance. A process for testing changes before they are released to production.

4. Monitoring. A process for monitoring the dashboard's performance, data freshness, and usage.

5. Deprecation. A process for retiring dashboards that are no longer needed.

Practical Example: Documenting a Sales Dashboard

Suppose you are documenting a sales dashboard.

1. Metric definitions.

   • Total Sales: SUM(Sales[Revenue]) where OrderDate is in the current year. Source: Sales database, Sales table. Owner: Data team. Refresh: Daily. Limitations: Excludes returns.
   • Profit Margin: (SUM(Sales[Revenue]) - SUM(Sales[Cost])) / SUM(Sales[Revenue]). Source: Sales database, Sales table. Owner: Data team. Refresh: Daily. Limitations: Cost data may be incomplete for recent orders.
   • Year-over-Year Growth: (Current Year Sales - Prior Year Sales) / Prior Year Sales. Source: Sales database, Sales table. Owner: Data team. Refresh: Daily. Limitations: Prior year data may be restated.

2. Data lineage.

   • Source: Sales database (SQL Server).
   • Extraction: SQL query that selects all orders from the Sales table.
   • Transformation: Join to Products table for product names, join to Customers table for customer segments.
   • Loading: Tableau extract, refreshed daily at 6 AM.
   • Calculation: DAX measures for Total Sales, Profit Margin, and Year-over-Year Growth.

3. Dashboard documentation.

   • Purpose: Track sales performance against targets.
   • Audience: Sales managers, executives.
   • Data sources: Sales database.
   • Refresh schedule: Daily at 6 AM.
   • Known issues: Cost data for the current month is incomplete.
   • Change log: 2024-01-15: Added Year-over-Year Growth metric.
   • Contact: data-team@company.com.

Common Pitfalls

1. Not documenting metric definitions. Without clear definitions, different people interpret metrics differently. Document every metric.

2. Not tracking data lineage. Without data lineage, debugging is guesswork. Document where data comes from and how it is transformed.

3. Not assigning owners. Without owners, no one is responsible for the dashboard's accuracy. Assign an owner to every metric and every dashboard.

4. Not monitoring the dashboard. Without monitoring, you do not know when the dashboard breaks. Set up monitoring for data freshness, performance, and usage.

5. Not deprecating old dashboards. Old dashboards consume resources and confuse users. Retire dashboards that are no longer needed.