Exploratory Data Analysis Reports

Why EDA Comes Before Everything

Exploratory Data Analysis is the first rigorous pass over a new dataset. It is not a formality. It is the process that determines whether your subsequent analysis will be valid or garbage.

The purpose of EDA is threefold:

1. Understand the data. What variables exist? What are their types? What are their distributions? What are their ranges? You cannot analyze what you do not understand.

2. Detect data quality issues. Missing values, duplicates, inconsistent formats, impossible values, and outliers — these problems are present in every real dataset. If you do not find them in EDA, they will silently corrupt your models and conclusions.

3. Generate hypotheses. EDA reveals patterns, relationships, and anomalies that suggest hypotheses for further investigation. Without EDA, you are guessing.

The EDA Workflow

A systematic EDA follows a predictable structure. You should be able to perform it on any new dataset without thinking about what to do next.

Step 1: Data Overview

Start with the basics — shape, column types, and a preview of the data.

import pandas as pd

df = pd.read_csv('dataustomer_data.csv')

print(f"Shape: {df.shape}")
print(f"\nColumn types:")
print(df.dtypes)
print(f"\nFirst 5 rows:")
print(df.head())
print(f"\nBasic info:")
print(df.info())

This tells you the size of the dataset, the data types (which determine what analyses are valid), and a preview of the actual values.

Step 2: Data Quality Assessment

Check for the five universal data quality problems:

# Missing values
missing = df.isnull().sum()
missing_pct = (missing / len(df)) * 100
print(pd.DataFrame({'missing': missing, 'pct': missing_pct}))

# Duplicates
duplicates = df.duplicated().sum()
print(f"Duplicate rows: {duplicates}")

# Unique values per column
print(df.nunique())

# Impossible values (e.g., negative age)
print(df[df['age'] < 0])

Step 3: Univariate Analysis

Profile each variable individually — distribution, central tendency, spread, and shape.

# Numeric variables
print(df.describe())

# Categorical variables
print(df['category'].value_counts())
print(df['category'].value_counts(normalize=True))

Step 4: Bivariate Analysis

Examine relationships between pairs of variables.

# Correlation matrix
print(df.corr(numeric_only=True))

# Cross-tabulation for categorical variables
print(pd.crosstab(df['category'], df['region']))

Step 5: Multivariate Analysis

Look at interactions between three or more variables — groupby aggregations, pivot tables, and segmented analysis.

print(df.groupby('category')['sales'].agg(['mean', 'median', 'std', 'count']))

What a Good EDA Report Contains

An EDA report is not a dump of every statistic you calculated. It is a curated summary that answers specific questions:

1. What does the data look like? — shape, size, types, ranges
2. What quality issues exist? — missing values, duplicates, outliers, inconsistencies
3. What are the key distributions? — central tendency, spread, shape for important variables
4. What relationships exist? — correlations, group differences, patterns
5. What questions does this raise? — hypotheses for further analysis

The report should be decision-oriented. Every finding should connect to a business question or a next step in the analysis.

Why Data Quality Comes First

The most sophisticated analysis built on dirty data is worthless. If 20% of your revenue values are missing, any revenue analysis is suspect. If dates are in three different formats, any time-series analysis will fail. If customer IDs are duplicated, any customer-level analysis is inflated.

Data quality assessment is not a preliminary step to rush through. It is the foundation that determines whether everything downstream can be trusted.

The Five Universal Data Quality Problems

1. Missing Values

Missing values are the most common data quality issue. They arise from system failures, data entry errors, survey non-response, and integration problems between systems.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'customer_id': [1, 2, 3, 4, 5, 6, 7, 8],
    'age': [25, 30, None, 45, None, 35, 28, 50],
    'income': [50000, None, 60000, 80000, 45000, None, 55000, 200000],
    'city': ['NYC', 'LA', 'NYC', None, 'Chicago', 'LA', 'NYC', 'Chicago']
})

# Count and percentage of missing values
missing = df.isnull().sum()
missing_pct = (missing / len(df)) * 100
quality_report = pd.DataFrame({
    'missing_count': missing,
    'missing_pct': missing_pct
})
print(quality_report)

Once you find missing values, you must decide how to handle them:

• Remove rows — if missing values are rare (< 5%) and randomly distributed
• Impute — fill with mean, median, mode, or a model-based estimate
• Flag — create a binary indicator column and keep the missing values
• Leave as-is — if the analysis method can handle missing values

The choice depends on the amount of missingness, the mechanism behind it (missing at random vs. not missing at random), and the analysis you plan to perform.

2. Duplicate Records

Duplicates inflate counts, distort averages, and violate the assumption that each row is an independent observation.

# Find duplicates
duplicates = df[df.duplicated(keep=False)]
print(duplicates)

# Remove duplicates
df_clean = df.drop_duplicates()
print(f"Removed {len(df) - len(df_clean)} duplicate rows")

Always check for duplicates before any aggregation. A single duplicated row can shift a mean enough to change a business decision.

3. Inconsistent Formats

The same value represented in different ways — "NYC", "New York", "new york", "N.Y.C." — will be treated as distinct categories by any analysis.

# Standardize text data
df['city'] = df['city'].str.strip().str.title()

# Standardize date formats
df['date'] = pd.to_datetime(df['date'], errors='coerce')

4. Impossible or Invalid Values

Values that violate logical constraints — negative ages, future dates, percentages above 100, zip codes with letters — indicate data entry errors or system bugs.

# Check for impossible values
print(df[df['age'] < 0])        # negative age
print(df[df['age'] > 120])      # unrealistic age
print(df[df['pct'] > 100])     # percentage above 100

5. Outliers

Not all outliers are errors. Some are genuine extreme values that deserve investigation. The key is to identify them and understand their source before deciding whether to keep, transform, or remove them.

# Identify outliers using IQR
Q1 = df['income'].quantile(0.25)
Q3 = df['income'].quantile(0.75)
IQR = Q3 - Q1

outliers = df[(df['income'] < Q1 - 1.5 * IQR) | (df['income'] > Q3 + 1.5 * IQR)]
print(f"Income outliers: {len(outliers)}")
print(outliers)

Documenting Data Quality Issues

Every data quality issue you find should be documented with:

1. What — the specific problem (e.g., "15% of income values are missing")
2. Where — which columns and how many rows are affected
3. Impact — how the issue affects the analysis
4. Action — what you did about it (removed, imputed, flagged)

This documentation is part of your EDA report. It tells stakeholders exactly what was wrong with the data and what you did to address it.

Univariate Profiling: Understanding Each Variable

Once data quality issues are addressed, profile each variable individually. For numeric variables, compute the full set of descriptive statistics:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'sales': np.random.lognormal(mean=8, sigma=1.5, size=1000),
    'customer_age': np.random.normal(loc=35, scale=10, size=1000).clip(18, 80),
    'satisfaction_score': np.random.beta(a=7, b=2, size=1000) * 5
})

for col in df.columns:
    print(f"\n=== {col} ===")
    print(f"  Count: {df[col].count()}")
    print(f"  Mean: {df[col].mean():.2f}")
    print(f"  Median: {df[col].median():.2f}")
    print(f"  Std: {df[col].std():.2f}")
    print(f"  Min: {df[col].min():.2f}")
    print(f"  Max: {df[col].max():.2f}")
    print(f"  Skewness: {df[col].skew():.2f}")
    print(f"  Kurtosis: {df[col].kurtosis():.2f}")
    print(f"  IQR: {df[col].quantile(0.75) - df[col].quantile(0.25):.2f}")

For each variable, ask:

1. Is the distribution symmetric or skewed? — determines whether mean or median is the better summary
2. Are there outliers? — do they represent errors or genuine extreme values?
3. Is the range reasonable? — are there impossible values?
4. What is the shape? — unimodal, bimodal, uniform?

For categorical variables, compute frequency distributions:

print(df['category'].value_counts())
print(df['category'].value_counts(normalize=True).round(3))

Ask:

1. How many categories? — too many categories may need grouping
2. Is the distribution balanced? — rare categories may need to be combined
3. Are there miscellaneous categories? — "Other", "Unknown", "N/A" often hide meaningful patterns

Bivariate Profiling: Understanding Relationships

Once individual variables are understood, examine relationships between them.

Numeric vs Numeric:

# Correlation matrix
corr = df.corr(numeric_only=True)
print(corr)

# Identify strong correlations
strong_corr = []
for i in range(len(corr.columns)):
    for j in range(i + 1, len(corr.columns)):
        r = corr.iloc[i, j]
        if abs(r) > 0.7:
            strong_corr.append((corr.columns[i], corr.columns[j], r))

for var1, var2, r in strong_corr:
    print(f"{var1} vs {var2}: r = {r:.3f}")

Categorical vs Numeric:

# Group statistics
print(df.groupby('category')['sales'].agg(['mean', 'median', 'std', 'count']))

# Identify which groups differ most
group_means = df.groupby('category')['sales'].mean()
print(f"Highest mean: {group_means.idxmax()} ({group_means.max():.2f})")
print(f"Lowest mean: {group_means.idxmin()} ({group_means.min():.2f})")

Categorical vs Categorical:

# Cross-tabulation
ct = pd.crosstab(df['category'], df['region'], normalize='index')
print(ct.round(3))

Multivariate Profiling: Segmented Analysis

The most interesting patterns often emerge when you segment by multiple variables:

# Sales by category and region
pivot = df.pivot_table(
    values='sales',
    index='category',
    columns='region',
    aggfunc='mean'
)
print(pivot.round(2))

# Identify the best and worst segments
print(f"\nBest segment: {pivot.stack().idxmax()} = {pivot.stack().max():.2f}")
print(f"Worst segment: {pivot.stack().idxmin()} = {pivot.stack().min():.2f}")

Segmented analysis reveals interactions that aggregate statistics hide. Overall sales might be flat, but one segment could be growing rapidly while another is declining. Only by segmenting do you see the full picture.

From Analysis to Report

The EDA report is the deliverable that communicates your findings to stakeholders. It is not a notebook dump. It is a structured document that tells a story about the data.

Report Structure

1. Executive Summary

One paragraph that answers: What did you find? What are the most important insights? What questions does this raise?

"Analysis of 50,000 customer transactions reveals that sales are highly right-skewed (skewness = 3.2), with 5% of customers generating 40% of revenue. Three data quality issues were identified: 12% missing values in the income field, 1,200 duplicate records, and 340 impossible negative ages. After cleaning, the data shows a strong positive correlation (r = 0.78) between engagement score and purchase frequency."

2. Data Overview

• Source of the data
• Time period covered
• Number of rows and columns
• Key variables and their types

3. Data Quality Findings

Present each issue with its impact and resolution:

4. Key Distributions

For each important variable, present:

• Central tendency (mean, median)
• Spread (std, IQR)
• Shape (skewness, modality)
• A brief interpretation

"Customer age is approximately symmetric (skewness = 0.1) with a mean of 38.2 and standard deviation of 12.4. The distribution is unimodal, suggesting a single customer population. The range (18–79) is reasonable for an adult customer base."

5. Key Relationships

Present the most important correlations and group differences:

"Sales and engagement score have a strong positive correlation (r = 0.78, p < 0.001). Customers in the top quartile of engagement spend 3.2x more than those in the bottom quartile. The relationship is approximately linear, as confirmed by scatter plot inspection."

6. Hypotheses and Next Steps

Every EDA should generate questions for further investigation:

"The strong correlation between engagement and sales suggests that increasing engagement may drive revenue. However, this is an observational finding — a controlled experiment would be needed to establish causation. The bimodal distribution of session duration suggests two distinct user behaviors that warrant separate analysis."

Principles of Effective EDA Reports

1. Lead with insights, not process.

Stakeholders do not need to see every step of your analysis. They need to know what you found and what it means. Put the most important findings first.

2. Connect every finding to a business question.

"Sales are right-skewed" is a statistical observation. "A small number of high-value customers generate a disproportionate share of revenue, so retention efforts should focus on this segment" is a business insight.

3. Be honest about limitations.

If the data has quality issues, say so. If a correlation might be spurious, say so. If a finding is based on a small sample, say so. Credibility is built on honesty, not certainty.

4. Use visualizations purposefully.

Every chart should answer a specific question. A histogram shows distribution shape. A scatter plot shows relationships. A bar chart compares groups. Do not include charts that do not teach something.

5. Make it reproducible.

Include the code or at least the methodology so that another analyst can reproduce your findings. An EDA that cannot be reproduced is not science — it is anecdote.

Common EDA Report Mistakes

1. Reporting statistics without interpretation. A table of means and standard deviations is not an insight. Explain what the numbers mean.

2. Ignoring data quality issues. Presenting analysis results without acknowledging missing values, duplicates, or outliers undermines trust in the entire report.

3. Overlooking the shape of distributions. Reporting only means and standard deviations without checking skewness, modality, or outliers can lead to misleading conclusions.

4. Correlation without context. A correlation coefficient without a scatter plot, without discussion of confounders, and without the causation caveat is irresponsible.

5. No next steps. An EDA that does not generate hypotheses or suggest further analysis is incomplete. The goal of EDA is not just to describe — it is to guide action.