GroupBy and Aggregation

The Split-Apply-Combine Pattern

Every grouped analysis in pandas follows the same three-step mental model: split the data into groups, apply a function to each group, and combine the results into a single output. This pattern is so fundamental that Hadley Wickham formalized it in 2011, and pandas implements it directly through the .groupby() method.

Why does this matter? Because when you understand that every groupby operation is just these three steps, you stop memorizing syntax and start reasoning about what you want. The split step determines which rows belong together. The apply step determines what computation happens per group. The combine step determines the shape of the output.

import pandas as pd

sales = pd.DataFrame({
    'region': ['East', 'West', 'East', 'West', 'East', 'West'],
    'product': ['A', 'A', 'B', 'B', 'A', 'B'],
    'revenue': [100, 150, 200, 250, 120, 300],
    'units': [10, 15, 20, 25, 12, 30]
})

# Split by 'region', apply sum, combine into a Series
region_totals = sales.groupby('region')['revenue'].sum()
print(region_totals)
# region
# East    420
# West    700
# Name: revenue, dtype: int64

The .groupby('region') call does not compute anything. It returns a DataFrameGroupBy object that knows how to split the data. The aggregation function (.sum()) triggers the apply and combine steps. This lazy evaluation is why you can chain operations and why pandas can optimize the execution plan.

A critical insight: the groupby key becomes the index of the result by default. If you want it back as a column, use .reset_index() or pass as_index=False to groupby(). This distinction matters when you need to merge the aggregated result back with another DataFrame.

# Keep the group key as a column
region_totals = sales.groupby('region', as_index=False)['revenue'].sum()
print(region_totals)
#   region  revenue
# 0   East      420
# 1   West      700

Understanding split-apply-combine also explains why certain operations are fast and others are slow. Built-in aggregations like .sum(), .mean(), and .count() are implemented in C and operate on each group's underlying NumPy array without Python-level iteration. Custom Python functions passed to .apply() are slower because pandas must call your function once per group with a DataFrame slice.

Built-in Aggregations and Named Aggregations

pandas provides a set of built-in aggregation methods that cover the vast majority of analytical needs. These methods are optimized and should be your default choice.

# Multiple aggregations at once
stats = sales.groupby('region')['revenue'].agg(['sum', 'mean', 'count', 'std'])
print(stats)
#         sum       mean  count        std
# region                                 
# East    420  140.000000      3   52.915026
# West    700  233.333333      3  104.403065

The .agg() method (short for aggregate) accepts a list of function names as strings, actual functions, or a dictionary mapping columns to functions. When you pass a list, pandas computes each aggregation for the selected column(s) and returns a DataFrame with the aggregation names as columns.

Named aggregations solve a real problem: when you compute multiple statistics on multiple columns, the default output has a MultiIndex for columns that is awkward to work with. Named aggregations let you specify the output column name explicitly using tuples of (column, aggfunc).

result = sales.groupby('region').agg(
    total_revenue=('revenue', 'sum'),
    avg_units=('units', 'mean'),
    transaction_count=('revenue', 'count'),
    max_revenue=('revenue', 'max')
).reset_index()

print(result)
#   region  total_revenue  avg_units  transaction_count  max_revenue
# 0   East            420  14.000000                  3          200
# 1   West            700  23.333333                  3          300

The syntax total_revenue=('revenue', 'sum') reads as: create a column named total_revenue by applying 'sum' to the 'revenue' column. This produces a flat DataFrame with meaningful column names, which is almost always what you want when exporting results or feeding them into a report.

You can also pass custom functions to .agg(). The function receives a Series (one group's values for one column) and must return a scalar.

def coefficient_of_variation(x):
    return x.std() / x.mean()

cv = sales.groupby('region')['revenue'].agg(coefficient_of_variation)
print(cv)
# region
# East    0.377964
# West    0.447523
# Name: revenue, dtype: float64

When choosing between built-in string names and custom functions, prefer built-ins. The string 'sum' dispatches to a C-optimized path. A Python function forces pandas to call it once per group, which is orders of magnitude slower on large data. Reserve custom functions for statistics that pandas does not provide natively.

Multi-Level GroupBy

Real analytical questions often require grouping by more than one dimension. A regional sales analysis might need breakdowns by region and product simultaneously. Multi-level groupby handles this naturally by passing a list of columns to .groupby().

multi = sales.groupby(['region', 'product'])['revenue'].sum()
print(multi)
# region  product
# East    A          220
#         B          200
# West    A          150
#         B          550
# Name: revenue, dtype: int64

The result is a Series with a MultiIndex — a hierarchical index where the first level is region and the second level is product. This structure is powerful but requires understanding how to access its levels.

# Access all rows for a specific region
print(multi.loc['East'])
# product
# A    220
# B    200
# Name: revenue, dtype: int64

# Access a specific (region, product) pair
print(multi.loc[('East', 'B')])
# 200

# Unstack: move the inner level to columns
wide = multi.unstack(level='product')
print(wide)
# product    A    B
# region
# East     220  200
# West     150  550

The .unstack() method pivots the innermost index level into columns, converting the long-format result into a wide-format DataFrame. This is the bridge between grouped analysis and reshaping, and it produces output that is immediately suitable for visualization or reporting.

Multi-level groupby also works with named aggregations. The group keys become a MultiIndex on the rows, and .reset_index() flattens them into columns.

detailed = sales.groupby(['region', 'product']).agg(
    total_revenue=('revenue', 'sum'),
    avg_revenue=('revenue', 'mean'),
    n_transactions=('revenue', 'count')
).reset_index()

print(detailed)
#   region product  total_revenue  avg_revenue  n_transactions
# 0   East       A            220        110.0               2
# 1   East       B            200        200.0               1
# 2   West       A            150        150.0               1
# 3   West       B            550        275.0               2

When grouping by multiple columns, the order matters. groupby(['region', 'product']) groups first by region, then by product within each region. Reversing the order produces the same set of groups but a different sort order in the result. Choose the order that matches how you want to read the output.

A practical consideration: multi-level groupby on large DataFrames can be memory-intensive because pandas must build the MultiIndex. If you only need the aggregated values and not the hierarchical structure, add .reset_index() immediately to flatten the result and release the MultiIndex overhead.

Transform vs Aggregate

One of the most common points of confusion in pandas is the difference between .agg() (or its aliases like .sum()) and .transform(). They look similar but serve fundamentally different purposes.

.agg() returns one value per group. The result has fewer rows than the input. .transform() returns a value for every row in the input, broadcast from the group-level computation. The result has the same shape as the input.

# Aggregate: one row per region
agg_result = sales.groupby('region')['revenue'].sum()
print(agg_result)
# region
# East    420
# West    700

# Transform: one row per original row
transform_result = sales.groupby('region')['revenue'].transform('sum')
print(transform_result)
# 0    420
# 1    700
# 2    420
# 3    700
# 4    420
# 5    700
# Name: revenue, dtype: int64

The transform result assigns each row the total revenue for its region. This is useful when you need to compute a per-row metric that depends on group-level statistics — for example, each transaction's share of its region's total.

sales['region_total'] = sales.groupby('region')['revenue'].transform('sum')
sales['pct_of_region'] = sales['revenue'] / sales['region_total'] * 100
print(sales[['region', 'revenue', 'region_total', 'pct_of_region']])

.transform() also works with multiple columns and custom functions, as long as the function returns a Series of the same length as the group.

# Z-score within each region
def zscore(x):
    return (x - x.mean()) / x.std()

sales['revenue_zscore'] = sales.groupby('region')['revenue'].transform(zscore)

When to use which: use .agg() when you want a summary table with one row per group. Use .transform() when you want to enrich the original DataFrame with group-level context while preserving every row. Confusing these two is a common source of bugs — if your result has the same number of rows as your input but you expected fewer, you probably used .transform() when you meant .agg().