Performance and Vectorization

Why Vectorization Beats Loops

The single most important performance concept in pandas is vectorization: operating on entire arrays at once instead of iterating over elements in Python. The speed difference is not incremental — it is orders of magnitude.

Consider a simple operation: computing a 10% tax on each row of a DataFrame with one million rows.

import pandas as pd
import numpy as np
import time

df = pd.DataFrame({'price': np.random.uniform(10, 100, size=1_000_000)})

# Approach 1: Python loop (slow)
start = time.time()
taxes_loop = []
for price in df['price']:
    taxes_loop.append(price * 0.10)
df['tax_loop'] = taxes_loop
print(f"Loop: {time.time() - start:.3f}s")
# Loop: 0.350s (varies by machine)

# Approach 2: Vectorized (fast)
start = time.time()
df['tax_vectorized'] = df['price'] * 0.10
print(f"Vectorized: {time.time() - start:.4f}s")
# Vectorized: 0.003s

The vectorized version is roughly 100x faster. The reason is fundamental: the loop executes one million Python bytecode instructions, each involving type checking, memory allocation, and reference counting. The vectorized version delegates the entire operation to a single C function that iterates over a contiguous NumPy array with no Python overhead per element.

This is not just about speed. Vectorized code is shorter, more readable, and less error-prone. The loop version requires initializing an empty list, appending in a loop, and assigning the result. The vectorized version is a single expression that states the intent directly.

The mental model: pandas is a thin wrapper around NumPy arrays. Every column is a NumPy array. When you call a vectorized operation, pandas passes the underlying array to a C or Fortran routine that processes it in a single pass. When you loop, pandas extracts each element as a Python object, your code processes it, and the result is stored back — one element at a time.

This principle extends to all pandas operations. String methods via .str, datetime operations via .dt, and arithmetic on entire Series are all vectorized. The .apply() method is the exception — it calls a Python function once per row or column and should be avoided when a vectorized alternative exists.

Vectorized Operations in Practice

The most common performance mistake in pandas is using .apply() or a Python loop when a vectorized method exists. Here are the patterns that matter most.

Arithmetic and comparisons are vectorized by default:

# Vectorized arithmetic
df['total'] = df['price'] * df['quantity']
df['discounted'] = df['price'] * 0.9

# Vectorized comparison
df['is_expensive'] = df['price'] > 50

# Vectorized string operations
df['name_upper'] = df['name'].str.upper()
df['name_length'] = df['name'].str.len()
df['domain'] = df['email'].str.split('@').str[1]

# Vectorized datetime operations
df['year'] = df['date'].dt.year
df['dayofweek'] = df['date'].dt.dayofweek
df['month_name'] = df['date'].dt.month_name()

Conditional logic uses np.where() or np.select() instead of loops:

import numpy as np

# Simple condition: 10% discount for premium customers
df['discount'] = np.where(df['is_premium'], df['price'] * 0.10, 0)

# Multiple conditions with np.select
conditions = [
    df['score'] >= 90,
    df['score'] >= 80,
    df['score'] >= 70
]
choices = ['A', 'B', 'C']
df['grade'] = np.select(conditions, choices, default='F')

Categorical replacement uses .map() with a dictionary:

# Map category codes to labels
category_map = {1: 'Electronics', 2: 'Clothing', 3: 'Food'}
df['category_name'] = df['category_code'].map(category_map)

The .map() method is vectorized and fast for this purpose. Avoid the loop alternative:

# Slow: Python loop
labels = []
for code in df['category_code']:
    labels.append(category_map.get(code, 'Unknown'))
df['category_name'] = labels

Aggregation is always vectorized. The groupby operations you learned earlier use C-optimized routines. The bottleneck is rarely the aggregation itself — it is the data preparation before it.

A practical rule: if you find yourself writing a loop over rows, stop and ask whether a vectorized method exists. Check the pandas documentation for .str, .dt, .cat accessors, and NumPy's universal functions. In most cases, a vectorized alternative exists and is both faster and more readable.

eval() and query() for Efficient Computation

pd.eval() and DataFrame.query() provide performance benefits for specific patterns by avoiding intermediate DataFrame creation and using Numexpr for computation.

query() filters rows using a string expression instead of boolean indexing:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'price': np.random.uniform(10, 100, size=1_000_000),
    'quantity': np.random.randint(1, 20, size=1_000_000),
    'category': np.random.choice(['A', 'B', 'C'], size=1_000_000)
})

# Standard boolean indexing
result_standard = df[(df['price'] > 50) & (df['category'] == 'A')]

# query() — faster for large DataFrames
result_query = df.query('price > 50 and category == "A"')

query() is faster because it evaluates the expression without creating intermediate boolean arrays for each condition. It parses the string into an expression tree and evaluates it in a single pass. The syntax is also more readable for complex conditions.

eval() computes expressions involving multiple columns:

# Standard: creates intermediate Series for each operation
df['revenue'] = df['price'] * df['quantity']
df['profit'] = df['revenue'] - df['cost']

# eval(): computes without intermediates
df.eval('revenue = price * quantity', inplace=True)
df.eval('profit = revenue - cost', inplace=True)

The performance benefit of eval() comes from avoiding the creation of temporary Series objects. For a DataFrame with many columns and complex expressions, this reduces memory allocation and garbage collection overhead.

# Complex expression with eval()
df.eval('''
    revenue = price * quantity
    discount = revenue * 0.1
    net_revenue = revenue - discount
''', inplace=True)

When to use these tools: query() for filtering large DataFrames with multiple conditions, and eval() for computing multiple derived columns in a single pass. For small DataFrames or simple operations, the standard syntax is fine — the performance difference is negligible and the standard syntax is more debuggable.

A caveat: query() and eval() use string expressions, which means you lose IDE autocompletion and static type checking. They are optimization tools, not defaults. Write the standard version first, profile it, and switch to query() or eval() only when profiling shows a bottleneck.

Profiling and Optimization Workflow

Optimizing without profiling is guesswork. The workflow is: measure first, optimize the bottleneck, then measure again to confirm the improvement.

Timing individual operations with %timeit in Jupyter or time.time() in scripts:

import time

start = time.time()
result = df.groupby('category')['price'].sum()
print(f"Groupby: {time.time() - start:.4f}s")

start = time.time()
result = df['price'] * df['quantity']
print(f"Vectorized multiply: {time.time() - start:.4f}s")

Using pandas built-in profiling with .info() and .memory_usage() to understand memory consumption:

# Memory usage per column
print(df.memory_usage(deep=True))

# Total memory in MB
print(f"Total: {df.memory_usage(deep=True).sum() / 1e6:.2f} MB")

Memory matters because pandas operations that exceed available RAM trigger disk swapping, which is orders of magnitude slower than in-memory computation. Reducing memory usage is often the highest-impact optimization.

Optimize data types. The default int64 and float64 types use 8 bytes per value. If your data fits in a smaller type, convert it:

# Downcast numeric types
df['quantity'] = pd.to_numeric(df['quantity'], downcast='integer')
df['price'] = pd.to_numeric(df['price'], downcast='float')

# Convert string columns with few unique values to category
df['category'] = df['category'].astype('category')

A column with 1 million rows and 10 unique string values uses ~80 MB as object dtype but ~1 MB as category dtype. This 80x reduction speeds up all operations on that column, especially groupby and merge.

Use categorical types for groupby keys. When grouping by a string column, converting it to category first can significantly speed up the operation:

df['category'] = df['category'].astype('category')
result = df.groupby('category')['price'].sum()

Chunking for out-of-core data. When the DataFrame does not fit in memory, process it in chunks:

chunks = []
for chunk in pd.read_csv('large_file.csv', chunksize=100_000):
    processed = chunk.groupby('category')['price'].sum()
    chunks.append(processed)

result = pd.concat(chunks).groupby(level=0).sum()

The profiling workflow in practice: start with the standard vectorized approach. If it is fast enough, stop. If not, profile to find the bottleneck. Optimize the bottleneck — usually data types, memory usage, or a hidden loop. Measure again. Do not optimize code that is not on the critical path.