Series and DataFrames

Introduction to pandas

pandas is the primary Python library for data manipulation and analysis. It provides two core data structures — Series and DataFrame — that let you work with tabular data using intuitive, expressive syntax.

Why pandas Exists

Without pandas, analyzing tabular data in Python means writing nested loops over lists of dictionaries, handling missing values manually, and implementing aggregation logic from scratch. pandas provides these operations as built-in methods, optimized with numpy under the hood.

pandas handles the tedious parts of data work: reading files, aligning data by labels, handling missing values, grouping and aggregating, merging datasets. You focus on the analysis, not the plumbing.

The Two Core Structures

Series is a one-dimensional labeled array. It holds a single column of data with an index — labels for each element.

DataFrame is a two-dimensional labeled structure. It is a collection of Series objects sharing the same index. Each column is a Series. Think of it as a spreadsheet or SQL table.

import pandas as pd

# A Series: one column with an index
sales = pd.Series([150, 230, 180], index=["A", "B", "C"])
print(sales)
# A    150
# B    230
# C    180
# dtype: int64

# A DataFrame: multiple columns sharing an index
df = pd.DataFrame({
    "product": ["A", "B", "C"],
    "sales": [150, 230, 180],
    "region": ["East", "West", "East"]
})
print(df)
#   product  sales region
# 0       A    150   East
# 1       B    230   West
# 2       C    180   East

The Index

The index is what distinguishes pandas from raw numpy arrays. It provides label-based access to rows. By default, DataFrames get a numeric index (0, 1, 2, ...), but you can set any column as the index:

df = df.set_index("product")
print(df)
#          sales region
# product
# A          150   East
# B          230   West
# C          180   East

With a meaningful index, you can select rows by label (df.loc["A"]) instead of position. This makes code more readable and robust — it does not break if rows are reordered.

Importing pandas

The convention is import pandas as pd. This is universal in the pandas ecosystem. Every tutorial, documentation page, and production codebase uses pd as the alias. Adopt it from the start.

How pandas Relates to SQL

If you know SQL, pandas maps directly: a DataFrame is a table, a Series is a column, df.head() is SELECT * LIMIT 5, and df.groupby() is GROUP BY. This mental model accelerates learning. The key difference is that pandas operates in memory — your entire dataset must fit in RAM. For data larger than memory, you would use databases or tools like Dask.

Series Attributes

A Series carries metadata that makes it more than a plain array:

s = pd.Series([150, 230, 180], index=["A", "B", "C"], name="sales")

s.name       # "sales"
s.index      # Index(['A', 'B', 'C'], dtype='object')
s.values     # array([150, 230, 180]) — numpy array
s.dtype      # int64
s.shape      # (3,)

The name attribute becomes the column name when a Series is added to a DataFrame. The values attribute gives direct access to the underlying numpy array for operations that pandas does not support. Understanding these attributes helps you debug issues — when a Series behaves unexpectedly, check its index, dtype, and name first.

Creating Series and DataFrames

You can create Series and DataFrames from Python data structures, numpy arrays, or files. This chapter covers the most common construction patterns.

Creating a Series

import pandas as pd

# From a list (default numeric index)
s = pd.Series([10, 20, 30])
# 0    10
# 1    20
# 2    30

# With a custom index
s = pd.Series([10, 20, 30], index=["x", "y", "z"])
# x    10
# y    20
# z    30

# From a dictionary (keys become the index)
s = pd.Series({"a": 10, "b": 20, "c": 30})
# a    10
# b    20
# c    30

Creating a DataFrame

# From a dictionary of lists (most common)
df = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [28, 34, 22],
    "city": ["NYC", "LA", "Chicago"]
})

# From a list of dictionaries (each dict is a row)
records = [
    {"name": "Alice", "age": 28, "city": "NYC"},
    {"name": "Bob", "age": 34, "city": "LA"},
    {"name": "Charlie", "age": 22, "city": "Chicago"}
]
df = pd.DataFrame(records)

# From a list of lists (no column names by default)
data = [["Alice", 28, "NYC"], ["Bob", 34, "LA"], ["Charlie", 22, "Chicago"]]
df = pd.DataFrame(data, columns=["name", "age", "city"])

The dictionary-of-lists approach is most common in practice. Each key becomes a column name, and each list becomes the column data. All lists must be the same length.

Specifying the Index

df = pd.DataFrame(
    {"name": ["Alice", "Bob"], "age": [28, 34]},
    index=["row1", "row2"]
)
#        name  age
# row1  Alice   28
# row2    Bob   34

Creating Empty Structures

# Empty DataFrame with column names
df = pd.DataFrame(columns=["name", "age", "city"])

# Empty Series
s = pd.Series(dtype=float)

Empty structures are useful when you build data incrementally in a loop, though appending row-by-row is slow. For large datasets, collect data in a list and create the DataFrame once at the end.

From numpy Arrays

import numpy as np

arr = np.array([[1, 2, 3], [4, 5, 6]])
df = pd.DataFrame(arr, columns=["a", "b", "c"])
#    a  b  c
# 0  1  2  3
# 1  4  5  6

pandas and numpy interoperate seamlessly. DataFrames store column data as numpy arrays, so numpy operations work directly on DataFrame columns.

Common Construction Patterns in Analysis

In real analysis, you rarely construct DataFrames by hand. You load them from files (pd.read_csv()), databases (pd.read_sql()), or APIs (pd.read_json()). However, understanding construction from Python structures is essential for testing, prototyping, and combining data from multiple sources. When you merge two DataFrames or concatenate them, you are creating new DataFrames from existing ones — the same principles apply.

Validating Construction

After creating a DataFrame, verify its structure immediately:

df = pd.DataFrame({"name": ["Alice", "Bob"], "age": [28, 34]})

assert len(df) == 2, "Expected 2 rows"
assert list(df.columns) == ["name", "age"], "Column names wrong"
assert df["age"].dtype == "int64", "Age should be int64"

These assertions catch construction errors early. In analysis, a DataFrame with wrong column names or unexpected types will produce confusing errors downstream. Validating at construction time localizes the problem to its source.

Inspecting Data

Before analyzing data, you need to understand its structure: what columns it has, what types they are, how many rows, and what the values look like. pandas provides methods for each of these questions.

head() and tail()

head() shows the first 5 rows. tail() shows the last 5. Pass a number to change the count.

df = pd.DataFrame({
    "product": ["A", "B", "C", "D", "E", "F"],
    "sales": [150, 230, 180, 200, 175, 210]
})

df.head()
#   product  sales
# 0       A    150
# 1       B    230
# 2       C    180
# 3       D    200
# 4       E    175

df.head(2)
#   product  sales
# 0       A    150
# 1       B    230

df.tail(2)
#   product  sales
# 4       E    175
# 5       F    210

Always call head() after loading data. It is the fastest way to confirm the data loaded correctly and matches your expectations.

shape, columns, and index

df.shape      # (6, 2) — 6 rows, 2 columns
df.columns    # Index(['product', 'sales'], dtype='object')
df.index     # RangeIndex(start=0, stop=6, step=1)

shape returns a tuple of (rows, columns). columns returns the column labels. index returns the row labels.

info()

info() provides a concise summary: number of rows, column names, non-null counts, and data types.

df.info()
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 6 entries, 0 to 5
# Data columns (total 2 columns):
#  #   Column   Non-Null Count  Dtype
# ---  ------   --------------  -----
#  0   product  6 non-null      object
#  1   sales    6 non-null      int64
# dtypes: int64(1), object(1)
# memory usage: 224.0+ bytes

info() is the first method to call on any new dataset. It reveals missing values (non-null count < total rows) and column types. The object dtype means strings. int64 means integers.

describe()

describe() computes summary statistics for numeric columns: count, mean, standard deviation, min, quartiles, and max.

df.describe()
#            sales
# count    6.000000
# mean   190.833333
# std     26.829708
# min    150.000000
# 25%    176.250000
# 50%    190.000000
# 75%    207.500000
# max    230.000000

describe() gives you a quick statistical profile. Use it to spot outliers (max far from 75th percentile), understand distribution (mean vs. median), and validate data quality (count matches expected rows).

dtypes

df.dtypes
# product    object
# sales       int64
# dtype: object

Each column has a dtype. Common types: int64 (integers), float64 (decimals), object (strings), datetime64 (dates), bool (True/False). Type matters — you cannot do math on object columns. If a numeric column loads as object, it likely contains non-numeric values that need cleaning.

A Systematic Inspection Routine

When you load a new dataset, follow this sequence: (1) df.shape — confirm row and column counts match expectations. (2) df.head() — eyeball the first few rows. (3) df.info() — check types and missing values. (4) df.describe() — review distributions. This routine takes 30 seconds and catches most data quality issues before they corrupt your analysis.

Value Counts for Categorical Columns

describe() only summarizes numeric columns. For categorical columns, use value_counts():

df["category"].value_counts()
# Food     3
# Tech     2
# Name: category, dtype: int64

This shows the frequency of each unique value. It is the fastest way to understand the distribution of a categorical column and spot unexpected values (e.g., "Food" vs "food" vs "FOOD" — these are three different values to pandas). Normalize to get proportions: df["category"].value_counts(normalize=True).

Modifying DataFrames

Real analysis requires modifying data: adding computed columns, renaming for clarity, removing irrelevant columns, and changing types. These operations are the bread and butter of data preparation.

Adding Columns

df = pd.DataFrame({
    "product": ["A", "B", "C"],
    "price": [19.99, 29.99, 9.99],
    "quantity": [10, 5, 20]
})

# Add a computed column
df["revenue"] = df["price"] * df["quantity"]
#   product  price  quantity  revenue
# 0       A  19.99        10   199.90
# 1       B  29.99         5   149.95
# 2       C   9.99        20   199.80

The new column is calculated using vectorized operations — pandas applies the multiplication to every row simultaneously using numpy. This is dramatically faster than a Python loop.

Renaming Columns

# Rename specific columns
df = df.rename(columns={"price": "unit_price", "quantity": "units_sold"})

# Rename all columns
df.columns = ["product", "unit_price", "units_sold", "revenue"]

Use rename() when you want to change a few columns. Assign to df.columns when you want to rename all of them at once.

Removing Columns

# Drop a column (returns a new DataFrame by default)
df = df.drop(columns=["unit_price"])

# Drop in-place (modifies the original)
df.drop(columns=["unit_price"], inplace=True)

# Drop rows by index
df = df.drop(index=[0, 2])

By default, drop() returns a new DataFrame and leaves the original unchanged. Use inplace=True to modify the original. Prefer the default (non-inplace) approach — it is safer because you can verify the result before reassigning.

Changing Data Types

# Convert string to numeric
df["revenue"] = df["revenue"].astype(float)

# Convert to datetime
df["date"] = pd.to_datetime(df["date"])

# Convert to category (saves memory for repeated strings)
df["region"] = df["region"].astype("category")

Type conversion is essential after loading data. CSV files load all columns as object (strings) by default. You must explicitly convert numeric columns before doing math.

Working with Series (Columns)

Every DataFrame column is a Series. You can extract it and use Series methods:

# Extract a column as a Series
revenue = df["revenue"]

# Series methods
revenue.mean()      # 183.216667
revenue.sum()       # 549.65
revenue.max()       # 199.90
revenue.min()       # 149.95
revenue.std()       # 28.87

# String methods (on object columns)
df["product"].str.lower()     # a, b, c
df["product"].str.contains("A")  # True, False, False

The .str accessor provides string methods that operate on every element. Without it, you would need a loop. This vectorized string processing is a key pandas feature.

Chaining Operations

pandas methods return new objects, so you can chain them:

result = (df
    .assign(revenue=df["price"] * df["quantity"])
    .query("revenue > 150")
    .sort_values("revenue", ascending=False)
)

Chaining keeps code concise and readable. Each method transforms the data and passes it to the next. Use parentheses to break chains across lines.

The assign() Method

assign() adds columns without modifying the original DataFrame:

df2 = df.assign(discount=df["price"] * 0.1)
# df is unchanged, df2 has the new column

This is useful in pipelines where you want to keep the original data intact while creating derived versions.

Inserting Columns at Specific Positions

New columns are added at the end by default. Use insert() to place them elsewhere:

df.insert(1, "discount", df["price"] * 0.1)
# Inserts 'discount' at position 1 (after the first column)

The insert() method modifies the DataFrame in-place. The arguments are: position, column name, and values. This is useful when column order matters for readability or when writing to systems that expect a specific column order.