Core Python Data Structures

Lists

A list is an ordered, mutable collection of items. It is the most versatile Python data structure and the one you will use most often in data analysis.

Creating Lists

# Empty list
empty = []

# List of numbers
temperatures = [72, 68, 75, 80, 77]

# List of strings
cities = ["New York", "London", "Tokyo"]

# Mixed types (valid but uncommon in analysis)
mixed = [42, "answer", True]

Lists preserve insertion order. Items are accessed by zero-based index.

Indexing and Slicing

cities = ["New York", "London", "Tokyo", "Paris"]

cities[0]      # "New York" (first item)
cities[-1]     # "Paris" (last item)
cities[1:3]    # ["London", "Tokyo"] (start inclusive, end exclusive)
cities[:2]     # ["New York", "London"] (from start)
cities[2:]     # ["Tokyo", "Paris"] (to end)

Slicing syntax is [start:stop:step]. The step parameter controls stride:

numbers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
numbers[::2]   # [0, 2, 4, 6, 8] (every other element)
numbers[::-1]  # [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] (reversed)

Common List Operations

sales = [150, 230, 180]

# Add item to end
sales.append(200)           # [150, 230, 180, 200]

# Insert at position
sales.insert(1, 175)        # [150, 175, 230, 180, 200]

# Remove by value
sales.remove(175)          # [150, 230, 180, 200]

# Remove by index
sales.pop(0)               # returns 150, list is now [230, 180, 200]

# Length
len(sales)                 # 3

# Membership test
230 in sales               # True

# Sorting (in-place)
sales.sort()                # [180, 200, 230]

# Sorted copy (original unchanged)
ordered = sorted(sales)

When to Use Lists

Use lists when order matters and you need to modify the collection. In analysis, lists typically represent a single column of data — a series of temperatures, a list of customer names, a sequence of daily sales figures.

Lists are the input format for pandas Series and DataFrame columns. When you pass a list to pd.Series(), it becomes a labeled column of data.

Performance Characteristics

• Indexing (lst[i]): O(1) — constant time. Lists are arrays under the hood.
• Append: O(1) amortized — fast.
• Insert at beginning: O(n) — all elements shift right.
• Search (x in lst): O(n) — linear scan.

For large datasets where you search frequently, a set or dict is faster. For sequential access and modifications at the end, lists are ideal.

Lists in Analysis Workflows

When you extract a column from a pandas DataFrame with df['column'].tolist(), you get a Python list. Many Python functions expect lists, not pandas Series. Converting to a list is sometimes necessary for interoperability with libraries that do not understand pandas objects. However, once you convert to a list, you lose pandas' vectorized operations — iterate only when necessary.

Copying Lists

Assigning a list to a new variable does not copy it — both variables reference the same object:

original = [1, 2, 3]
reference = original
reference.append(4)
print(original)  # [1, 2, 3, 4] — original changed!

# To create an independent copy:
copy = original.copy()
# or
copy = original[:]
# or
copy = list(original)

This behavior surprises new Python programmers and causes subtle bugs. In analysis, if you filter a list and want to preserve the original, explicitly create a copy before modifying.

Dictionaries

A dictionary (dict) is an unordered collection of key-value pairs. It maps unique keys to values. Dictionaries are the fastest lookup structure in Python and the natural choice when you need to associate data with identifiers.

Creating Dictionaries

# Empty dict
empty = {}

# Dict with string keys
employee = {
    "name": "Alice Chen",
    "department": "Engineering",
    "salary": 95000
}

# Dict from pairs
pairs = [("a", 1), ("b", 2)]
dict_from_pairs = dict(pairs)  # {"a": 1, "b": 2}

Accessing and Modifying Values

employee = {"name": "Alice Chen", "department": "Engineering", "salary": 95000}

# Access by key
employee["name"]           # "Alice Chen"

# Safe access with default
employee.get("email", "N/A")  # "N/A" (key does not exist)

# Add or update
employee["email"] = "alice@company.com"

# Remove
del employee["salary"]

# Check key exists
"department" in employee   # True

Iterating Over Dictionaries

# Iterate keys (default)
for key in employee:
    print(key)

# Iterate values
for value in employee.values():
    print(value)

# Iterate key-value pairs
for key, value in employee.items():
    print(f"{key}: {value}")

When to Use Dictionaries

Use dictionaries when you need fast lookup by a meaningful key. In analysis, dictionaries represent records — a single row of data where column names are keys. A list of dictionaries is a common intermediate format between raw data and a pandas DataFrame:

records = [
    {"product": "A", "sales": 150},
    {"product": "B", "sales": 230},
    {"product": "C", "sales": 180}
]

df = pd.DataFrame(records)

Dictionaries also power configuration, mapping tables, and counting operations.

Performance Characteristics

• Lookup by key (d[key]): O(1) average — hash table implementation.
• Insert: O(1) average.
• Delete: O(1) average.
• Iteration: O(n).

Dictionaries trade memory for speed. They use more memory than lists because they store hash tables. For analysis workloads, this tradeoff is almost always worth it.

Dictionary Comprehensions

Like lists, dictionaries support comprehension syntax:

prices = {"A": 19.99, "B": 29.99, "C": 9.99}
with_tax = {k: v * 1.08 for k, v in prices.items()}
# {"A": 21.59, "B": 32.39, "C": 10.79}

This is useful for transforming lookup tables — applying a conversion factor, renaming keys, or filtering entries.

Counting with Dictionaries

Dictionaries are the natural tool for counting occurrences:

categories = ["Food", "Tech", "Food", "Tech", "Food"]
counts = {}
for cat in categories:
    counts[cat] = counts.get(cat, 0) + 1
# {"Food": 3, "Tech": 2}

The dict.get(key, default) method returns the current count (or 0 if the key is new), and we store the incremented value back. This pattern is so common that Python provides collections.Counter:

from collections import Counter
counts = Counter(categories)
# Counter({"Food": 3, "Tech": 2})

In pandas, the equivalent is df['column'].value_counts(), which returns a Series of counts sorted in descending order.

Sets and Tuples

Sets and tuples serve different purposes than lists and dicts. Understanding when to use them makes your code more expressive and efficient.

Sets

A set is an unordered collection of unique elements. Sets excel at membership testing and eliminating duplicates.

# Creating sets
unique_cities = {"New York", "London", "Tokyo"}
from_list = set([1, 2, 2, 3, 3, 3])  # {1, 2, 3}

# Add and remove
unique_cities.add("Paris")
unique_cities.remove("London")

# Membership test (O(1) vs O(n) for lists)
"Tokyo" in unique_cities  # True

Set Operations

Sets support mathematical set operations, which are extremely useful in data analysis for comparing groups:

q1_customers = {"Alice", "Bob", "Charlie"}
q2_customers = {"Bob", "Charlie", "Diana"}

# Union: all unique customers
q1_customers | q2_customers  # {"Alice", "Bob", "Charlie", "Diana"}

# Intersection: customers in both quarters
q1_customers & q2_customers  # {"Bob", "Charlie"}

# Difference: Q1 customers not in Q2
q1_customers - q2_customers  # {"Alice"}

# Symmetric difference: in either but not both
q1_customers ^ q2_customers  # {"Alice", "Diana"}

These operations run in O(len(smaller set)) time. For comparing large customer bases, sets are orders of magnitude faster than nested loops over lists.

Tuples

A tuple is an ordered, immutable collection. Once created, it cannot be modified.

# Creating tuples
point = (3, 7)
coordinates = (40.7128, -74.0060)  # (latitude, longitude)

# Access (same as lists)
point[0]   # 3
point[-1]  # 7

# Unpacking
lat, lon = coordinates

# Tuples as dict keys (lists cannot be keys)
locations = {
    (40.7128, -74.0060): "New York",
    (51.5074, -0.1278): "London"
}

When to Use Tuples

Use tuples for fixed collections that should not change — coordinates, database connection parameters, function return values with multiple parts. Tuples are hashable (because they are immutable), so they can be dictionary keys and set elements. Lists cannot.

When to Use Sets

Use sets when you need uniqueness or fast membership testing. Common analysis use cases: deduplicating customer IDs, finding common elements between two datasets, filtering against a blocklist.

Performance Comparison

*amortized for append

Practical Analysis Example

Suppose you have two datasets and need to find overlapping records. With lists, you would write a nested loop — O(n*m) time. With sets, you convert both to sets and use intersection — O(n+m) time. For datasets with millions of rows, this difference is the difference between seconds and hours.

Named Tuples for Readable Records

When tuples have many elements, accessing them by position (record[3]) becomes unreadable. collections.namedtuple gives names to positions:

from collections import namedtuple

Sale = namedtuple("Sale", ["product", "quantity", "price"])
s = Sale("Widget", 5, 9.99)

s.product    # "Widget" — readable
s.quantity   # 5
s[0]         # "Widget" — still works as a regular tuple

Named tuples are immutable like regular tuples, so they can be dictionary keys. They are a lightweight alternative to full classes when you need structured data without methods. In analysis, named tuples are useful for representing rows when you want readability without the overhead of a DataFrame.

List Comprehensions and Practical Patterns

List comprehensions are a concise syntax for creating lists from existing iterables. They replace multi-line loops with a single readable expression and are faster than equivalent for loops.

Basic Syntax

# Traditional loop
squares = []
for n in range(10):
    squares.append(n ** 2)

# List comprehension (equivalent)
squares = [n ** 2 for n in range(10)]
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

The syntax is [expression for item in iterable]. The expression transforms each item before adding it to the new list.

Filtering with Conditions

Add an if clause to include only items that meet a condition:

numbers = [15, 23, 8, 42, 16, 4, 31]

# Only even numbers
evens = [n for n in numbers if n % 2 == 0]
# [8, 42, 16, 4]

# Only numbers greater than 20, squared
large_squares = [n ** 2 for n in numbers if n > 20]
# [529, 1764, 961]

Transforming Data

List comprehensions shine in data cleaning and transformation:

raw_names = ["  Alice Chen  ", "BOB SMITH", "charlie brown  "]

# Strip whitespace and title-case
clean_names = [name.strip().title() for name in raw_names]
# ["Alice Chen", "Bob Smith", "Charlie Brown"]

# Extract domain from emails
emails = ["alice@company.com", "bob@gmail.com", "charlie@company.com"]
domains = [email.split("@")[1] for email in emails]
# ["company.com", "gmail.com", "company.com"]

# Convert types
price_strings = ["$19.99", "$29.99", "$9.99"]
prices = [float(p.replace("$", "")) for p in price_strings]
# [19.99, 29.99, 9.99]

Dictionary and Set Comprehensions

The same pattern works for dictionaries and sets:

# Dictionary comprehension
prices = {"A": 19.99, "B": 29.99, "C": 9.99}
with_tax = {k: v * 1.08 for k, v in prices.items()}
# {"A": 21.59, "B": 32.39, "C": 10.79}

# Set comprehension
tags = ["python", "data", "python", "analysis", "data"]
unique_tags = {tag for tag in tags}
# {"python", "data", "analysis"}

When to Avoid Comprehensions

Comprehensions are powerful but not always readable. Avoid them when:

1. The logic is complex. If you need nested conditionals or multiple transformations, a regular loop with named variables is clearer.
2. You need side effects. Comprehensions create new collections. If you are modifying an existing structure or calling functions with side effects (printing, writing files), use a loop.
3. The expression is long. If the transformation spans multiple lines, extract it into a named function and call it from the comprehension.

Nested Comprehensions

For working with matrix-like data:

matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

# Flatten a matrix
flat = [item for row in matrix for item in row]
# [1, 2, 3, 4, 5, 6, 7, 8, 9]

Read nested comprehensions left to right: the outer loop first, then the inner loop. This example iterates each row, then each item within that row.

Comprehensions vs. map() and filter()

Before comprehensions, Python programmers used map() and filter() with lambda functions. Comprehensions are more readable and generally preferred:

# Old style
result = list(map(lambda x: x * 2, filter(lambda x: x > 5, numbers)))

# Comprehension style (preferred)
result = [x * 2 for x in numbers if x > 5]

The comprehension version is easier to read because the transformation and filter are visible in one place, not buried inside function calls.