AWS Glue & ETL

Serverless ETL Concepts

Glue is serverless Spark (plus lightweight Python-shell jobs) wrapped around the Data Catalog. You get crawlers that infer schema, an authoring environment (Glue Studio), job bookmarks for incremental state, and triggers — no clusters to size or patch.

Fit it to the workload: batch transforms over S3 data measured in minutes to hours with SQL/Spark logic. Choose Lambda when the job is short and simple (15-minute timeout ceiling, limited memory). Choose EMR when you need heavy custom clusters, specific Hadoop ecosystem versions, or tightly tuned instances. Glue bills per DPU-hour consumed during the run, so cost tracks data volume and complexity, not idle time.

Jobs, Catalog, and Bookmarks

A typical job reads a cataloged table, transforms, and writes Parquet:

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
job = Job(context)
job.init(args["JOB_NAME"], args)

df = spark.read.table("raw_events")
df.filter("status = 'ok'") \
  .write.mode("append") \
  .parquet("s3://lake/curated/events/")

job.commit()

Job bookmarks remember which input keys were processed, so the next run picks up only new files — job.commit() advances that state. Trigger runs from S3 notifications via EventBridge, and keep table metadata in sync by having the sink update the catalog or re-running a crawler after writes.

Pitfalls and Operations

Schema drift is the top failure: a new column or type change makes reads fail or silently null data — resolve with explicit mappings (DynamicFrame ResolveChoice/ApplyMapping) or by freezing schemas upstream. Bookmarks only advance on a successful commit; a crash mid-job reprocesses the batch on the next run, so downstream writes must be idempotent or you double-count.

Parallel workers routinely emit thousands of small output files — coalesce or repartition before writing. Missing IAM permissions on S3 prefixes or KMS keys surface as opaque AccessDenied at task level. Tune worker count to data size: too many DPUs on a small job burns money on coordination overhead.