Tabla de Contenidos: “…Chapter 5: Big Data Processing with Apache Spark -- Technical requirements -- Getting started with Spark -- Installing Spark locally -- Spark architecture -- Spark executors -- Components of execution -- Starting a Spark program -- The DataFrame API and the Spark SQL API -- Transformations -- Actions -- Lazy evaluation -- Data partitioning -- Narrow versus wide transformations -- Analyzing the titanic dataset -- Working with real data -- How Spark performs joins -- Joining IMDb tables -- Summary -- Chapter 6: Building Pipelines with Apache Airflow -- Technical requirements -- Getting started with Airflow -- Installing Airflow with Astro -- Airflow architecture -- Airflow's distributed architecture -- Building a data pipeline -- Airflow integration with other tools -- Summary -- Chapter 7: Apache Kafka for Real-Time Events and Data Ingestion -- Technical requirements -- Getting started with Kafka -- Exploring the Kafka architecture -- The PubSub design -- How Kafka delivers exactly-once semantics -- First producer and consumer -- Streaming from a database with Kafka Connect -- Real-time data processing with Kafka and Spark -- Summary -- Part 3: Connecting It All Together -- Chapter 8: Deploying the Big Data Stack on Kubernetes -- Technical requirements -- Deploying Spark on Kubernetes -- Deploying Airflow on Kubernetes -- Deploying Kafka on Kubernetes -- Summary -- Chapter 9: Data Consumption Layer -- Technical requirements -- Getting started with SQL query engines -- The limitations of traditional data warehouses -- The rise of SQL query engines -- The architecture of SQL query engines -- Deploying Trino in Kubernetes -- Connecting DBeaver with Trino -- Deploying Elasticsearch in Kubernetes -- How Elasticsearch stores, indexes and manages data -- Elasticsearch deployment -- Summary -- Chapter 10: Building a Big Data Pipeline on Kubernetes…”

Tabla de Contenidos: “…Table of Contents Introduction to Data Ingestion Principals of Data Access – Accessing your Data Data Discovery – Understanding Our Data Before Ingesting It Reading CSV and JSON Files and Solving Problems Ingesting Data from Structured and Unstructured Databases Using PySpark with Defined and Non-Defined Schemas Ingesting Analytical Data Designing Monitored Data Workflows Putting Everything Together with Airflow Logging and Monitoring Your Data Ingest in Airflow Automating Your Data Ingestion Pipelines Using Data Observability for Debugging, Error Handling, and Preventing Downtime…”