CoursesAI WorkshopCompaniesPricingBlogNewsletter
  • Courses
  • AI Workshop
  • Companies
  • Pricing
  • Blog
  • Newsletter
Subscribe
  • Courses
  • Companies
  • Communities
  • Blog
  • Gift card
  • Newsletter
  • Help
  • Shop
  • ConfAiBot
  • Contact
  • Legal notice
  • General conditions
  • Privacy policy
  • Cookies policy

Course of Spark: Best Practices and Deployment

Standard
Javier FerrerChristian Herrera

By Javier Ferrer y Christian Herrera – Software Design and Architecture y DevOps

Learn how to take Spark to production by following best practices in deployment.

Go to course

💸 First lesson available without registration 💸**

The real challenge when working with Apache Spark is not just to write functional code but to do it efficiently, tested, and production-ready.

In this course, you will discover how to apply good practices in the development, testing, and deployment of Spark, ensuring optimal performance in real environments.

Among other things, in the course we cover:

  • 🎉 Introduction and local environment: Set up your environment and get an overview of the key topics we will cover.
  • ✅ Testing in Apache Spark: Learn how to design and execute effective tests to ensure the quality of your Spark applications.
  • 💨 Query Optimization: Understand the inner workings of Spark and how to apply techniques to improve the efficiency of your queries.
  • 🎸 Spark Deployment: Discover the essential concepts for configuring and deploying Spark in production environments.
  • 🌩️ Performance Analysis with Grafana: Learn to integrate Spark with tools like Grafana to monitor and optimize the performance of your applications.
  • 🚀 Production Deployment: Master the necessary steps to bring a Spark cluster from development to production.
  • 🔜 Conclusion and Next Steps: Define your action plan to implement these practices in real projects.

This course is ideal for developers looking to professionalize their use of Spark, ensuring efficient, maintainable, and production-ready solutions.

Videos of the course

  • ⚡ Course Introduction: What You Will Learn and Project Creation
  • 🗺️ Apache Spark Stack Production Ready: Kafka, Spark Cluster, Spark Thrift Server, Hive Metastore, AWS S3, Docker Compose 🎥

  • 🛋️ Make Your Spark Code Testable: Initial Refactor Isolating Business Logic
  • 🫠 Validate that Your Application Does What You Expect Automatically: End-to-End Tests with Apache Spark
  • 🤏 Unit Tests in Spark: Avoid Triggering Your Build Time
  • 🤔 Let’s Review!
  • 🛠 Let’s Practice! Creating Unit Tests for Our Transformations

  • ⚡ Analyze How Spark Works with Spark Shell and Spark UI: Jobs, Stages, and Tasks
  • 🤔 How to Interpret a Query Plan with explain()
  • 🦉 Help Us Improve
  • 🤔 Let’s Review!
  • 🛠 Let’s Practice! Writing Code from a Query Plan

  • 🛫 Deploy Apache Spark in a Cluster with Docker Compose
  • ☕ Deployment Modes with Apache Spark
  • 🤔 Let’s Review!
  • 🛠 Let’s Practice! Reading Data from Kafka and Deployment with spark-submit

  • 📊 Monitor Apache Spark with Grafana: Detect Bottlenecks in Real-Time
  • 🪄 Optimize Transformations in Apache Spark: Broadcast Joins
  • 🥴 Query Optimization in Non-Uniform Data Environments (AQE)
  • 🤔 Let’s Review!
  • 🛠 Let’s Practice! Practical Exercise

  • 🔥 Creating Infrastructure in Amazon Web Services (AWS): Deploying Spark with State Machine via Step Function
  • 🚀 Deployment Process
  • 🤔 Let’s Review!
  • 🛠 Let’s Practice! Adding a Schedule to a Step Function with Amazon EventBridge

  • 🚶‍➡️ Conclusion and Next Steps
Go to course

Related courses

Standard
Dependabot: Keep your repo up to date, securely and automatically

🛡️ Dependabot: Keep your repo up to date, securely and automatically

Javier FerrerRafa Gómez

Update your dependencies safely without PR spam.

See course
Standard
Design patterns: Inbox and Outbox

📬 Design patterns: Inbox and Outbox

Javier FerrerRafa Gómez

Learn when to use (and when not to use) the Inbox and Outbox patterns.

See course
Standard
Event Bus in a Database [Infrastructure Design]

🚏 Event Bus in a Database [Infrastructure Design]

Javier FerrerRafa Gómez

Add scalability by using your own database as an event bus.

See course
See full catalogue

Pay according to your needs

lite (only monthly)

19 €
per month
  • Essential programming courses to build a solid foundation
Popular

standard

24,92 €
Discount vs monthly of 49 €
Annual payment of 299 €
per month
  • Main catalog to master writing maintainable, scalable, and testable code
  • Receive job offers verified by Codely
  • Company invoice
  • Accumulate renewal discounts

premium

41,58 €
Discount vs monthly of 89 €
Annual payment of 499 €
per month
  • Exclusive AI courses to keep you always up to date
  • Early access to new courses
  • Discount on workshops
  • Receive job offers verified by Codely
  • Company invoice
  • Accumulate renewal discounts

We won't increase the price while your subscription is active

SubscribeSign in