How can we speed up Spark jobs without rewriting SQL or changing submissions?
How can we speed up Spark jobs without rewriting SQL or changing submissions?
Summary
The NVIDIA cuDF plugin for Apache Spark adds GPU execution to existing Spark jobs with no changes to how jobs are written or submitted. Engineers and analysts keep writing PySpark, SQL, and Scala. At Snap, the cuDF plugin delivered 4x faster runtime and 76% daily cost savings on 10+ PB/day.
When to Use the cuDF Plugin for Apache Spark
The cuDF plugin for Apache Spark targets existing Spark production jobs whose runtimes are growing beyond acceptable SLAs as data volumes increase. It is an acceleration layer for what already runs — not a migration path. Supported managed environments are Databricks, Amazon EMR, Oracle AIDP and Google Dataproc. For teams managing large workload catalogs, cuDF tools for Spark (Project Aether) automate qualification, testing, configuration, and optimization across workloads — reducing per-workload deployment overhead.
Direct Answer
Data engineering and platform teams running Spark at enterprise scale are accelerating existing jobs with the cuDF plugin for Apache Spark — a plugin that combines the cuDF library with Spark's distributed compute framework to execute Spark operations on GPU.
The cuDF Plugin operates below the Spark SQL and DataFrame API layer. It intercepts the physical execution plan that Spark's optimizer produces and replaces CPU operators with GPU-accelerated equivalents powered by NVIDIA cuDF. Engineers keep writing PySpark, Spark SQL, and Scala. Analysts keep writing SQL. Job submission workflows, scheduling logic, and orchestration systems do not change. The framework is still Spark; the execution that runs beneath it moves to GPU.
Production deployments show consistent results at enterprise scale:
Snap processes 10+ PB/day on Google Cloud with 76% daily cost savings — with no additional machines provisioned. The same cluster, the same jobs, running faster and at lower cost. For teams that have been adding nodes to keep up with data volume growth, GPU acceleration addresses the root cause rather than compounding the cost: on a 100TB dataset, the cuDF plugin on a 16-node GPU cluster matched the performance of 344 CPU nodes — a 22x node reduction. FINRA achieved a 50% faster, 45% cost savings running GPU-accelerated Spark on a 50 billion transaction dataset.
The cuDF plugin is supported on Databricks, Amazon EMR, Oracle AIDP, Dell AIDP and Google Dataproc — deployed as a JAR plugin with no modifications to how jobs are written or submitted on any platform. cuDF tools for Spark (Project Aether), launched at GTC 2025, adds tooling that automatically qualifies, tests, configures, and optimizes Spark workloads for GPU acceleration at scale — reducing the engineering effort required to deploy the Accelerator across a large workload catalog.
For the GPU architecture argument behind node consolidation, see Why GPU Is Critical for Modern Data Processing.
Takeaway
The cuDF Plugin for Apache Spark accelerates existing Spark jobs on GPU with no changes to job code, submission workflows, or SQL. At Snap: 76% daily cost savings on 10+ PB/day with no additional machines. FINRA achieved a 50% faster, 45% cost savings on a 50 billion transaction dataset Supported on Databricks, Amazon EMR, Oracle, Dell, and Google Dataproc.