A CPU user-defined function can interrupt an otherwise GPU-accelerated Apache Spark query. The cuDF plugin cannot generally translate arbitrary UDF code: it converts columnar data to rows, runs the function on the CPU, then converts the result back. NVIDIA's October 13 guide describes agent skills for rewriting a UDF as Spark SQL, a columnar RapidsUDF, or a RapidsUDF backed by CUDA. The useful question for a data engineer is whether one rewrite produces the same answers, stays on the GPU in the relevant plan, and improves the actual job.

This is a documentation-based guide. BIG CHANGE has not run the skills, tested a UDF, or measured a Spark workload.

The big change

  • What changed: NVIDIA has published a staged agent workflow for generating UDF tests, converting a function, judging the conversion, benchmarking CPU and GPU versions, and optionally optimizing a cuDF implementation.
  • Who it helps: Engineers with a GPU-capable RAPIDS/cuDF Spark pipeline and a specific CPU UDF that interrupts it.
  • What to verify: Matching outputs on representative edge cases, an execution plan without the targeted CPU fallback, and repeatable job-level performance on the intended hardware.

Before you start

Select one UDF that you are authorized to share with the coding agent. Keep credentials, sensitive input rows and proprietary datasets outside the prompt unless that agent and its environment are approved for them. Provide the UDF source, its registration and call site, relevant schemas, expected null and error behavior, and a small permitted test fixture. Record the existing Spark job's plan and timing so the comparison has a baseline.

The skills README is the versioned starting point for the workflow. It calls for Maven and a JDK to build UDFs, plus a compatible Spark/RAPIDS/cuDF/CUDA stack and a GPU. Check the current plugin compatibility guide against your own Spark distribution, Scala version, driver, CUDA build and GPU before installing anything. A cloud GPU is an alternative to local hardware, with instance time and storage billed by the provider. Agent/model use may also have a cost. The README supplies a project scaffold and examples, but it cannot establish compatibility or performance for your cluster.

Rewrite and check one function

  1. Lock down the original behavior. Ask the agent to use udf-gen-test on the selected UDF and inspect the generated tests yourself. Add real cases from the contract: nulls, empty strings or collections, malformed inputs, boundary values, numeric precision, ordering and error behavior as applicable. Run the original UDF against them. The expected output is a passing baseline test suite, not a rewrite yet. NVIDIA's example temperature function explicitly returns null for null input; its SQL rewrite is useful only if that behavior also survives your actual data types and Spark settings.
  2. Choose the simplest supported target. For logic expressible through Spark SQL or DataFrame expressions, ask for udf-convert-to-sql. This exposes the expression to Spark's planner and may let the plugin replace supported operations with GPU equivalents. If SQL cannot express the required logic cleanly, ask for udf-convert-to-cudf: its RapidsUDF.evaluateColumnar receives GPU column vectors and composes cuDF operations. Reserve udf-convert-to-cuda for logic that needs a custom kernel through JNI and whose extra implementation and maintenance burden is justified. These are choices to evaluate, not a promise that every operation runs on a GPU.
  3. Review the conversion before accepting it. Compare the rewritten implementation with the original and run both on the same approved test cases. The repository also lists udf-judge-conversion for a separate review of test gaps and conversion bugs. Check types, null propagation, rounding, exceptions, ordering and resource handling. For a cuDF rewrite, inspect the lifetime of temporary columns and scalars; the tutorial's Scala example uses scoped resources and returns the result column. A passing generated test suite is useful evidence, but a human still has to decide whether it covers the function's contract.
  4. Inspect the executed plan. Run the changed query with the cuDF plugin enabled and inspect the plan and RAPIDS diagnostics for the target operator and nearby conversions. NVIDIA's plugin overview says unsupported SQL operations fall back to Spark CPU execution. If the plan still shows the UDF or surrounding operators on the CPU, identify the unsupported expression or type and revisit the target choice. Keep the before and after plans with the test result. A class named RapidsUDF alone does not prove the entire job stays on the GPU.
  5. Benchmark the job you intend to run. Use udf-benchmark to compare the original and changed implementations under the same inputs, cluster size, configuration and measurement method. Record elapsed job time, the input size, output equivalence and GPU/CPU resource use. Repeat runs to spot startup, cache or contention effects. If a correct cuDF version is slow, the repository offers udf-optimize-cudf to profile and revise it; rerun correctness and plan checks after each change. The expected result is a comparison you can reproduce, even if the rewrite loses.

What a successful result looks like

The candidate is ready for an ordinary code review when the chosen UDF's outputs match its documented behavior on the approved cases, the executed plan no longer contains the targeted CPU fallback, and a representative benchmark shows a worthwhile end-to-end result. If any check fails, keep the original implementation, reduce the rewrite to the failing expression, and rerun that stage. A semantic mismatch takes precedence over a faster benchmark; an unchanged plan means the intended GPU path has not been established.

NVIDIA reports one production example in which a cuDF-enabled UDF reduced operator time from 274.55 to 12.78 hours on a 600-billion-row workload. Those are vendor-reported operator times for that workload, not a forecast for another UDF or an end-to-end job result. The broader 22x claim should be read in that context. GPU capacity, data movement, unsupported operations, code review and cloud charges can change the outcome for your pipeline.

Sources & further reading

  • NVIDIA Technical Blog, October 13, 2026: explains CPU fallback, the SQL/cuDF/CUDA choices, the four workflow stages and the company's single cited performance example. Its performance figures are vendor reported.
  • NVIDIA cuDF Spark skills, pinned repository revision: versioned entry point for the scaffold and individual skills. The pinned GitHub page was unavailable to our retrieval route; the workflow detail was cross-checked against NVIDIA's tutorial and the accessible skills README on the repository's main branch.
  • cuDF plugin for Apache Spark overview: current setup and compatibility information, plus the explicit rule that unsupported operations use Spark CPU execution. Check the release-specific compatibility list before implementation.