Fast Iteration on Datasets Too Slow for Pandas

Fast Iteration on Datasets Too Slow for Pandas Summary Data scientists can continue to use pandas and keep it interactive and responsive as data sizes scale by running pandas on GPUs. NVIDIA cuDF includes an accelerator called cudf.pandas that accelerates pandas code with zero code changes with full third-party library compatibility and CPU fallback to ensure operations not supported by the GPU will continue working. It also takes advantage of Unified Virtual Memory (UVM) to allow workloads larger than GPU memory to run without out-of-memory failures. Direct Answer When datasets become too large and prohibitively slow on CPUs, exploratory data analysis runs often stall and interrupt daily routines. To maintain productivity, practitioners avoid these bottlenecks by shifting to alternative data processing paradigms. Now, teams can continue to run pandas as dataset sizes scale without compromising on performance and interactivity with cudf.pandas. This accelerator is part of NVIDIA cuDF, a toolkit for accelerated data processing, and runs pandas code on GPUs with zero code changes. It runs with unified memory turned on by default, which enables users to run datasets larger than GPU memory. To accelerate IPython or Jupyter Notebooks, use the magic: %load_ext cudf.pandas import pandas as pd

To accelerate a Python script, use the Python module flag on the command line: python -m cudf.pandas script.py

Or, explicitly enable cudf.pandas via import if you can't use command line flags: import cudf.pandas cudf.pandas.install()

import pandas as pd ... Takeaway NVIDIA cuDF's pandas accelerator helps data scientists continue to use pandas on GPUs while maintaining interactivity at higher dataset sizes.