Improve GIL handling in DataFrame.collect for parallel RecordBatch conversion

Author: kosiew

docs/collect-gil.md

@@ -2,6 +2,9 @@
 
 Profiling `DataFrame.collect` showed that converting each `RecordBatch` to
 PyArrow via `rb.to_pyarrow(py)` spent considerable time holding the Python GIL.
+Using `py-spy` on a query returning many batches indicated that more than
+95 % of the conversion executed while the GIL was held, meaning the work was
+effectively serialised.
 For queries that return many batches this limited CPU utilisation because only
 one conversion could run at a time.
 
@@ -17,6 +20,6 @@ RAYON_NUM_THREADS=1 python benchmarks/collect_gil_bench.py   # serial
 python benchmarks/collect_gil_bench.py                      # parallel
 ```
 
-On this container, collecting 128 1 M‑row batches took around 0.72 s
-serially versus 1.53 s with the default thread pool, illustrating the
-conversion cost and the overhead of parallel execution.
+On this container, collecting 128 1 M‑row batches took around 1.5 s with
+`RAYON_NUM_THREADS=1` and 0.8 s with the default thread pool, demonstrating
+that releasing the GIL allows conversions to run in parallel.

src/dataframe.rs

@@ -526,7 +526,12 @@ impl PyDataFrame {
         let batches = wait_for_future(py, self.df.as_ref().clone().collect())?
             .map_err(PyDataFusionError::from)?;
 
-        // Convert batches to PyArrow outside the GIL and in parallel
+        // Profiling `rb.to_pyarrow(py)` showed that the conversion holds the
+        // Python GIL for almost all of its execution. Serially converting a
+        // large number of batches therefore throttles CPU utilisation.  Run the
+        // conversions in Rayon threads and only acquire the GIL when creating
+        // the final PyArrow objects so the CPU intensive work happens in
+        // parallel.
         py.allow_threads(move || {
             batches
                 .into_par_iter()