Enhance documentation for target partitions in SessionConfig to clarify parallelism control and provide usage example.

Author: kosiew

docs/source/user-guide/configuration.rst

@@ -47,5 +47,26 @@ a :py:class:`~datafusion.context.SessionConfig` and :py:class:`~datafusion.conte
     print(ctx)
 
 
+.. _target_partitions:
+
+Target partitions and threads
+-----------------------------
+
+The :py:meth:`~datafusion.context.SessionConfig.with_target_partitions` method
+controls how many partitions DataFusion uses when executing a query. Each
+partition is processed on its own thread, so this setting effectively limits
+the number of threads that will be scheduled.
+
+For most workloads a good starting value is the number of logical CPU cores on
+your machine. You can use :func:`os.cpu_count` to automatically configure this::
+
+    import os
+    config = SessionConfig().with_target_partitions(os.cpu_count())
+
+Choosing a value significantly higher than the available cores can lead to
+excessive context switching without performance gains, while a much lower value
+may underutilize the machine.
+
+
 You can read more about available :py:class:`~datafusion.context.SessionConfig` options in the `rust DataFusion Configuration guide <https://arrow.apache.org/datafusion/user-guide/configs.html>`_,
 and about :code:`RuntimeEnvBuilder` options in the rust `online API documentation <https://docs.rs/datafusion/latest/datafusion/execution/runtime_env/struct.RuntimeEnvBuilder.html>`_.

python/datafusion/context.py

@@ -161,7 +161,13 @@ def with_batch_size(self, batch_size: int) -> SessionConfig:
     def with_target_partitions(self, target_partitions: int) -> SessionConfig:
         """Customize the number of target partitions for query execution.
 
-        Increasing partitions can increase concurrency.
+        Each partition is processed on its own thread, so this value controls
+        the degree of parallelism. A good starting point is the number of
+        logical CPU cores on your machine, for example
+        ``SessionConfig().with_target_partitions(os.cpu_count())``.
+
+        See the :ref:`configuration guide <target_partitions>` for more
+        discussion on choosing a value.
 
         Args:
             target_partitions: Number of target partitions.