Skip to content

table_provider.rst

Latest commit

 

History

History
62 lines (45 loc) · 2.25 KB

table_provider.rst

File metadata and controls

62 lines (45 loc) · 2.25 KB

Custom Table Provider

If you have a custom data source that you want to integrate with DataFusion, you can do so by implementing the TableProvider interface in Rust and then exposing it in Python. To do so, you must use DataFusion 43.0.0 or later and expose a FFI_TableProvider via PyCapsule.

A complete example can be found in the examples folder.

#[pymethods]
impl MyTableProvider {

    fn __datafusion_table_provider__<'py>(
        &self,
        py: Python<'py>,
    ) -> PyResult<Bound<'py, PyCapsule>> {
        let name = cr"datafusion_table_provider".into();

        let provider = Arc::new(self.clone());
        let provider = FFI_TableProvider::new(provider, false, None);

        PyCapsule::new_bound(py, provider, Some(name.clone()))
    }
}

Once you have this library available, you can construct a :py:class:`~datafusion.Table` in Python and register it with the SessionContext.

from datafusion import SessionContext, Table

ctx = SessionContext()
provider = MyTableProvider()

ctx.register_table("capsule_table", provider)

ctx.table("capsule_table").show()