feat: enhance Copilot instructions with detailed project architecture and development workflows

Author: kosiew

.github/copilot-instructions.md

@@ -1,4 +1,18 @@
-# Copilot Instructions
+# Copilot Instructions for datafusion-python
+
+## Project Architecture
+
+- **Core Engine:** Rust (`src/`) implements the main logic using Apache Arrow DataFusion.
+- **Python Bindings:** Exposed via PyO3 (`src/`, `python/datafusion/`), with high-level Python APIs wrapping Rust internals.
+- **Data Flow:** Python API → PyO3 FFI → DataFusion Core (Rust) → PyArrow for data exchange.
+- **Key Entry Points:** `SessionContext` (`context.py`), `DataFrame` (`dataframe.py`), UDF/UDAF/UDWF registration.
+
+## Development Workflows
+
+- **Build & Install:** Use `maturin develop` (optionally with `--uv`) to build Rust and install the Python extension.
+- **Testing:** Run `pytest` in the root or `python/tests/` for Python tests. Rust code is tested via `cargo test`.
+- **Linting:** Use `ruff` for Python (`./ci/scripts/python_lint.sh`), `rustfmt` and `clippy` for Rust (`./ci/scripts/rust_fmt.sh`, `./ci/scripts/rust_clippy.sh`).
+- **Pre-commit:** Install with `pre-commit install` and run checks before pushing.
 
 ## Python Code Style
 
@@ -38,6 +52,26 @@ When generating Python code for the DataFusion Python project, adhere to the fol
    - Group related tests in classes
    - Use pytest style assertions instead of unittest style
 
+## Patterns & Examples
+
+- **SessionContext**: Central object for query execution. See `examples/create-context.py`.
+- **DataFrame**: Immutable, chainable API for query building. See `examples/dataframe-parquet.py`.
+- **UDF/UDAF/UDWF**: Register Python functions for use in SQL/DataFrame queries. See `examples/python-udf.py`, `examples/python-udaf.py`.
+- **PyArrow**: Used for all data interchange; prefer Arrow types in UDFs and DataFrame construction.
+
+## External Dependencies
+
+- **Build:** maturin, PyO3, uv/pip, pytest, ruff, rustfmt, clippy
+- **Data:** Example datasets in `docs/` and referenced in `examples/`
+
+## Directory Structure
+
+- `src/`: Rust core and FFI
+- `python/datafusion/`: Python API
+- `python/tests/`: Python tests
+- `examples/`: Usage examples
+- `ci/scripts/`: Lint/build scripts
+
 ## Rust Code Style
 
 When generating Rust code for DataFusion, follow these guidelines:
@@ -47,7 +81,9 @@ When generating Rust code for DataFusion, follow these guidelines:
    - Correct: `if some_condition`
    - Incorrect: `if (some_condition)`
 
-2. Follow the standard Rust style conventions from rustfmt
+2. Avoid `expect()` or `panic!()` except in tests - use proper error handling with `Result` types
+
+3. Follow the standard Rust style conventions from rustfmt
 
 ### Code Organization