Add lambda substrait support

[gstvg]

Which issue does this PR close?

Part of #21172

Rationale for this change

Substrait support wasn't implemented in the core lambda support to reduce PR size

What changes are included in this PR?

Substrait consuming and producing of higher-order functions, lambdas and lambda variables

Are these changes tested?

Unit tests added to datafusion/substrait/tests/cases/roundtrip_logical_plan.rs

Are there any user-facing changes?

Yes, there are breaking changes, new methods without default implementation have been added to SubstraitConsumer and SubstraitProducer

[benbellick]

👋 Hello from @substrait-io. Great to see the core lambda PR has gotten through! Once this PR is in a ready to review state and is rebased off of main, I will be more than happy to help review it 🙂

[gstvg]

Thanks @benbellick, I will open this tonight. Besides rebasing, I believe it misses some tests (I tested with sqllogictests only)

[gstvg]

@benbellick this is ready for review. Failing CI correctly detects breaking changes but apparently fails to create a comment here with the changes summary

[benbellick]

Great! I will try and find some time to take a look tomorrow :)

Thanks for working on this!

[rluvaton]

I updated your branch with main since the fix for detect breaking changes was resolved now, sorry for the trouble

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning origin/main
    Building datafusion-substrait v53.1.0 (current)
       Built [ 303.949s] (current)
     Parsing datafusion-substrait v53.1.0 (current)
      Parsed [   0.016s] (current)
    Building datafusion-substrait v53.1.0 (baseline)
       Built [ 300.456s] (baseline)
     Parsing datafusion-substrait v53.1.0 (baseline)
      Parsed [   0.015s] (baseline)
    Checking datafusion-substrait v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.219s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_added: pub trait method added ---

Description:
A non-sealed public trait added a new method without a default implementation, which breaks downstream implementations of the trait
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-new-item-no-default
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/trait_method_added.ron

Failed in:
  trait method datafusion_substrait::logical_plan::consumer::SubstraitConsumer::with_lambda_parameters in file /home/runner/work/datafusion/datafusion/datafusion/substrait/src/logical_plan/consumer/substrait_consumer.rs:535
  trait method datafusion_substrait::logical_plan::consumer::SubstraitConsumer::lambda_variable in file /home/runner/work/datafusion/datafusion/datafusion/substrait/src/logical_plan/consumer/substrait_consumer.rs:545
  trait method datafusion_substrait::logical_plan::producer::SubstraitProducer::push_lambda_parameters in file /home/runner/work/datafusion/datafusion/datafusion/substrait/src/logical_plan/producer/substrait_producer.rs:463
  trait method datafusion_substrait::logical_plan::producer::SubstraitProducer::pop_lambda_parameters in file /home/runner/work/datafusion/datafusion/datafusion/substrait/src/logical_plan/producer/substrait_producer.rs:471
  trait method datafusion_substrait::logical_plan::producer::SubstraitProducer::lambda_variable in file /home/runner/work/datafusion/datafusion/datafusion/substrait/src/logical_plan/producer/substrait_producer.rs:477
  trait method datafusion_substrait::logical_plan::producer::SubstraitProducer::lambda_parameter_type in file /home/runner/work/datafusion/datafusion/datafusion/substrait/src/logical_plan/producer/substrait_producer.rs:482

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 606.647s] datafusion-substrait