Merge pull request #5 from fmguerreiro/chore/merge-upstream-main

chore: merge upstream apache/datafusion-sqlparser-rs main

Author: fmguerreiro

AGENTS.md

@@ -0,0 +1,41 @@
+# Extensible SQL Lexer and Parser for Rust Agents Guidelines
+
+## General Agent Workflow
+1. You will write unit tests to ensure your code change is working as expected.
+2. You will run the commands in the Pre Commit Checks section below to ensure your change is ready for a pull request.
+3. When instructed to open a PR, you will follow the instructions in the Pull Request Guidelines section below.
+
+## General Coding Guidelines
+1. Refrain from adding conditions on specific dialects, such as `dialect_is!(...)` or `dialect_of!(... | ...)`. Instead, define a new function in the `Dialect` trait that describes the condition, so that dialects can turn this condition on more easily.
+2. Make targeted code changes and refrain from refactoring, unless it's absolutely required.
+
+## Unit Tests Guidelines
+- New unit tests should be added to the `tests` module in the corresponding dialect file (e.g., `tests/sqlparser_redshift.rs` for Redshift), and should be placed at the end of the file.
+- If the new functionality is gated using a dialect function, and the SQL is likely relevant in most dialects, tests should be placed under `tests/sqlparser_common.rs`.
+- When testing a multi-line SQL statement, use a raw string literal, i.e. `r#"..."#` to preserve formatting.
+- The parser builds an abstract syntax tree (AST) from the SQL statement and has functionality to display the tree as SQL. Use the following template for simple unit tests where you expect the SQL created from the AST to be the same as the input SQL:
+```rust
+<dialect>().verified_stmt(r#"..."#);
+```
+For example: `snowflake().verified_stmt(r#"SELECT * FROM my_table"#)`. Use `one_statement_parses_to` instead of `verified_stmt` when you expect the SQL created by the AST to differ than the input SQL. For example:
+```rust
+snowflake().one_statement_parses_to(
+    "SELECT * FROM my_table t",
+    "SELECT * FROM my_table AS t",
+)
+```
+
+## Analyzing Parsing Issues
+You can try to simplify the SQL statement to identify the root cause of the parsing issue. This may involve removing certain clauses or components of the SQL statement to see if it can be parsed successfully. Additionally, you can compare the problematic SQL statement with similar statements that are parsed correctly to identify any differences that may be causing the issue.
+
+## Pre Commit Checks
+Run the following commands before you commit to ensure the change will pass the CI process:
+```bash
+cargo test --all-features
+cargo fmt --all
+cargo clippy --all-targets --all-features -- -D warnings
+```
+
+## Pull Request Guidelines
+1. PR title should follow this format: `<DIALECT>: <SHORT DESCRIPTION>`, For example, `Showflake: Add support for casting to VARIANT`.
+2. Make the PR comment short, provide an example of what was not working and a short description of the fix. Be succint.

Cargo.toml

@@ -18,7 +18,7 @@
 [package]
 name = "pgmold-sqlparser"
 description = "Fork of sqlparser with additional PostgreSQL features (PARTITION OF, SECURITY DEFINER/INVOKER, SET params)"
-version = "0.60.6"
+version = "0.60.7"
 authors = ["Filipe Guerreiro <filipe.m.guerreiro@gmail.com>"]
 homepage = "https://github.com/fmguerreiro/datafusion-sqlparser-rs"
 documentation = "https://docs.rs/pgmold-sqlparser/"
@@ -42,6 +42,7 @@ std = []
 recursive-protection = ["std", "recursive"]
 # Enable JSON output in the `cli` example:
 json_example = ["serde_json", "serde"]
+derive-dialect = ["sqlparser_derive"]
 visitor = ["sqlparser_derive"]
 
 [dependencies]
@@ -54,13 +55,17 @@ serde = { version = "1.0", default-features = false, features = ["derive", "allo
 # of dev-dependencies because of
 # https://github.com/rust-lang/cargo/issues/1596
 serde_json = { version = "1.0", optional = true }
-sqlparser_derive = { version = "0.4.0", path = "derive", optional = true }
+sqlparser_derive = { version = "0.5.0", path = "derive", optional = true }
 
 [dev-dependencies]
 simple_logger = "5.0"
 matches = "0.1"
 pretty_assertions = "1"
 
+[[test]]
+name = "sqlparser_derive_dialect"
+required-features = ["derive-dialect"]
+
 [package.metadata.docs.rs]
 # Document these features on docs.rs
-features = ["serde", "visitor"]
+features = ["serde", "visitor", "derive-dialect"]

README.md

@@ -159,9 +159,9 @@ $ cargo run --features json_example --example cli FILENAME.sql [--dialectname]
 
 ## Users
 
-This parser is currently being used by the [DataFusion] query engine, [LocustDB],
-[Ballista], [GlueSQL], [Opteryx], [Polars], [PRQL], [Qrlew], [JumpWire], [ParadeDB], [CipherStash Proxy],
-and [GreptimeDB].
+This parser is currently being used by the [DataFusion] query engine,
+[LocustDB], [Ballista], [GlueSQL], [Opteryx], [Polars], [PRQL], [Qrlew],
+[JumpWire], [ParadeDB], [CipherStash Proxy], [Readyset] and [GreptimeDB].
 
 If your project is using sqlparser-rs feel free to make a PR to add it
 to this list.
@@ -282,3 +282,4 @@ licensed as above, without any additional terms or conditions.
 [`GenericDialect`]: https://docs.rs/sqlparser/latest/sqlparser/dialect/struct.GenericDialect.html
 [CipherStash Proxy]: https://github.com/cipherstash/proxy
 [GreptimeDB]: https://github.com/GreptimeTeam/greptimedb
+[Readyset]: https://github.com/readysettech/readyset

changelog/0.61.0.md

@@ -0,0 +1,127 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# sqlparser-rs 0.61.0 Changelog
+
+This release consists of 66 commits from 22 contributors. See credits at the end of this changelog for more information.
+
+**Performance related:**
+
+- perf: remove unnecessary string clone in maybe_concat_string_literal [#2173](https://github.com/apache/datafusion-sqlparser-rs/pull/2173) (andygrove)
+- perf: optimize `make_word()` to avoid unnecessary allocations [#2176](https://github.com/apache/datafusion-sqlparser-rs/pull/2176) (andygrove)
+
+**Fixed bugs:**
+
+- fix: Set the current folder as a "primary" for the `find` command [#2120](https://github.com/apache/datafusion-sqlparser-rs/pull/2120) (martin-g)
+- fix: qualified column names with SQL keywords parse as identifiers [#2157](https://github.com/apache/datafusion-sqlparser-rs/pull/2157) (bombsimon)
+
+**Other:**
+
+- Fixing location for extrenal tables [#2108](https://github.com/apache/datafusion-sqlparser-rs/pull/2108) (romanoff)
+- Added support for `ALTER OPERATOR` syntax [#2114](https://github.com/apache/datafusion-sqlparser-rs/pull/2114) (LucaCappelletti94)
+- Oracle: Support for MERGE predicates [#2101](https://github.com/apache/datafusion-sqlparser-rs/pull/2101) (xitep)
+- [Oracle] Lower StringConcat precedence [#2115](https://github.com/apache/datafusion-sqlparser-rs/pull/2115) (xitep)
+- Added alter external table support for snowflake [#2122](https://github.com/apache/datafusion-sqlparser-rs/pull/2122) (romanoff)
+- MSSQL: Add support for parenthesized stored procedure name in EXEC [#2126](https://github.com/apache/datafusion-sqlparser-rs/pull/2126) (yoavcloud)
+- MSSQL: Parse IF/ELSE without semicolon delimiters [#2128](https://github.com/apache/datafusion-sqlparser-rs/pull/2128) (yoavcloud)
+- Extract source comments [#2107](https://github.com/apache/datafusion-sqlparser-rs/pull/2107) (xitep)
+- PostgreSQL: Support schema-qualified operator classes in CREATE INDEX [#2131](https://github.com/apache/datafusion-sqlparser-rs/pull/2131) (dddenis)
+- Oracle: Support for quote delimited strings [#2130](https://github.com/apache/datafusion-sqlparser-rs/pull/2130) (xitep)
+- Added support for `ALTER OPERATOR FAMILY` syntax [#2125](https://github.com/apache/datafusion-sqlparser-rs/pull/2125) (LucaCappelletti94)
+- PostgreSQL Tokenization: Fix unexpected characters after question mark being silently ignored [#2129](https://github.com/apache/datafusion-sqlparser-rs/pull/2129) (jnlt3)
+- Support parsing parenthesized wildcard `(*)` [#2123](https://github.com/apache/datafusion-sqlparser-rs/pull/2123) (romanoff)
+- Make benchmark statement valid [#2139](https://github.com/apache/datafusion-sqlparser-rs/pull/2139) (xitep)
+- Fix parse_identifiers not taking semicolons into account [#2137](https://github.com/apache/datafusion-sqlparser-rs/pull/2137) (jnlt3)
+- Add PostgreSQL PARTITION OF syntax support [#2127](https://github.com/apache/datafusion-sqlparser-rs/pull/2127) (fmguerreiro)
+- Databricks: Support Timetravel With "TIMESTAMP AS OF" [#2134](https://github.com/apache/datafusion-sqlparser-rs/pull/2134) (JamesVorder)
+- MySQL: Parse bitwise shift left/right operators [#2152](https://github.com/apache/datafusion-sqlparser-rs/pull/2152) (mvzink)
+- Redshift: Add support for optional JSON format in copy option [#2141](https://github.com/apache/datafusion-sqlparser-rs/pull/2141) (yoavcloud)
+- MySQL: Add missing support for TREE explain format [#2145](https://github.com/apache/datafusion-sqlparser-rs/pull/2145) (yoavcloud)
+- MySQL: Add support for && as boolean AND [#2144](https://github.com/apache/datafusion-sqlparser-rs/pull/2144) (yoavcloud)
+- PostgreSQL: ALTER USER password option [#2142](https://github.com/apache/datafusion-sqlparser-rs/pull/2142) (yoavcloud)
+- Key Value Options: add support for trailing semicolon [#2140](https://github.com/apache/datafusion-sqlparser-rs/pull/2140) (yoavcloud)
+- Added support for `ALTER OPERATOR CLASS` syntax [#2135](https://github.com/apache/datafusion-sqlparser-rs/pull/2135) (LucaCappelletti94)
+- Added missing `Copy` derives [#2158](https://github.com/apache/datafusion-sqlparser-rs/pull/2158) (LucaCappelletti94)
+- Tokenize empty line comments correctly [#2161](https://github.com/apache/datafusion-sqlparser-rs/pull/2161) (zyuiop)
+- Add support for DuckDB `LAMBDA` keyword syntax [#2149](https://github.com/apache/datafusion-sqlparser-rs/pull/2149) (lovasoa)
+- MySQL: Add support for casting using the BINARY keyword [#2146](https://github.com/apache/datafusion-sqlparser-rs/pull/2146) (yoavcloud)
+- Added missing `From` impls for `Statement` variants [#2160](https://github.com/apache/datafusion-sqlparser-rs/pull/2160) (LucaCappelletti94)
+- GenericDialect: support colon operator for JsonAccess [#2124](https://github.com/apache/datafusion-sqlparser-rs/pull/2124) (Samyak2)
+- Databricks: Support Timetravel With "VERSION AS OF" [#2155](https://github.com/apache/datafusion-sqlparser-rs/pull/2155) (JamesVorder)
+- Fixed truncate table if exists for snowflake [#2166](https://github.com/apache/datafusion-sqlparser-rs/pull/2166) (romanoff)
+- Refactor: replace some `dialect_of!` checks with `Dialect` trait methods [#2171](https://github.com/apache/datafusion-sqlparser-rs/pull/2171) (andygrove)
+- MySQL: Support `CAST(... AS ... ARRAY)` syntax [#2151](https://github.com/apache/datafusion-sqlparser-rs/pull/2151) (mvzink)
+- Snowflake: Support SAMPLE clause on subqueries [#2164](https://github.com/apache/datafusion-sqlparser-rs/pull/2164) (finchxxia)
+- refactor: use `to_ident()` instead of `clone().into_ident()` for borrowed Words [#2177](https://github.com/apache/datafusion-sqlparser-rs/pull/2177) (andygrove)
+- Refactor: replace more `dialect_of!` checks with `Dialect` trait methods [#2175](https://github.com/apache/datafusion-sqlparser-rs/pull/2175) (andygrove)
+- minor: reduce unnecessary string allocations [#2178](https://github.com/apache/datafusion-sqlparser-rs/pull/2178) (andygrove)
+- PostgreSQL: Support force row level security [#2169](https://github.com/apache/datafusion-sqlparser-rs/pull/2169) (isaacparker0)
+- PostgreSQL: Add support for `*` (descendant) option in TRUNCATE [#2181](https://github.com/apache/datafusion-sqlparser-rs/pull/2181) (mvzink)
+- Fix identifier parsing not breaking on the `|>` pipe operator [#2156](https://github.com/apache/datafusion-sqlparser-rs/pull/2156) (alexander-beedie)
+- [MySQL, Oracle] Parse optimizer hints [#2162](https://github.com/apache/datafusion-sqlparser-rs/pull/2162) (xitep)
+- Redshift: Support implicit string concatenation using newline [#2167](https://github.com/apache/datafusion-sqlparser-rs/pull/2167) (yoavcloud)
+- PostgreSQL: Fix REPLICA IDENTITY to use NOTHING [#2179](https://github.com/apache/datafusion-sqlparser-rs/pull/2179) (mvzink)
+- Add ENFORCED/NOT ENFORCED support for column-level CHECK constraints [#2180](https://github.com/apache/datafusion-sqlparser-rs/pull/2180) (mvzink)
+- Implement `core::error::Error` for `ParserError` and `TokenizerError` [#2189](https://github.com/apache/datafusion-sqlparser-rs/pull/2189) (LucaCappelletti94)
+- Moved more structs outside of Statement to facilitate reuse [#2188](https://github.com/apache/datafusion-sqlparser-rs/pull/2188) (LucaCappelletti94)
+- Fix parsing cast operator after parenthesized `DEFAULT` expression [#2168](https://github.com/apache/datafusion-sqlparser-rs/pull/2168) (isaacparker0)
+- Streamlined derivation of new `Dialect` objects [#2174](https://github.com/apache/datafusion-sqlparser-rs/pull/2174) (alexander-beedie)
+- MSSQL: Support standalone BEGIN...END blocks [#2186](https://github.com/apache/datafusion-sqlparser-rs/pull/2186) (guan404ming)
+- MySQL: Add support for `SELECT` modifiers [#2172](https://github.com/apache/datafusion-sqlparser-rs/pull/2172) (mvzink)
+- MySQL: Add support for DEFAULT CHARACTER SET in CREATE DATABASE [#2182](https://github.com/apache/datafusion-sqlparser-rs/pull/2182) (mvzink)
+- [Oracle] Support hierarchical queries [#2185](https://github.com/apache/datafusion-sqlparser-rs/pull/2185) (xitep)
+- MySQL: Allow optional constraint name after CONSTRAINT keyword [#2183](https://github.com/apache/datafusion-sqlparser-rs/pull/2183) (mvzink)
+- Added missing derives to dialect marker structs [#2191](https://github.com/apache/datafusion-sqlparser-rs/pull/2191) (LucaCappelletti94)
+- Fixed overflow error, recursion counter was not included for parenthesis [#2199](https://github.com/apache/datafusion-sqlparser-rs/pull/2199) (LucaCappelletti94)
+- Add support for C-style comments [#2034](https://github.com/apache/datafusion-sqlparser-rs/pull/2034) (altmannmarcelo)
+- PostgreSQL: Support PostgreSQL ANALYZE with optional table and column [#2187](https://github.com/apache/datafusion-sqlparser-rs/pull/2187) (guan404ming)
+- Add Tokenizer custom token mapper support [#2184](https://github.com/apache/datafusion-sqlparser-rs/pull/2184) (askalt)
+- Fix MAP literals parsing [#2205](https://github.com/apache/datafusion-sqlparser-rs/pull/2205) (Samyak2)
+
+## Credits
+
+Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.
+
+```
+     9	Luca Cappelletti
+     9	Yoav Cohen
+     8	Michael Victor Zink
+     7	xitep
+     6	Andy Grove
+     4	Andriy Romanov
+     2	Alexander Beedie
+     2	Andrew Lamb
+     2	Guan-Ming (Wesley) Chiu
+     2	James Vorderbruggen
+     2	Samyak Sarnayak
+     2	isaacparker0
+     2	jnlt3
+     1	Albert Skalt
+     1	Denis Goncharenko
+     1	Filipe Guerreiro
+     1	Louis Vialar
+     1	Marcelo Altmann
+     1	Martin Grigorov
+     1	Ophir LOJKINE
+     1	Simon Sawert
+     1	finchxxia
+```
+
+Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.
+

derive/Cargo.toml

@@ -18,7 +18,7 @@
 [package]
 name = "sqlparser_derive"
 description = "Procedural (proc) macros for sqlparser"
-version = "0.4.0"
+version = "0.5.0"
 authors = ["sqlparser-rs authors"]
 homepage = "https://github.com/sqlparser-rs/sqlparser-rs"
 documentation = "https://docs.rs/sqlparser_derive/"
@@ -36,6 +36,6 @@ edition = "2021"
 proc-macro = true
 
 [dependencies]
-syn = { version = "2.0", default-features = false, features = ["printing", "parsing", "derive", "proc-macro"] }
+syn = { version = "2.0", default-features = false, features = ["full", "printing", "parsing", "derive", "proc-macro", "clone-impls"] }
 proc-macro2 = "1.0"
 quote = "1.0"