Feat: Databento integration improvements (#10)

* Improvements from previous PR. Adhere to QC code conventions better. More robust code. Refactored and properly passing tests.

* update: .net 10

* feat: setup project configurations

* remove: demonstration file

* test:refactor: config file

* refactor: HistoryProvider, DataDownloader, SymbolMapper

* refactor: clean, style, spacing, missed license block

* feat: debug mode config

* remove: IsSupported() doesn't validate anything

* feat: map Lean.Market <-> DataBento.DataSet

* feat: Data Live TcpClient Wrapper
feat: Record Type of different DataBento scheme
feat: handle Auth-Request/Response msgs
test:feat: deserialize Heartbeat json msg, parse tcp response/request
test:feat: LiveApiClient quick connection and debug process

* feat: Data Queue Handler

* feat: reconnection process and resubscription

* refactor: deserialization and model base types

Unify market data deserialization using a new DataConverter and MarketDataBase abstract class. Remove custom JsonSettings and legacy deserialization methods. Update all model classes to inherit from MarketDataBase, make LevelOneBookLevel prices nullable, and improve error handling for unknown/invalid JSON. Move event args to .Models.Events. Expand and update tests for new d
eserialization logic and edge cases. These changes improve robustness, extensibility, and maintainability of DataBento data ingestion.

* feat: improve DataBento API error handling and test coverage

- Add ErrorResponse model for structured API error parsing
- Refactor HistoricalAPIClient to auto-correct invalid date ranges
- Expand unit tests for error scenarios and data bounds
- Add tests for error response deserialization
- Support additional OHLCV record types in DataConverter
- Use snake-case JSON settings for consistent deserialization
- Improve logging and handling of empty/malformed responses

* refactor: dataset mapping to use DataSetSpecifications

Introduce DataSetSpecifications and PredefinedDataSets for richer dataset metadata, including delay info and documentation links. Refactor DataBentoSymbolMapper to map markets to DataSetSpecifications. Update provider logic to log delay warnings and extract DataSetID as needed. Add support for new error case DataStartAfterAvailableEnd and corresponding test. Remove obsolete methods and simplify subscription/history logic.

* refactor: schema usage and add large-range tests

Replaced hardcoded schema strings with documented constants in HistoricalAPIClient for maintainability. Refactored GetRange to set record limits by schema, preventing timeouts on large requests. Added tests for 11-year minute and second data ranges. Updated all schema usages to use new constants.

* refactor: HistoryProvider to use MappedSynchronizingHistoryProvider

Refactored DataBentoProvider to inherit from MappedSynchronizingHistoryProvider, removing custom map file handling and multi-request GetHistory logic. Cleaned up unused usings and simplified history request processing by leveraging the new base class for map file resolution and request splitting.

* revert: FilterHistory()

* feat: improve error logging in HistoricalAPIClient

Refactored error handling in HistoricalAPIClient to prevent repeated log entries for the same error type by introducing boolean flags. Added a helper method to standardize unprocessable content error logs, making logging more concise and maintainable. Removed redundant log statements.

* feat: support for BBO-1s/1m Level 1 data and refactor API

- Added support for DataBento BBO-1s (1s) and BBO-1m (1m) Level 1 quote schemas alongside MBP-1 (tick)
- Refactored API and provider to fetch Level 1 data at tick, second, and minute resolutions, mapping to correct schema
- Replaced GetTickBars with GetLevelOneData throughout codebase and tests
- Enhanced DataBentoProvider to support quote ticks and intraday/interday quote bars with new helper methods
- Improved test coverage for Level 1 data at various resolutions and edge cases (null price, ulong.MaxValue ts_event)
- Updated LevelOneData and Header models: TsRecv/TsEvent now ulong, Price is nullable, UtcTime handles max value
- Added HasBidOrAskPrice extension for LevelOneBookLevel
- Improved error handling/logging and updated copyright headers

* refactor: process only top level in LevelOneData

Refactored GetIntraDayQuoteBars and GetQuoteTicks to process only the top level (best bid/ask) from LevelOneData, instead of iterating through all levels. This simplifies QuoteBar and Tick construction, yields only top-of-book data, and improves performance for Level 1 data scenarios.

* refactor: logging for clarity and consistency

* feat: improve timestamp handling in DataBento integration

Centralize timestamp conversion logic in MarketDataBase with TryGetDateTimeUtc, prioritizing Header.TsEvent and falling back to TsRecv. Remove Header.UtcTime and update all consumers to use the new method, skipping or logging on invalid timestamps. Add exchange time zone conversion and caching in the history provider. Refactor quote bar aggregation and enhance error logging. Update tests for new timestamp logic.

* refactor: timestamp handling with UtcDateTime properties

Replaces TryGetDateTimeUtc with explicit UtcDateTime properties on Header and LevelOneData. Updates all consumers to use these properties directly, with null checks for undefined timestamps. Removes MarketHoursDatabase/time zone cache and inlines exchange time conversion.

* feat: productId in ValidateSubscription to 427

* refactor: LiveDataTcpClientWrapper connection handling

- Replace _isConnected with ManualResetEventSlim for connection state signaling
- Start data receiver task in constructor; separate connection monitoring and data receiving
- Improve authentication error handling and throw AuthenticationException with descriptive message
- Add CleanupConnection() for centralized resource cleanup
- Enhance Dispose() to properly cancel, reset, and dispose resources
- Raise ConnectionLost events when connection is lost
- Improve exception handling and resource management throughout
- Add new test case for "open connection limit" authentication error
- Minor code style and documentation improvements

* remove: redundant semicolon

* refactor: LiveDataTcpClientWrapper to be fully synchronous

Remove async/await from data receiving and authentication logic.
Replace async methods with blocking StreamReader.ReadLine().
Set NetworkStream.ReadTimeout to detect stalled connections.
Improve exception handling and connection reset logic.
Simplifies code and makes network operations timeout-aware.

* refactor: AuthenticationMessageRequest constructor, add docs

Refactored AuthenticationMessageRequest to require a non-null heartbeat interval, removing default and minimum value logic. Updated related test to pass explicit interval. Added comprehensive XML documentation to the struct, its fields, and methods for improved clarity and maintainability.

* refactor: Level 1 data: split tick vs interval, add types

Introduce LevelOneDataBase to unify Level 1 data structure.
Split tick (MBP1) and interval (BBO1s/BBO1m) data into
LevelOneData and BestBidOfferInterval, respectively. Add
ActionType enum for type-safe action handling. Update
HistoricalAPIClient, DataConverter, provider, and tests to
use new types and methods. Improves type safety, clarity,
and test coverage for DataBento Level 1 data.

* refactor: LevelOneData to use enums and single book level

- Replace char/int with strongly-typed Side and Flags enums
- Change LevelOneDataBase to use LevelOne (single) instead of Levels (list)
- Update deserialization to enforce single level and use new property
- Refactor providers, history, and tests to use LevelOne and enums
- Add shared assertion helpers and improve documentation and licensing

* refactor: LevelOneData deserialization in DataConverter

* remove: Depth property from LevelOneData and related tests

* refactor: handle Quote/Trade updates properlly
feat: support for DataBento live error messages

Introduced ErrorMessage class and updated deserialization logic to handle error records from the DataBento live feed. LiveAPIClient now logs and terminates on error messages. Updated ActionType handling in DataBentoProvider and LiveAPIClient, and removed unused Fill value from ActionType enum. Added unit test for error message deserialization.

* feat: throw LiveApiErrorException on Live API error messages
feat: improve Live API error handling and test coverage

- Add LiveApiErrorException class to wrap ErrorMessage
- Update LiveDataTcpClientWrapper to propagate API errors
- Refactor DataBentoJsonConverterTests to use parameterized tests for error messages

* feat: improve thread safety
remove: unused symbol property
feat: skip several system messages
- Remove StypeOutSymbol from SymbolMappingMessage and related test
- Add locking to _subscribedSymbolsByDataBentoInstrumentId for thread safety
- Refactor OnSymbolMappingConfirmation and HandleLevelOneData to use locking and improve logic
- Use TryAdd for _pendingSubscriptions to prevent duplicates in Subscribe
- Lock dictionary access in Unsubscribe to ensure safe removals

* feat: make symbol mapper static, restrict subscribe methods

Changed _symbolMapper to static for shared access. Made Subscribe and Unsubscribe methods private. Added market mapping check and error logging in CanSubscribe to prevent unsupported subscriptions.

* feat: API key validation and auth error handling

- Added AuthAuthenticationFailed error case for authentication failures.
- Updated JSON converter tests to cover authentication errors.
- Implemented IsValidApiKey() in HistoricalAPIClient to verify API key validity.
- DataBentoProvider now validates API key on initialization and throws AuthenticationException if invalid.
- Adjusted initialization order to ensure early API key validation.

* update: new markets in databento.json

* feat: increase delay of reconnection amount

* feat: improve error handling in LiveDataTcpClientWrapper

Enhances robustness of connection monitoring by handling exceptions during ConnectionLost event invocation, logging errors, and reporting connection failures via IResultHandler. Prevents unhandled exceptions from disrupting the monitoring loop.

* feat: improve error handling for DataBento Live API connections

Enhanced logging and error reporting for connection failures.
Added runtime error notification to result handler in LiveAPIClient.
Introduced MaxConnectionAttempts constant for retries.
Removed redundant runtime error reporting in LiveDataTcpClientWrapper.
Clarified error logs for connection monitoring and event handling.

* feat: improve error handling for DataBento live data failures

Refactored connection failure handling in LiveAPIClient and LiveDataTcpClientWrapper. Now, exceptions are thrown directly on connection failure, and a new method notifies the result handler to terminate the algorithm on critical errors. Enhanced error messages and logging for better diagnostics. Addresses Lean issue #9272.

---------

Co-authored-by: Joseph Scorsone <scorsonejoseph4@gmail.com>

Author: Romazes

Demonstration.cs

@@ -2,9 +2,9 @@ Microsoft Visual Studio Solution File, Format Version 12.00
 # Visual Studio Version 17
 VisualStudioVersion = 17.5.2.0
 MinimumVisualStudioVersion = 10.0.40219.1
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "QuantConnect.DataBento", "QuantConnect.DataBento\QuantConnect.DataSource.DataBento.csproj", "{367AEEDC-F0B3-7F47-539D-10E5EC242C2A}"
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "QuantConnect.DataSource.DataBento", "QuantConnect.DataBento\QuantConnect.DataSource.DataBento.csproj", "{367AEEDC-F0B3-7F47-539D-10E5EC242C2A}"
 EndProject
-Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "QuantConnect.DataBento.Tests", "QuantConnect.DataBento.Tests\QuantConnect.DataSource.DataBento.Tests.csproj", "{9CF47860-2CEA-F379-09D8-9AEF27965D12}"
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "QuantConnect.DataSource.DataBento.Tests", "QuantConnect.DataBento.Tests\QuantConnect.DataSource.DataBento.Tests.csproj", "{9CF47860-2CEA-F379-09D8-9AEF27965D12}"
 EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
@@ -16,10 +16,6 @@ Global
 		{367AEEDC-F0B3-7F47-539D-10E5EC242C2A}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{367AEEDC-F0B3-7F47-539D-10E5EC242C2A}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{367AEEDC-F0B3-7F47-539D-10E5EC242C2A}.Release|Any CPU.Build.0 = Release|Any CPU
-		{4B379C8F-16CE-1972-73E3-C14F6410D428}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
-		{4B379C8F-16CE-1972-73E3-C14F6410D428}.Debug|Any CPU.Build.0 = Debug|Any CPU
-		{4B379C8F-16CE-1972-73E3-C14F6410D428}.Release|Any CPU.ActiveCfg = Release|Any CPU
-		{4B379C8F-16CE-1972-73E3-C14F6410D428}.Release|Any CPU.Build.0 = Release|Any CPU
 		{9CF47860-2CEA-F379-09D8-9AEF27965D12}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{9CF47860-2CEA-F379-09D8-9AEF27965D12}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{9CF47860-2CEA-F379-09D8-9AEF27965D12}.Release|Any CPU.ActiveCfg = Release|Any CPU

Lean.DataSource.DataBento.sln

@@ -0,0 +1,53 @@
+/*
+ * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+ * Lean Algorithmic Trading Engine v2.0. Copyright 2026 QuantConnect Corporation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+using System;
+using QuantConnect.Lean.DataSource.DataBento.Models;
+
+namespace QuantConnect.Lean.DataSource.DataBento.Tests;
+
+public static class Assertions
+{
+    public static void AssertLevelOneBookLevel(LevelOneBookLevel level)
+    {
+        Assert.IsNotNull(level);
+        AssertPositiveOrNull(level.BidPx);
+        AssertPositiveOrNull(level.AskPx);
+        Assert.GreaterOrEqual(level.BidSz, 0);
+        Assert.GreaterOrEqual(level.AskSz, 0);
+        Assert.GreaterOrEqual(level.BidCt, 0);
+        Assert.GreaterOrEqual(level.AskCt, 0);
+    }
+
+    public static void AssertPositiveOrNull(decimal? price)
+    {
+        if (price.HasValue)
+        {
+            Assert.Greater(price.Value, 0);
+        }
+        else
+        {
+            Assert.IsNull(price);
+        }
+    }
+
+    public static void AssertEnumIsDefined<TEnum>(TEnum value, string paramName) where TEnum : struct, Enum
+    {
+        if (!Enum.IsDefined(value))
+        {
+            Assert.Fail($"{paramName} must be one of {string.Join(", ", Enum.GetValues<TEnum>())}");
+        }
+    }
+}