feat: improve Application Insights logging and telemetry handling

[Abdul-Microsoft]

Purpose

• This pull request introduces significant improvements to the backend's observability and telemetry by refactoring Application Insights event tracking, enhancing OpenTelemetry span attribute usage, and making logging more context-rich. The Application Insights integration is simplified to use the new azure.monitor.events.extension package, and OpenTelemetry spans now consistently include important contextual attributes such as batch_id, file_id, and user_id. Additionally, error and event logging now provides more relevant details for debugging and monitoring.

Telemetry and Observability Enhancements:

• Refactored Application Insights integration to use the azure.monitor.events.extension package and removed the legacy TelemetryClient-based implementation, simplifying event tracking logic in event_utils.py.
• Added a reusable set_span_attributes function in api_routes.py to attach contextual attributes (e.g., batch_id, file_id, user_id) to the current OpenTelemetry span for improved traceability across API endpoints.
• Updated all relevant API endpoints in api_routes.py to call set_span_attributes with appropriate identifiers, ensuring trace context is enriched for batch and file operations. [1] [2] [3] [4] [5] [6] [7] [8]

Event Tracking and Logging Improvements:

• Enhanced event tracking calls (track_event_if_configured) to consistently include relevant context (such as batch_id, file_id, and user_id) in event payloads for better monitoring and diagnostics. [1] [2] [3] [4] [5] [6] [7]
• Improved error and info logging throughout api_routes.py to include contextual identifiers, making logs more actionable and easier to correlate with specific operations or failures. [1] [2] [3] [4] [5] [6]

Azure Monitor and OpenTelemetry Configuration:

• Updated backend initialization in app.py to use configure_azure_monitor for streamlined Azure Monitor setup and removed unused OpenTelemetry exporter imports. [1] [2]
• Added suppression of noisy Azure SDK and OpenTelemetry loggers to reduce log clutter and focus on relevant application logs.
• Minor logging improvements, such as using the correct logger instance for startup messages.

Infrastructure Adjustment:

• Removed the unused diagnosticSettings property from the Application Insights Bicep module, likely as part of infrastructure cleanup or simplification.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information

[Copilot]

Pull request overview

This PR refactors backend telemetry/logging to improve observability by switching custom event tracking to the Azure Monitor Events extension, enriching API telemetry with contextual IDs, and tightening logging output by suppressing noisy SDK loggers.

Changes:

• Refactored custom event tracking to use azure.monitor.events.extension.track_event and updated tests accordingly.
• Added span attribute enrichment and expanded event/log context (e.g., batch_id, file_id, user_id) across API routes and batch processing code.
• Updated app startup telemetry configuration to use configure_azure_monitor() and added an instrumentor monkey-patch for an Azure AI telemetry serialization bug.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/backend/api/event_utils.py	Switches custom event emission to Azure Monitor Events extension.
src/backend/app.py	Moves to configure_azure_monitor() and adds SDK log suppression + instrumentor patching.
src/backend/api/api_routes.py	Adds span attribute enrichment and more contextual telemetry/events/logs.
src/backend/sql_agents/process_batch.py	Refactors logging to structured AppLogger and adds batch/file context.
src/backend/sql_agents/convert_script.py	Refactors logging to structured AppLogger and reduces raw content logging.
src/backend/common/logger/app_logger.py	Changes error() to forward exc_info (now defaulting to True).
src/backend/common/telemetry/patch_instrumentor.py	Adds monkey-patch for Azure AI instrumentor response_format serialization.
src/backend/common/telemetry/init.py	Exports patch_instrumentors only.
src/backend/common/services/batch_service.py	Adds informational logs for upload/delete/status update operations.
src/backend/requirements.txt	Replaces applicationinsights with azure-monitor-events-extension.
src/tests/backend/api/event_utils_test.py	Updates tests to mock the new track_event path.
src/tests/backend/common/storage/blob_azure_test.py	Updates expectation to include exc_info=True on error logging.
infra/main.bicep	Removes unused diagnosticSettings from App Insights module configuration.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

🎉 This PR is included in version 1.6.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀