Skip to content

troubleshoot-telemetry-endpoint.md

Latest commit

 

History

History
219 lines (138 loc) · 11.7 KB

troubleshoot-telemetry-endpoint.md

File metadata and controls

219 lines (138 loc) · 11.7 KB
title Troubleshoot Connectivity to Data Processing Service and Telemetry Endpoints
description Describes how to troubleshoot connectivity to the data processing service (DPS) and telemetry endpoints.
author twright-msft
ms.author twright
ms.reviewer mikeray
ms.date 06/30/2025
ms.topic troubleshooting

Troubleshoot connectivity to the data processing service and telemetry endpoints

[!INCLUDE sqlserver]

In addition to the usual endpoints, the Azure Arc extension for SQL Server connects to two other endpoints:

  • Data processing service (DPS) endpoint

    The collected inventory information about SQL Server instances, databases, availability groups, and usage data for billing purposes is sent to this endpoint.

  • Telemetry endpoint

    The Azure Connected Machine agent logs, the Azure extension for SQL Server logs, and the Dynamic Management Views (DMV) data is sent to this endpoint.

Communication to these endpoints uses HTTPS with SSL/TLS and port TCP/443 for encrypted secure connections. The agent initiates communication to send the data to Azure. Azure never initiates communication. Connectivity to these endpoints is therefore only one way.

When communication to these endpoints is blocked, the service has the following symptoms:

Azure extension current state

You can view the current state of the Azure extension for SQL Server in the portal. The status is refreshed every 15 minutes.

:::row::: :::column:::

Healthy state:

:::image type="content" source="media/troubleshoot-telemetry-endpoint/healthy-state.png" alt-text="Screenshot of portal for Azure extension for SQL Server in a healthy state.":::

:::column-end:::
:::column:::

Unhealthy state:

:::image type="content" source="media/troubleshoot-telemetry-endpoint/unhealthy-state.png" alt-text="Screenshot of portal for Azure extension for SQL Server in an unhealthy state." lightbox="media/troubleshoot-telemetry-endpoint/unhealthy-state.png":::

:::column-end:::

:::row-end:::

Check if you have a problem connecting to the DPS or telemetry endpoints

There are two ways to check if you have connectivity problems to the DPS or telemetry endpoints.

Check the Azure Extension for SQL Server status in the Azure portal

If it's connected to Azure in general, the Azure Extension for SQL Server reports its status in the Azure portal.

  • Navigate to the Machines - Azure Arc view in the Azure portal and locate the machine by name and select it.
  • Select Extensions.
  • Select WindowsAgent.SqlServer or LinuxAgent.SqlServer to bring up the details.
  • Look at the Status message and the uploadStatus value. If it's anything other than OK, there's a problem with connecting to the DPS. If it's 0, it's likely that there's a firewall blocking the communication to the DPS endpoint. There could be more details in the status message or the uploadStatus error code that can provide insights into the connectivity problem.

Check the Azure Extension for SQL Server logs

The extension log file is at:

C:\ProgramData\GuestConfig\extension_logs\Microsoft.AzureData.WindowsAgent.SqlServer\

The log file name depends on the version Azure Extension for SQL Server. For the latest version of Azure Extension for SQL Server, the log file is:

unifiedagent.log

For extension version 1.1.24724.69 and earlier, the log file is:

ExtensionLog_0.log

Check for log entries that indicate a problem connecting to the DPS or telemetry endpoints.

Probe web server endpoints

You can use various tools to probe the web server endpoints for DPS and telemetry. For example, Invoke-WebRequest or curl.

The following example uses Invoke-Webrequest:

Invoke-WebRequest telemetry.<region>.arcdataservices.com

A possible response status code is:

Invoke-WebRequest: Response status code does not indicate success: 401 (Unauthorized).

401 is expected because there's no unauthenticated route on the telemetry endpoint.

For DPS:

Invoke-WebRequest dataprocessingservice.<region>.arcdataservices.com

For US Government Virginia, replace arcdataservices.com with arcdataservices.azure.us. This example would be:

Invoke-WebRequest dataprocessingservice.<region>.arcdataservices.azure.us

A possible response status code is:

StatusCode        : 200

StatusDescription : OK

200 is expected as there's an unauthenticated route.

Probe connectivity to all regions

You can probe connectivity to all regions with the test-connectivity.ps1 PowerShell script.

Note

For the US Government Virginia region, replace arcdataservices.com with arcdataservices.azure.us.

:::code language="powershell" source="~/../sql-server-samples/samples/features/azure-arc/troubleshooting/test-connectivity.ps1":::

Check TLS version compatibility

The data processing service endpoint supports the following TLS versions: TLS 1.2 and 1.3. Windows Server 2012 and older versions aren't supported.

For telemetry endpoints, Windows Server 2012 R2 and older aren't supported.

If an unsupported TLS version is being used, you might see an error in the log

<date time>|ERROR|SqlServerExtension.Service|Request failed with exception 'System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.

---> System.Security.Authentication.AuthenticationException: Authentication failed because the remote party sent a TLS alert: 'HandshakeFailure'.

---> System.ComponentModel.Win32Exception (0x80090326): The message received was unexpected or badly formatted.

Endpoint reference

To connect to Azure, the endpoints use *.arcdataservices.com.

Additional background

Azure Extension for SQL Server uses the following endpoints:

  • DPS: dataprocessingservice.<region>.arcdataservices.com, or for the US Government Virginia region, *.<region>.arcdataservices.azure.us.
  • Telemetry telemetry.<region>.arcdataservices.com

Replace <region> with the short name of the Azure region where the Arc machine resource is located. The short name is derived from the Azure region name without spaces and all lower case.

For example, if your Arc machine resource is located in East US 2 the short name of the region is eastus2 and the telemetry endpoint is:

telemetry.eastus2.arcdataservices.com

If your extension is older than March 11, 2024, it might use older endpoints. Update your extension to use the current endpoint.

Note

The endpoint values before *.arcdataservices.com are subject to change.

Use an HTTPS proxy server for outbound connectivity

If your network requires using an HTTPS proxy server for outbound connectivity, you can read more about configuring that at Update or remove proxy settings.

Query Azure Resource Graph for telemetry upload stats

Use Azure Resource Graph to query the upload status for your environment.

resources
    | where type =~ 'microsoft.hybridcompute/machines/extensions'
    | where properties.type in ('WindowsAgent.SqlServer','LinuxAgent.SqlServer')
    | parse id with * '/providers/Microsoft.HybridCompute/machines/' machineName '/extensions/' *
    | parse properties with * 'uploadStatus : ' uploadStatus ';' *
    | project uploadStatus, subscriptionId, resourceGroup, machineName
    | where uploadStatus !in ('OK') //comment this out to see all upload stats
    | order by uploadStatus desc

Find SQL extensions that haven't connected to DPS in a long time

Query Azure Resource Graph to find extensions that haven't connected to DPS recently.

:::code language="powershell" source="~/../sql-server-samples/samples/features/azure-arc/troubleshooting/hybrid-compute-extension-last-connect.kql":::

Error codes

The following table shows some of the common DPS upload status values and what you can do to troubleshoot further.

DPS upload status value HTTP error code Troubleshooting suggestions
0 Likely cause: a firewall is blocking the transmission of the data to the DPS. Open the firewall to the DNS endpoint for the DPS (TCP, port: 443).
OK 200 The connection is working as expected.
Bad request 400 Possible cause: The resource name (SQL Server instance or database name) doesn't conform to Azure resource naming conventions. For example, if the database name is a reserved word.
Unauthorized 401 Likely cause: the extension is configured to send data through an HTTP proxy that requires authentication. Using an HTTP proxy that requires authentication isn't currently supported. Use an unauthenticated HTTP proxy or no proxy.
Forbidden 403 Check to make sure the Microsoft.AzureArcData resource provider is registered on the subscription. If the Azure Connected Machine agent is otherwise working as expected and this error doesn't resolve itself after a reboot, create a support case with Microsoft Support through the Azure portal.
NotFound 404 The endpoint that the extension is trying to connect to doesn't exist.

To check which endpoint it's trying to connect to, search the logs for dataprocessingservice. This condition can happen if the Azure Connected Machine agent was deployed and connected to an Azure region in which the Microsoft.AzureArcData resource provider isn't yet available. Redeploy the Azure Connected Machine agent in a region that the Microsoft.AzureArcData resource provider for aSQL Server enabled by Azure Arc is available. See also Region availability.
It's possible that the DNS resolver cache isn't refreshed for your machine. To refresh:
- On Windows run: ipconfig /flushdns
- On Linux (if systemd is being used) run: sudo resolvectl flush-caches
Conflict 409 Likely cause: temporary error happening inside of the DPS. If this doesn't resolve itself, create a support case with Microsoft Support through the Azure portal.
InternalServerError 500 This is an error that is happening inside of the DPS. Create a support case with Microsoft Support through the Azure portal.

Related content