Skip to content

polybase-versioned-feature-summary.md

Latest commit

 

History

History
81 lines (54 loc) · 5.66 KB

polybase-versioned-feature-summary.md

File metadata and controls

81 lines (54 loc) · 5.66 KB
title PolyBase features and limitations
description PolyBase features available for SQL Server products and services, including a list of T-SQL operators supported for pushdown and known limitations.
author MikeRayMSFT
ms.author mikeray
ms.reviewer randolphwest, hudequei
ms.date 01/09/2024
ms.service sql
ms.subservice polybase
ms.topic concept-article
monikerRange >=aps-pdw-2016||=azuresqldb-current||=azure-sqldw-latest||>=sql-server-2016||>=sql-server-linux-2017||=azuresqldb-mi-current

PolyBase features and limitations

[!INCLUDE sqlserver2016-asdb-asa-pdw-md]

This article is a summary of PolyBase features available for [!INCLUDE ssNoVersion] products and services.

Feature summary for product releases

This table lists the key features for PolyBase and the products in which they're available.

Feature [!INCLUDE ssNoVersion] (Beginning with 2016) Azure SQL Database [!INCLUDE ssazuresynapse-md] Parallel Data Warehouse
Query Hadoop data with [!INCLUDE tsql] Yes No No Yes
Import data from Hadoop Yes No No Yes
Export data to Hadoop Yes No No Yes
Query, import from, export to Azure HDInsight No No No No
Push down query computations to Hadoop Yes No No Yes
Import data from Azure Blob storage Yes Yes 1 Yes Yes
Export data to Azure Blob storage Yes No Yes Yes
Import data from Azure Data Lake Store No No Yes No
Export data to Azure Data Lake Store No No Yes No
Run PolyBase queries from Microsoft BI tools Yes No Yes Yes

1 Introduced in [!INCLUDE sssql17-md], see Examples of bulk access to data in Azure Blob Storage.

Known limitations

PolyBase has the following limitations:

  • Before [!INCLUDE sssql19-md], the maximum possible row size, which includes the full length of variable length columns, can't exceed 32 KB in [!INCLUDE ssNoVersion] or 1 MB in [!INCLUDE ssazuresynapse-md]. In [!INCLUDE sssql19-md] and later versions, this limitation is lifted. The limit remains 1 MB for Hadoop data sources, but is limited only by the maximum [!INCLUDE ssNoVersion] limit for other data sources.

  • When data is exported into an ORC file format from [!INCLUDE ssNoVersion] or [!INCLUDE ssazuresynapse-md], text-heavy columns might be limited. They can be limited to as few as 50 columns because of Java out-of-memory error messages. To work around this issue, export only a subset of the columns.

  • PolyBase can't connect to any Hadoop instance if Knox is enabled.

  • If you use Hive tables with transactional = true, PolyBase can't access the data in the Hive table's directory.

  • PolyBase services require SQL Server service to have TCP/IP network protocol enabled to function correctly. Additionally, if TCP/IP Protocol configuration setting Listen All is set to No, the following TCP/IP configurations are required:

    • An entry for the correct listener port in either TCP Dynamic Ports or TCP Ports under IPAll. This is required due to the way PolyBase services resolve the listener port of the SQL Server Engine.
    • Enable the desired IP address(es) in TCP/IP Properties > IP Addresses; otherwise, SQL Server Engine will not accept any TCP/IP connections and PolyBase services will still fail to connect to SQL Server Engine.

  • PolyBase on SQL Server on Linux doesn't function if IPv6 is disabled in the kernel. For more information, see SQL Server on Linux: Known issues.

  • PolyBase services require Shared Memory protocol to be enabled to function properly.

  • If you have a default SQL Server instance that is configured to listen on TCP port other than 1433, you can't use it as a head node in a PolyBase scale-out group. When you execute sp_polybase_join_group, if you pass 'MSSQLSERVER' as the instance name, SQL Server assumes port 1433 is the listener port, so the Data Movement service is unable to connect to the head node when starting.

  • Oracle synonyms aren't supported for usage with PolyBase.

  • UTF-8 collations aren't supported for Hadoop external data sources.

  • Hadoop is no longer supported in [!INCLUDE sssql22-md] and later versions.

  • Server name length is bound by 15-character NetBIOS name length limitation. If you change the server name after creating it to be 16 characters or longer, PolyBase Data Movement service won't start.

  • PolyBase scale-out groups are not supported with Always On availability groups.

  • Databases that get installed with PolyBase (DWConfiguration, DWDiagnostics, DWQueue) must be in read-write mode in order for PolyBase to function and for Cumulative Update installations to succeed. Therefore, it is not supported to have these databases in an availability group.

::: moniker range="= sql-server-2016 "

::: moniker-end

Related content