You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: 0_Azure/WhatsNew/2_FabricConfVegas2025.md
+108Lines changed: 108 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,6 +20,7 @@ Last updated: 2025-04-07
20
20
-[FabCon 2025: Fueling tomorrow’s AI with new agentic capabilities and security innovations in Fabric](https://www.microsoft.com/en-us/microsoft-fabric/blog/2025/03/31/fabcon-2025-fueling-tomorrows-ai-with-new-agentic-capabilities-and-security-innovations-in-fabric/)
-[Best practices for OneLake security](https://learn.microsoft.com/en-us/fabric/onelake/security/best-practices-secure-data-in-onelake)
23
+
-[What is Mirroring in Fabric?](https://learn.microsoft.com/en-us/fabric/database/mirrored-database/overview)
23
24
24
25
</details>
25
26
@@ -79,6 +80,113 @@ Last updated: 2025-04-07
79
80
-**Enhanced Data Pipelines**: The Variable Library is `already available for use in data pipelines, allowing for more streamlined and efficient data processing`.
80
81
-**Future Integrations**: Plans are in place to `expand the use of the Variable Library to other areas within the Fabric platform, further enhancing its utility`.
81
82
83
+
## Data Integration
84
+
85
+
> Database mirroring -> enables seamless data replication and high availability for critical workloads.
|**Azure SQL Database**| Current Support | On-Premises Data Gateway or VNET Data Gateway for secure and efficient data movement | - **Mirroring Mechanism**: Utilizes SQL’s Change Data Capture (CDC) stack optimized for lake-centric architecture. CDC stores changes locally in the database, while mirroring reads data from the transaction log and publishes it to OneLake.<br>- **Data Format**: Converts data to Parquet format for analytics.<br>- **Features**: Supports DDL (Data Definition Language) operations like Alter/Drop/Rename tables/columns, and Truncate tables while mirroring is active.<br>- **APIs**: Programmatic APIs available for setup and management. |
90
+
|**Snowflake**| Current Support | On-Premises Data Gateway or VNET Data Gateway for secure and efficient data movement | - **Mirroring Mechanism**: Similar to Azure SQL Database, leveraging secure gateways for data replication.<br>- **Data Format**: Ensures data is analytics-ready in OneLake. |
91
+
|**Azure SQL Managed Instance (MI)**| Coming Soon | Will utilize On-Premises Data Gateway or VNET Data Gateway for secure connectivity | - **Mirroring Mechanism**: Expected to follow the same CDC-based approach as Azure SQL Database.<br>- **Data Format**: Will convert data to Parquet format for seamless integration. |
92
+
|**On-Premises SQL Server, Oracle, and Dataverse**| Future Support | Will follow secure connectivity protocols using On-Premises Data Gateway | - **Mirroring Mechanism**: Anticipated to use similar CDC-based replication.<br>- **Data Format**: Data will be converted to Parquet format for analytics. |
93
+
94
+
> Installation Steps:
95
+
<details>
96
+
<summary>Azure SQL Database</summary>
97
+
98
+
1.**Enable System Assigned Managed Identity (SAMI)**:
99
+
- In the Azure portal, navigate to your Azure SQL logical server.
100
+
- Enable System Assigned Managed Identity (SAMI) with a single step.
101
+
102
+
2.**Configure Mirroring**:
103
+
- Go to the Fabric workspace and select the ⚙️.
104
+
- Choose `Manage connection and gateways` to configure the connection to your Azure SQL Database.
105
+
106
+
3.**Set Up Gateway**:
107
+
- Deploy either the On-Premises Data Gateway or VNET Data Gateway depending on your network setup.
108
+
- Configure the gateway to securely connect to your Azure SQL Database.
109
+
110
+
4.**Initial Replication**:
111
+
- Start the mirroring process. The initial replication time depends on the size of the data being brought in.
112
+
- Data is stored in a landing zone in OneLake, improving performance when converting files into delta verti-parquet.
113
+
114
+
5.**Monitor Replication**:
115
+
- Use Dynamic Management Views (DMVs) and stored procedures to validate configuration and monitor replication status.
116
+
- Execute queries like `SELECT * FROM sys.dm_change_feed_log_scan_sessions` to check if changes are properly flowing.
117
+
118
+
6.**Manage Connections**:
119
+
- Regularly check and manage connections through the Fabric workspace settings.
120
+
- Ensure compliance with security protocols and monitor data transfer activities.
121
+
122
+
7.**Troubleshooting**: If experiencing mirroring problems, perform database-level checks and contact support if needed.
123
+
124
+
</details>
125
+
126
+
<details>
127
+
<summary>Snowflake</summary>
128
+
129
+
1.**Configure Snowflake Account**: Ensure your Snowflake account is set up and accessible.
130
+
131
+
2.**Set Up Gateway**:
132
+
- Deploy either the On-Premises Data Gateway or VNET Data Gateway depending on your network setup.
133
+
- Configure the gateway to securely connect to your Snowflake account.
134
+
135
+
3.**Configure Mirroring**:
136
+
- Go to the Fabric workspace and select the ⚙️.
137
+
- Choose "Manage connection and gateways" to configure the connection to your Snowflake account.
138
+
139
+
4.**Initial Replication**:
140
+
- Start the mirroring process. The initial replication time depends on the size of the data being brought in.
141
+
- Data is stored in a landing zone in OneLake, improving performance when converting files into delta verti-parquet.
142
+
143
+
5.**Monitor Replication**: Use Snowflake's monitoring tools to validate configuration and monitor replication status.
144
+
145
+
6.**Manage Connections**:
146
+
- Regularly check and manage connections through the Fabric workspace settings.
147
+
- Ensure compliance with security protocols and monitor data transfer activities.
148
+
149
+
7.**Troubleshooting**: If experiencing mirroring problems, perform account-level checks and contact support if needed.
|**On-Premises Data Gateway**| Allows Fabric services to communicate with on-premises data sources without exposing them directly to the internet. | Ensures encrypted data transfers while maintaining compliance and security protocols. | Ideal for organizations needing to access mirrored databases behind on-premises firewalls. |
189
+
|**VNET Data Gateway**| Enables Fabric services like Data Factory and Power BI to securely access cloud databases without requiring on-premises infrastructure. | Maintains network isolation while ensuring low-latency and high-throughput connections. | Suitable for databases residing behind virtual network (VNET) firewalls. |
0 commit comments