From 9f99fe7fcbdf8f7444550760d8217a2d0ea38f0a Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Mon, 7 Apr 2025 08:44:03 -0600
Subject: [PATCH 22/31] in progress
---
0_Azure/WhatsNew/2_FabricConfVegas2025.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/0_Azure/WhatsNew/2_FabricConfVegas2025.md b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
index 795c7a566..f97deeaf3 100644
--- a/0_Azure/WhatsNew/2_FabricConfVegas2025.md
+++ b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
@@ -188,6 +188,7 @@ Last updated: 2025-04-07
| **On-Premises Data Gateway** | Allows Fabric services to communicate with on-premises data sources without exposing them directly to the internet. | Ensures encrypted data transfers while maintaining compliance and security protocols. | Ideal for organizations needing to access mirrored databases behind on-premises firewalls. |
| **VNET Data Gateway** | Enables Fabric services like Data Factory and Power BI to securely access cloud databases without requiring on-premises infrastructure. | Maintains network isolation while ensuring low-latency and high-throughput connections. | Suitable for databases residing behind virtual network (VNET) firewalls. |
+## Data Engineering & Data Science
Total Visitors
From 0bb5f2a144241b1b7bea17a1cf416caa8ea77762 Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Mon, 7 Apr 2025 09:02:40 -0600
Subject: [PATCH 23/31] in progress
---
.../0_Fabric/demos/34_FromSynapse-Fabric.md | 72 +++++++++++++++++++
1 file changed, 72 insertions(+)
create mode 100644 0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md
diff --git a/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md b/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md
new file mode 100644
index 000000000..7f9041ed9
--- /dev/null
+++ b/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md
@@ -0,0 +1,72 @@
+# Migrate to Microsoft Fabric - Overview
+
+ Costa Rica
+
+[](https://github.com)
+[](https://github.com/)
+[brown9804](https://github.com/brown9804)
+
+Last updated: 2025-04-07
+
+----------
+
+
+List of References (Click to expand)
+
+- [Migrating from Azure Synapse Spark to Fabric](https://learn.microsoft.com/en-us/fabric/data-engineering/migrate-synapse-overview)
+- [Migration planning: Azure Synapse Analytics dedicated SQL pools to Fabric Data Warehouse](https://learn.microsoft.com/en-us/fabric/data-warehouse/migration-synapse-dedicated-sql-pool-warehouse#prepare-for-migration)
+
+
+
+> Overall process:
+> - Create a Fabric workspace and link it to your existing Azure Data Lake Storage account.
+> - Import existing notebooks, Spark jobs into the Fabric workspace.
+> - Update references to external tables or files using the OneLake syntax.
+
+ ## Overview
+
+1. **Create a Fabric Workspace**
+ - **Navigate to Microsoft Fabric**: Access the [Microsoft Fabric portal](https://app.fabric.microsoft.com/).
+
+
+
+ - **Create Workspace**: Follow the steps to create a new workspace.
+
+ https://github.com/user-attachments/assets/209f18fd-d14a-4e2f-aff2-d6cb7be13cc1
+
+2. **Link ADLS Account**: Connect your existing Azure Data Lake Storage account to the new Fabric workspace.
+ - **Set Up ADLS Connection**:
+ - **Open Fabric Workspace**: Go to the workspace where you want to set up the connection.
+
+
+
+ - **Create a lakehouse**: Click on `New item`, search for `Lakehouse`:
+
+
+
+
+
+ - **Get Data**: Select the option to get data from Azure Data Lake Storage Gen2.
+ - **Enter ADLS URL**: Provide the URL to your Azure Data Lake Storage Gen2 account.
+ - **Authentication**: Choose the authentication method (e.g., account key, organizational account, service principal, or shared access signature).
+ - **Sign In**: Sign in to your Azure Data Lake Storage account using the selected authentication method.
+ - **Configure Connection**: Complete the configuration by specifying any additional settings required for the connection.
+ - **Create OneLake Shortcut**:
+ - **Navigate to Lakehouse View**: In the Fabric workspace, go to the Lakehouse view.
+ - **Create Shortcut**: Right-click and select "New Shortcut," then choose Azure Data Lake Storage Gen2 as the source.
+ - **Enter Details**: Provide the URL of the ADLS Gen2 account, connection name, authentication method, shortcut name, and subpath pointing to the root of the data you want to reference.
+ - **Create Shortcut**: Click "Create" and wait for the shortcut to appear in the Lakehouse view.
+ - **Preview Data**: Preview the data from the shortcut to ensure it is virtualized from the source.
+2. **Import Existing Notebooks and Spark Jobs**
+ - **Export from Synapse**: Export your notebooks and Spark job definitions from Azure Synapse.
+ - **Import to Fabric**: Use the import functionality in Fabric to bring these items into your new workspace.
+3. **Update References**
+ - **Identify External References**: Locate all references to external tables or files in your notebooks and Spark jobs.
+ - **Modify Syntax**: Update these references to use the OneLake syntax, ensuring they point to the correct locations in Fabric.
+
+
+
+
+
Total Visitors
+
+
From 630ef0df3ec0d4909c95b462b76c66d383839f10 Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Mon, 7 Apr 2025 09:32:29 -0600
Subject: [PATCH 24/31] title
---
.../2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md b/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md
index 7f9041ed9..7038f82f8 100644
--- a/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md
+++ b/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md
@@ -1,4 +1,4 @@
-# Migrate to Microsoft Fabric - Overview
+# Azure Synapse migrate to Microsoft Fabric - Overview
Costa Rica
From 8b0a008a5b0485fc4add5bf0d8a4c1f3207dfe94 Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Mon, 7 Apr 2025 09:40:55 -0600
Subject: [PATCH 25/31] format
---
.../0_Fabric/demos/20_FabricCapacityMetrics.md | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/0_Azure/2_AzureAnalytics/0_Fabric/demos/20_FabricCapacityMetrics.md b/0_Azure/2_AzureAnalytics/0_Fabric/demos/20_FabricCapacityMetrics.md
index abd99b0d6..908f02942 100644
--- a/0_Azure/2_AzureAnalytics/0_Fabric/demos/20_FabricCapacityMetrics.md
+++ b/0_Azure/2_AzureAnalytics/0_Fabric/demos/20_FabricCapacityMetrics.md
@@ -16,7 +16,7 @@ Last updated: 2024-11-28
## Wiki
-Table of Wiki (Click to expand)
+Lis of References (Click to expand)
- [Install the Microsoft Fabric capacity metrics app](https://learn.microsoft.com/en-us/fabric/enterprise/metrics-app-install?tabs=1st)
- [What is the Microsoft Fabric Capacity Metrics app?](https://learn.microsoft.com/en-us/fabric/enterprise/metrics-app)
@@ -31,8 +31,6 @@ Last updated: 2024-11-28
## Content
-- [Wiki](#wiki)
-- [Content](#content)
- [Microsoft Fabric Capacity Metrics app](#microsoft-fabric-capacity-metrics-app)
- [Installation Steps](#installation-steps)
- [Configuration Steps](#configuration-steps)
From bc02a9281f44d29adfbf232535070b3abe1f6eb6 Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Mon, 7 Apr 2025 09:55:47 -0600
Subject: [PATCH 26/31] format
---
.../0_Fabric/demos/20_FabricCapacityMetrics.md | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/0_Azure/2_AzureAnalytics/0_Fabric/demos/20_FabricCapacityMetrics.md b/0_Azure/2_AzureAnalytics/0_Fabric/demos/20_FabricCapacityMetrics.md
index 908f02942..09ce33d5a 100644
--- a/0_Azure/2_AzureAnalytics/0_Fabric/demos/20_FabricCapacityMetrics.md
+++ b/0_Azure/2_AzureAnalytics/0_Fabric/demos/20_FabricCapacityMetrics.md
@@ -288,3 +288,8 @@ Steps to Access Microsoft Purview via Audit Logs:
+
+
+
Total Visitors
+
+
- **Data Format**: Converts data to Parquet format for analytics.
- **Features**: Supports DDL (Data Definition Language) operations like Alter/Drop/Rename tables/columns, and Truncate tables while mirroring is active.
- **APIs**: Programmatic APIs available for setup and management. |
| **Snowflake** | Current Support | On-Premises Data Gateway or VNET Data Gateway for secure and efficient data movement | - **Mirroring Mechanism**: Similar to Azure SQL Database, leveraging secure gateways for data replication.
- **Data Format**: Ensures data is analytics-ready in OneLake. |
-| **Azure SQL Managed Instance (MI)**| Coming Soon | Will utilize On-Premises Data Gateway or VNET Data Gateway for secure connectivity | - **Mirroring Mechanism**: Expected to follow the same CDC-based approach as Azure SQL Database.
- **Data Format**: Will convert data to Parquet format for seamless integration. |
+| **Azure SQL Managed Instance (MI)**| Coming Soon | Will use On-Premises Data Gateway or VNET Data Gateway for secure connectivity | - **Mirroring Mechanism**: Expected to follow the same CDC-based approach as Azure SQL Database.
- **Data Format**: Will convert data to Parquet format for seamless integration. |
| **On-Premises SQL Server, Oracle, and Dataverse** | Future Support | Will follow secure connectivity protocols using On-Premises Data Gateway | - **Mirroring Mechanism**: Anticipated to use similar CDC-based replication.
- **Data Format**: Data will be converted to Parquet format for analytics. |
> Installation Steps:
@@ -96,19 +96,31 @@ Last updated: 2025-04-07
Azure SQL Database
1. **Enable System Assigned Managed Identity (SAMI)**:
- - In the Azure portal, navigate to your Azure SQL logical server.
+ - In the [Azure portal](https://portal.azure.com/), navigate to your Azure SQL logical server.
- Enable System Assigned Managed Identity (SAMI) with a single step.
+ 
+
2. **Configure Mirroring**:
- - Go to the Fabric workspace and select the ⚙️.
+ - Go to the [Fabric workspace](https://app.fabric.microsoft.com) and select the ⚙️.
- Choose `Manage connection and gateways` to configure the connection to your Azure SQL Database.
+ 
+
3. **Set Up Gateway**:
- Deploy either the On-Premises Data Gateway or VNET Data Gateway depending on your network setup.
+
+ 
+
- Configure the gateway to securely connect to your Azure SQL Database.
+ 
+
4. **Initial Replication**:
- - Start the mirroring process. The initial replication time depends on the size of the data being brought in.
+ - Start the mirroring process. The initial replication time depends on the size of the data being brought in. Click on `New item`, search for `Mirrored`:
+
+ 
+
- Data is stored in a landing zone in OneLake, improving performance when converting files into delta verti-parquet.
5. **Monitor Replication**:
From b93f3d9353fdad6266e5cb393d9fbb2ea3c75f41 Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Mon, 7 Apr 2025 11:32:40 -0600
Subject: [PATCH 28/31] Update 2_FabricConfVegas2025.more visual guidnace about
setting that in tenant
---
0_Azure/WhatsNew/2_FabricConfVegas2025.md | 47 +++++++++++++++++------
1 file changed, 36 insertions(+), 11 deletions(-)
diff --git a/0_Azure/WhatsNew/2_FabricConfVegas2025.md b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
index dfcc169f4..1902a27ec 100644
--- a/0_Azure/WhatsNew/2_FabricConfVegas2025.md
+++ b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
@@ -24,19 +24,14 @@ Last updated: 2025-04-07
-## Overview
+## Content
+
+
+## Platform Enhancements
| **Category** | **Details** |
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Platform Enhancements** | - **OneLake Security (Private Preview)**: Seamless security integration across Spark, SQL Endpoint, and Power BI in Direct Lake mode.
- **Command Line Interface (CLI)**: Execute commands across Fabric using terminal prompts or pre-written scripts.
- **Variable Library**: Efficiently manage variables at a workspace level. |
-| **Data Integration** | **Database Mirroring**: Supports sources behind firewalls such as Azure SQL Database and Snowflake, with Azure SQL MI coming soon. On-prem SQL Server, Oracle, and Dataverse support is also on the way.|
-| **Data Engineering & Data Science** | - **Fabric Data Agents (Former AI Skills)**: Integrate Data Agents with Azure AI Foundry.
- **Copilot & AI Features**: Available with all paid-SKU, starting with F2.
- **Autoscale Billing for Spark**: Serverless, pay-as-you-go billing, no longer consuming your Fabric capacity when enabled.
- **Copilot Improvements**: Copilot is now pre-installed, supports data-specific queries, and tracks historic interactions.
- **AI Functions**: Apply LLM models for summarization, text generation, classification, sentiment analysis, and translation.
- **Row-Level & Column-Level Security in Spark**. |
-| **Data Warehouse** | - **AI Functions**: Bring AI features to Data Warehouse, similar to Data Engineering & Data Science.
- **User-Defined Functions (UDFs)**: Write custom Python functions and execute them directly through T-SQL.
- **Migration Tool**: Public preview for migrating Azure Synapse Analytics warehouses to Fabric. |
-| **Real-Time Intelligence** | **Streaming & Event Connectors**: New integrations for MQTT, Solace, ADX, Real-Time Weather, and Azure Event Grid.|
-| **Power BI** | - **Direct Lake Semantic Models**: Create models directly in Power BI Desktop, and use data from multiple OneLake sources.
- **Datapoint Annotations**: Add descriptive text to specific data points directly in PowerPoint presentations.
- **Copilot Improvements**: Ask data questions from your model.
- **Notebook Tools**: Quickly analyze semantic models with pre-built notebooks for best practices and memory optimization. |
-| **Additional Highlights** | - **Workshops and Sessions**: Over 200 expert-led sessions covering AI, databases, analytics, business intelligence, and more.
- **Hands-On Workshops**: Pre- and post-conference workshops offering immersive, practical learning experiences. |
-
-## Platform Enhancements
> Setting Up OneLake Security:
@@ -80,8 +75,14 @@ Last updated: 2025-04-07
- **Enhanced Data Pipelines**: The Variable Library is `already available for use in data pipelines, allowing for more streamlined and efficient data processing`.
- **Future Integrations**: Plans are in place to `expand the use of the Variable Library to other areas within the Fabric platform, further enhancing its utility`.
+ https://github.com/user-attachments/assets/3ee8eee2-cd0b-4575-aacc-5601c0ab49e2
+
## Data Integration
+| **Category** | **Details** |
+|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Data Integration** | **Database Mirroring**: Supports sources behind firewalls such as Azure SQL Database and Snowflake, with Azure SQL MI coming soon. On-prem SQL Server, Oracle, and Dataverse support is also on the way.|
+
> Database mirroring -> enables seamless data replication and high availability for critical workloads.
| **Source** | **Support Status** | **Connectivity** | **Technical Details**|
@@ -146,7 +147,7 @@ Last updated: 2025-04-07
3. **Configure Mirroring**:
- Go to the Fabric workspace and select the ⚙️.
- - Choose "Manage connection and gateways" to configure the connection to your Snowflake account.
+ - Choose `Manage connection and gateways` to configure the connection to your Snowflake account.
4. **Initial Replication**:
- Start the mirroring process. The initial replication time depends on the size of the data being brought in.
@@ -171,7 +172,7 @@ Last updated: 2025-04-07
2. **Configure Mirroring**:
- Go to the Fabric workspace and select the ⚙️.
- - Choose "Manage connection and gateways" to configure the connection to your Azure SQL Managed Instance.
+ - Choose `Manage connection and gateways` to configure the connection to your Azure SQL Managed Instance.
3. **Set Up Gateway**:
- Deploy either the On-Premises Data Gateway or VNET Data Gateway depending on your network setup.
@@ -202,6 +203,30 @@ Last updated: 2025-04-07
## Data Engineering & Data Science
+| **Category** | **Details** |
+|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Data Engineering & Data Science** | - **Fabric Data Agents (Former AI Skills)**: Integrate Data Agents with Azure AI Foundry.
- **Copilot & AI Features**: Available with all paid-SKU, starting with F2.
- **Autoscale Billing for Spark**: Serverless, pay-as-you-go billing, no longer consuming your Fabric capacity when enabled.
- **Copilot Improvements**: Copilot is now pre-installed, supports data-specific queries, and tracks historic interactions.
- **AI Functions**: Apply LLM models for summarization, text generation, classification, sentiment analysis, and translation.
- **Row-Level & Column-Level Security in Spark**. |
+
+## Data Warehouse
+
+| **Category** | **Details** |
+|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Data Warehouse** | - **AI Functions**: Bring AI features to Data Warehouse, similar to Data Engineering & Data Science.
- **User-Defined Functions (UDFs)**: Write custom Python functions and execute them directly through T-SQL.
- **Migration Tool**: Public preview for migrating Azure Synapse Analytics warehouses to Fabric. |
+
+## Real-Time Intelligence
+
+| **Category** | **Details** |
+|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Real-Time Intelligence** | **Streaming & Event Connectors**: New integrations for MQTT, Solace, ADX, Real-Time Weather, and Azure Event Grid.|
+
+## Power BI
+
+| **Category** | **Details** |
+|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Power BI** | - **Direct Lake Semantic Models**: Create models directly in Power BI Desktop, and use data from multiple OneLake sources.
- **Datapoint Annotations**: Add descriptive text to specific data points directly in PowerPoint presentations.
- **Copilot Improvements**: Ask data questions from your model.
- **Notebook Tools**: Quickly analyze semantic models with pre-built notebooks for best practices and memory optimization. |
+
+
+
Total Visitors
From ddd0be4d251b4e7d17927b6b734d8156ab485ccb Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Mon, 7 Apr 2025 16:03:02 -0600
Subject: [PATCH 29/31] in progress
---
0_Azure/WhatsNew/2_FabricConfVegas2025.md | 111 +++++++++++++++++++++-
1 file changed, 110 insertions(+), 1 deletion(-)
diff --git a/0_Azure/WhatsNew/2_FabricConfVegas2025.md b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
index 1902a27ec..1c9396805 100644
--- a/0_Azure/WhatsNew/2_FabricConfVegas2025.md
+++ b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
@@ -21,6 +21,13 @@ Last updated: 2025-04-07
- [OneLake security overview](https://learn.microsoft.com/en-us/fabric/onelake/security/get-started-security)
- [Best practices for OneLake security](https://learn.microsoft.com/en-us/fabric/onelake/security/best-practices-secure-data-in-onelake)
- [What is Mirroring in Fabric?](https://learn.microsoft.com/en-us/fabric/database/mirrored-database/overview)
+- [Introducing Autoscale Billing for Spark in Microsoft Fabric](https://blog.fabric.microsoft.com/en-US/blog/introducing-autoscale-billing-for-data-engineering-in-microsoft-fabric/)
+- [Understand the metrics app Autoscale compute for Spark page](https://learn.microsoft.com/en-us/fabric/enterprise/metrics-app-feature-autoscale-page)
+- [Billing and utilization reporting for Apache Spark in Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/data-engineering/billing-capacity-management-for-spark)
+- [Overview of Copilot for Data Science and Data Engineering (preview)](https://learn.microsoft.com/en-us/fabric/data-engineering/copilot-notebooks-overview)
+- [Transform and enrich data seamlessly with AI functions (Preview)](https://learn.microsoft.com/en-us/fabric/data-science/ai-functions/overview?tabs=pandas)
+- [Summarize text with the ai.summarize function](https://learn.microsoft.com/en-us/fabric/data-science/ai-functions/summarize?tabs=column-summary)
+
@@ -149,6 +156,8 @@ Last updated: 2025-04-07
- Go to the Fabric workspace and select the ⚙️.
- Choose `Manage connection and gateways` to configure the connection to your Snowflake account.
+
+
4. **Initial Replication**:
- Start the mirroring process. The initial replication time depends on the size of the data being brought in.
- Data is stored in a landing zone in OneLake, improving performance when converting files into delta verti-parquet.
@@ -174,6 +183,8 @@ Last updated: 2025-04-07
- Go to the Fabric workspace and select the ⚙️.
- Choose `Manage connection and gateways` to configure the connection to your Azure SQL Managed Instance.
+
+
3. **Set Up Gateway**:
- Deploy either the On-Premises Data Gateway or VNET Data Gateway depending on your network setup.
- Configure the gateway to securely connect to your Azure SQL Managed Instance.
@@ -205,7 +216,105 @@ Last updated: 2025-04-07
| **Category** | **Details** |
|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Data Engineering & Data Science** | - **Fabric Data Agents (Former AI Skills)**: Integrate Data Agents with Azure AI Foundry.
- **Copilot & AI Features**: Available with all paid-SKU, starting with F2.
- **Autoscale Billing for Spark**: Serverless, pay-as-you-go billing, no longer consuming your Fabric capacity when enabled.
- **Copilot Improvements**: Copilot is now pre-installed, supports data-specific queries, and tracks historic interactions.
- **AI Functions**: Apply LLM models for summarization, text generation, classification, sentiment analysis, and translation.
- **Row-Level & Column-Level Security in Spark**. |
+| **Data Engineering & Data Science** | - **Fabric Data Agents (Former AI Skills)**: Integrate Data Agents with Azure AI Foundry. Click [here to see a quick demo about it](https://github.com/MicrosoftCloudEssentials-LearningHub/DemosScenarios-TechTalks/blob/main/0_Azure/2_AzureAnalytics/0_Fabric/demos/2_FabricAISkills.md)
- **Copilot & AI Features**: Available with all paid-SKU, starting with F2.
- **Autoscale Billing for Spark**: Serverless, pay-as-you-go billing, no longer consuming your Fabric capacity when enabled.
- **Copilot Improvements**: Copilot is now pre-installed, supports data-specific queries, and tracks historic interactions.
- **AI Functions**: Apply LLM models for summarization, text generation, classification, sentiment analysis, and translation.
- **Row-Level & Column-Level Security in Spark**. |
+
+### Fabric Data Agents (Former AI Skills)
+
+> Fabric Data Agents, previously known as AI Skills, are integrated with Azure AI Foundry to enhance AI capabilities. These agents are designed to retrieve, process, and present data from various sources effectively, leveraging specialized query languages.
+
+
+Data Sources
+
+> Fabric Data Agents can access and integrate data from a wide range of sources, ensuring comprehensive data retrieval and analysis. Here are the specific data sources they can interact with:
+
+- **Lakehouse Data**:
+ - **Description**: A unified data platform that combines the best features of data lakes and data warehouses.
+ - **Usage**: Ideal for storing large volumes of raw data and structured data, enabling advanced analytics and machine learning.
+
+- **Warehouse Data**:
+ - **Description**: Traditional data warehouses store structured data optimized for query performance and reporting.
+ - **Usage**: Suitable for business intelligence, reporting, and data analysis tasks.
+
+- **Power BI Semantic Models**:
+ - **Description**: Semantic models in Power BI provide a structured and meaningful representation of data.
+ - **Usage**: Used for creating interactive reports and dashboards, enabling users to explore data visually.
+
+- **KQL Databases**:
+ - **Description**: Databases that use Kusto Query Language (KQL) for querying large datasets.
+ - **Usage**: Commonly used in log analytics, monitoring, and telemetry data analysis.
+
+
+
+Query Languages
+
+> Fabric Data Agents utilize specialized query languages to handle complex queries and data manipulations efficiently:
+
+- **SQL (Structured Query Language)**:
+ - **Usage**: Widely used for managing and querying relational databases.
+ - **Capabilities**: Supports data retrieval, insertion, updating, and deletion operations.
+
+- **KQL (Kusto Query Language)**:
+ - **Usage**: Designed for querying large datasets, particularly in log analytics and telemetry data.
+ - **Capabilities**: Provides powerful data exploration and analysis features.
+
+- **DAX (Data Analysis Expressions)**:
+ - **Usage**: Used in Power BI, Excel, and SQL Server Analysis Services for data modeling and analysis.
+ - **Capabilities**: Enables the creation of calculated columns, measures, and custom aggregations.
+
+
+
+Custom Conversational AI Agents
+
+> Organizations can create custom conversational AI agents leveraging domain expertise. These agents provide real-time insights and enhance data-driven decision-making processes. Key features include:
+
+- **Domain Expertise Integration**: Custom agents can be tailored to specific industry needs, incorporating domain-specific knowledge.
+- **Real-Time Insights**: Agents can analyze data in real-time, providing immediate insights and recommendations.
+- **Enhanced Decision-Making**: By leveraging AI capabilities, organizations can make informed decisions based on comprehensive data analysis.
+
+
+### Copilot & AI Features
+
+> Copilot and AI features are accessible with all paid-SKU plans, starting from the F2 tier. This ensures that users subscribing to any paid plan can leverage advanced AI functionalities to enhance their data science and engineering tasks.
+
+| **Capability** | **Description** | **Benefit** |
+|------------------------------------|---------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|
+| **Advanced Data Analysis Tools** | Tools for complex data analyses, including statistical analysis, trend detection, and anomaly identification. | Helps uncover insights from large datasets, identify patterns, and make data-driven decisions more efficiently. |
+| **Model Building and Training** | Features for creating, training, and deploying machine learning models. | Facilitates building sophisticated models with minimal effort, leveraging automated processes for hyperparameter tuning, model evaluation, and deployment. |
+| **Automation of Repetitive Tasks** | AI functionalities that automate routine tasks such as data cleaning, preprocessing, and feature engineering. | Reduces time and effort required for data preparation, allowing focus on more strategic aspects of projects. |
+| **Natural Language Processing (NLP)** | Capabilities for text analysis, including summarization, sentiment analysis, classification, and translation. | Enables extraction of meaningful information from unstructured text data, perform sentiment analysis, classify documents, and translate text between languages. |
+| **Predictive Analytics** | Tools that use historical data to predict future trends and outcomes. | Helps organizations anticipate future events, optimize operations, and make proactive decisions. |
+| **Visualization and Reporting** | Features that generate visualizations and reports to present data insights in an easily understandable format. | Helps communicate findings effectively, making it easier to share insights with stakeholders and drive informed decision-making. |
+| **Integration with Other Services**| Enhanced integration capabilities with other Azure services and third-party tools. | Allows leveraging a broader ecosystem of tools and services, enhancing overall functionality and interoperability of data science workflows. |
+
+### Autoscale Billing for Spark
+
+Serverless, pay-as-you-go billing, no longer consuming your Fabric capacity when enabled. Click [here to read a blog about Introducing Autoscale Billing for Spark in Microsoft Fabric](https://blog.fabric.microsoft.com/en-US/blog/introducing-autoscale-billing-for-data-engineering-in-microsoft-fabric/) and see how it looks.
+
+### Copilot Improvements
+
+> Copilot is now pre-installed, supports data-specific queries, and tracks historic interactions. Click here to see more about [Overview of Copilot for Data Science and Data Engineering (preview)](https://learn.microsoft.com/en-us/fabric/data-engineering/copilot-notebooks-overview)
+
+> Data-Specific Queries:
+
+- **Enhanced Understanding**: Copilot `now supports data-specific queries, allowing users to ask questions directly related to their datasets`.
+- **Context-Aware Responses**: It `understands the schema and metadata of the data, providing accurate and context-aware answers or code snippets`.
+- **Code Generation**: Copilot `can generate code for data transformations, visualizations, and machine learning models based on the user's queries`.
+
+> Historic Interaction Tracking:
+
+- **Memory of Past Interactions**: Copilot tracks `historic interactions, enabling it to provide more personalized and relevant responses based on previous queries`.
+- **Improved Recommendations**: This feature helps in `refining suggestions and improving the accuracy of generated code or insights`.
+
+
+### AI Functions
+
+> Apply LLM models for summarization, text generation, classification, sentiment analysis, and translation.
+
+
+
+
+- Row-Level & Column-Level Security in Spark.
+
## Data Warehouse
From 33c8f72d27bffe41b973093ce3f00b9416c0bf40 Mon Sep 17 00:00:00 2001
From: Timna Brown <24630902+brown9804@users.noreply.github.com>
Date: Mon, 7 Apr 2025 22:21:10 -0600
Subject: [PATCH 30/31] notes
---
0_Azure/WhatsNew/2_FabricConfVegas2025.md | 55 +++++++++++++++--------
1 file changed, 37 insertions(+), 18 deletions(-)
diff --git a/0_Azure/WhatsNew/2_FabricConfVegas2025.md b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
index 1c9396805..e0e8b090c 100644
--- a/0_Azure/WhatsNew/2_FabricConfVegas2025.md
+++ b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
@@ -28,11 +28,28 @@ Last updated: 2025-04-07
- [Transform and enrich data seamlessly with AI functions (Preview)](https://learn.microsoft.com/en-us/fabric/data-science/ai-functions/overview?tabs=pandas)
- [Summarize text with the ai.summarize function](https://learn.microsoft.com/en-us/fabric/data-science/ai-functions/summarize?tabs=column-summary)
-
-## Content
+
+Table of Content (Click to expand)
+
+- [Platform Enhancements](#platform-enhancements)
+ - [OneLake Security Private Preview](#onelake-security-private-preview)
+ - [Command Line Interface CLI](#command-line-interface-cli)
+ - [Variable Library](#variable-library)
+- [Data Integration](#data-integration)
+ - [Gateway Solutions](#gateway-solutions)
+- [Data Engineering & Data Science](#data-engineering--data-science)
+ - [Fabric Data Agents Former AI Skills](#fabric-data-agents-former-ai-skills)
+ - [Copilot & AI Features](#copilot--ai-features)
+ - [Autoscale Billing for Spark](#autoscale-billing-for-spark)
+ - [Copilot Improvements](#copilot-improvements)
+ - [AI Functions](#ai-functions)
+- [Data Warehouse](#data-warehouse)
+- [Real-Time Intelligence](#real-time-intelligence)
+- [Power BI](#power-bi)
+
## Platform Enhancements
@@ -305,36 +322,38 @@ Serverless, pay-as-you-go billing, no longer consuming your Fabric capacity when
- **Memory of Past Interactions**: Copilot tracks `historic interactions, enabling it to provide more personalized and relevant responses based on previous queries`.
- **Improved Recommendations**: This feature helps in `refining suggestions and improving the accuracy of generated code or insights`.
-
### AI Functions
> Apply LLM models for summarization, text generation, classification, sentiment analysis, and translation.
-
-
-- Row-Level & Column-Level Security in Spark.
-
+| **Function** | **Description** | **Use Cases** | **Supported Environments** |
+|--------------------------|---------------------------------------------------------------------------------|---------------------------------------------------|----------------------------|
+| `ai.summarize` | Generates concise summaries of input text. | Summarizing articles, reports, and descriptions. | pandas, PySpark |
+| `ai.generate_response` | Produces coherent and contextually relevant text based on custom user prompts. | Creating content for blogs, stories, and marketing materials. | pandas, PySpark |
+| `ai.classify` | Categorizes input text values according to predefined labels. | Sorting emails, customer feedback, and social media posts. | pandas, PySpark |
+| `ai.analyze_sentiment` | Identifies the emotional state expressed in the input text. | Monitoring brand sentiment, analyzing customer reviews, and gauging public opinion. | pandas, PySpark|
+| `ai.translate` | Translates text from one language to another while maintaining context and meaning. | Translating documents, websites, and communication in multilingual environments. | pandas, PySpark|
+| `ai.extract` | Finds and extracts specific types of information from text. | Extracting locations, names, and other entities from text. | pandas, PySpark |
+| `ai.fix_grammar` | Corrects spelling, grammar, and punctuation in text. | Improving the quality of written content. | pandas, PySpark |
+| `ai.similarity` | Compares the meaning of input text with a common text value or corresponding text values in another column. | Finding similar text entries, deduplication. | pandas, PySpark |
## Data Warehouse
-| **Category** | **Details** |
-|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Data Warehouse** | - **AI Functions**: Bring AI features to Data Warehouse, similar to Data Engineering & Data Science.
- **User-Defined Functions (UDFs)**: Write custom Python functions and execute them directly through T-SQL.
- **Migration Tool**: Public preview for migrating Azure Synapse Analytics warehouses to Fabric. |
+- **AI Functions**: Bring AI features to Data Warehouse, similar to Data Engineering & Data Science.
+- **User-Defined Functions (UDFs)**: Write custom Python functions and execute them directly through T-SQL.
+- **Migration Tool**: Public preview for migrating Azure Synapse Analytics warehouses to Fabric.
## Real-Time Intelligence
-| **Category** | **Details** |
-|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Real-Time Intelligence** | **Streaming & Event Connectors**: New integrations for MQTT, Solace, ADX, Real-Time Weather, and Azure Event Grid.|
+> **Streaming & Event Connectors**: New integrations for MQTT, Solace, ADX, Real-Time Weather, and Azure Event Grid.
## Power BI
-| **Category** | **Details** |
-|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Power BI** | - **Direct Lake Semantic Models**: Create models directly in Power BI Desktop, and use data from multiple OneLake sources.
- **Datapoint Annotations**: Add descriptive text to specific data points directly in PowerPoint presentations.
- **Copilot Improvements**: Ask data questions from your model.
- **Notebook Tools**: Quickly analyze semantic models with pre-built notebooks for best practices and memory optimization. |
-
-
+- **Direct Lake Semantic Models**: Create models directly in Power BI Desktop, and use data from multiple OneLake sources.
+- **Datapoint Annotations**: Add descriptive text to specific data points directly in PowerPoint presentations.
+- **Copilot Improvements**: Ask data questions from your model.
+- **Notebook Tools**: Quickly analyze semantic models with pre-built notebooks for best practices and memory optimization.
Total Visitors
From 8ffae520c30b4073b017c743faf744c9fd98ff5a Mon Sep 17 00:00:00 2001
From: "github-actions[bot]"
Date: Tue, 8 Apr 2025 04:21:23 +0000
Subject: [PATCH 31/31] Update last modified date in Markdown files
---
.../2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md | 2 +-
0_Azure/3_AzureAI/README.md | 2 +-
0_Azure/README.md | 2 +-
0_Azure/WhatsNew/0_IgniteNews_2024.md | 2 +-
0_Azure/WhatsNew/1_AIvisionMarch2025.md | 2 +-
0_Azure/WhatsNew/2_FabricConfVegas2025.md | 2 +-
0_Azure/WhatsNew/README.md | 2 +-
0_Azure/_industry-specific/0_Healthcare/README.md | 2 +-
8 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md b/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md
index 7038f82f8..21ea08db4 100644
--- a/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md
+++ b/0_Azure/2_AzureAnalytics/0_Fabric/demos/34_FromSynapse-Fabric.md
@@ -6,7 +6,7 @@
[](https://github.com/)
[brown9804](https://github.com/brown9804)
-Last updated: 2025-04-07
+Last updated: 2025-04-08
----------
diff --git a/0_Azure/3_AzureAI/README.md b/0_Azure/3_AzureAI/README.md
index 2e8505be9..aee7883bd 100644
--- a/0_Azure/3_AzureAI/README.md
+++ b/0_Azure/3_AzureAI/README.md
@@ -6,7 +6,7 @@ Costa Rica
[](https://github.com/)
[brown9804](https://github.com/brown9804)
-Last updated: 2025-04-07
+Last updated: 2025-04-08
----------
diff --git a/0_Azure/README.md b/0_Azure/README.md
index 8202788f1..a0278dc8e 100644
--- a/0_Azure/README.md
+++ b/0_Azure/README.md
@@ -5,7 +5,7 @@ Costa Rica
[](https://github.com/)
[brown9804](https://github.com/brown9804)
-Last updated: 2025-04-07
+Last updated: 2025-04-08
------------------------------------------
diff --git a/0_Azure/WhatsNew/0_IgniteNews_2024.md b/0_Azure/WhatsNew/0_IgniteNews_2024.md
index 8f51964d0..992c356eb 100644
--- a/0_Azure/WhatsNew/0_IgniteNews_2024.md
+++ b/0_Azure/WhatsNew/0_IgniteNews_2024.md
@@ -6,7 +6,7 @@ Costa Rica
[](https://github.com/)
[brown9804](https://github.com/brown9804)
-Last updated: 2025-04-07
+Last updated: 2025-04-08
----------
diff --git a/0_Azure/WhatsNew/1_AIvisionMarch2025.md b/0_Azure/WhatsNew/1_AIvisionMarch2025.md
index 8f2de46fc..214d119af 100644
--- a/0_Azure/WhatsNew/1_AIvisionMarch2025.md
+++ b/0_Azure/WhatsNew/1_AIvisionMarch2025.md
@@ -6,7 +6,7 @@ Costa Rica
[](https://github.com/)
[brown9804](https://github.com/brown9804)
-Last updated: 2025-04-07
+Last updated: 2025-04-08
----------
diff --git a/0_Azure/WhatsNew/2_FabricConfVegas2025.md b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
index e0e8b090c..6bda02500 100644
--- a/0_Azure/WhatsNew/2_FabricConfVegas2025.md
+++ b/0_Azure/WhatsNew/2_FabricConfVegas2025.md
@@ -6,7 +6,7 @@ Costa Rica
[](https://github.com/)
[brown9804](https://github.com/brown9804)
-Last updated: 2025-04-07
+Last updated: 2025-04-08
----------
diff --git a/0_Azure/WhatsNew/README.md b/0_Azure/WhatsNew/README.md
index 7946e25e1..fe760c205 100644
--- a/0_Azure/WhatsNew/README.md
+++ b/0_Azure/WhatsNew/README.md
@@ -5,7 +5,7 @@ Costa Rica
[](https://github.com/)
[brown9804](https://github.com/brown9804)
-Last updated: 2025-04-07
+Last updated: 2025-04-08
------------------------------------------
diff --git a/0_Azure/_industry-specific/0_Healthcare/README.md b/0_Azure/_industry-specific/0_Healthcare/README.md
index 492340811..f746d4097 100644
--- a/0_Azure/_industry-specific/0_Healthcare/README.md
+++ b/0_Azure/_industry-specific/0_Healthcare/README.md
@@ -6,7 +6,7 @@ Costa Rica
[](https://github.com/)
[brown9804](https://github.com/brown9804)
-Last updated: 2025-04-07
+Last updated: 2025-04-08
----------