Merge pull request #81 from MicrosoftCloudEssentials-LearningHub/fabric-databricks

brown9804 · web-flow · commit 9dd8e2881e2b · 2025-02-13T15:19:01.000-06:00
Fabric + Databricks
diff --git a/0_Azure/1_AzureData/README.md b/0_Azure/1_AzureData/README.md
@@ -5,25 +5,17 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-19
+Last updated: 2025-02-13
 
 ----------
 
 ## Content 
 
 - [Data Storage](./0_DataStorage/)
 - [Databases](./1_Databases/)
-- [Azure Databricks](./2_Databricks)
 
 Azure Data Storage provides scalable, secure, and accessible cloud storage, ideal for big data and analytics, with various storage tiers. It supports a wide range of services and tools. Azure also offers relational and non-relational databases, with built-in management for high availability and performance, catering to different application needs.
 
-
-| Area | Category | Service | Overview |
-| ---- | ---- | ---- | ---- | 
-| Big Data Analytics | Service | [Azure Databricks](https://azure.microsoft.com/en-us/products/databricks/) | Azure Databricks is a fast, easy, and collaborative Apache Spark-based analytics platform optimized for Azure. It provides an interactive workspace for data engineers, data scientists, and business analysts. <br/> <br/> For more information: <br/> [Azure Databricks Overview](https://azure.microsoft.com/en-us/products/databricks/) <br/> [What is Azure Databricks?](https://learn.microsoft.com/en-us/azure/databricks/scenarios/what-is-azure-databricks) <br/> [Azure Databricks Learning documents](https://learn.microsoft.com/en-us/azure/databricks/). |
-| Data Integration | Service | [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/) | Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale. <br/> <br/> For more information: <br/> [Azure Data Factory Overview](https://azure.microsoft.com/en-us/products/data-factory/) <br/> [What is Azure Data Factory?](https://learn.microsoft.com/en-us/azure/data-factory/introduction) <br/> [Azure Data Factory Learning documents](https://learn.microsoft.com/en-us/azure/data-factory/). |
-
-
 ## Differences between Azure Data Storage and Databases
 
 Azure Data Storage and Databases both persist data but are optimized for different purposes. Storage provides durable capacity while databases structure data for efficient access. Storage suits long-term file retention while databases enable interactive applications.
@@ -41,8 +33,6 @@ Image from [here](https://www.edureka.co/blog/azure-storage-tutorial/)
 | Durable long-term retention | Temporary persistence tier |
 
 
-
-
 ## DataFrames types: 
 
 Comparative analysis of various types of DataFrames. Each type of DataFrame has its unique features and is suited for different use cases. The table below summarizes the key characteristics and common applications of each type:
@@ -62,4 +52,4 @@ Comparative analysis of various types of DataFrames. Each type of DataFrame has
 <div align="center">
   <h3 style="color: #4CAF50;">Total Visitors</h3>
   <img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
-</div>
+</div>
diff --git a/0_Azure/2_AzureAnalytics/0_Fabric/demos/14_PbiManageAccess/README.md b/0_Azure/2_AzureAnalytics/0_Fabric/demos/14_PbiManageAccess/README.md
@@ -5,11 +5,11 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-19
+Last updated: 2025-02-13
 
 ------------------------------------------
 
-## Wiki 
+## References 
 
 <details>
 <summary><b>Table of References </b> (Click to expand)</summary>
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/0_ADLS/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/0_ADLS/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/1_Synapse/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/1_Synapse/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/2_PowerBI/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/2_PowerBI/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/3_CosmosDB/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/3_CosmosDB/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/4_DataFactory/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/4_DataFactory/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/5_EventHubs/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/5_EventHubs/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/6_AzureML/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/6_AzureML/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/7_SQL-DBs/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/7_SQL-DBs/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/8_Fabric/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/8_Fabric/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/0_Connections/README.md
@@ -5,7 +5,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/0_MedallionArch_Fabric+Databricks/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/0_MedallionArch_Fabric+Databricks/README.md
@@ -0,0 +1,58 @@
+# Fabric + Databricks: Medallion Architecture 
+
+Costa Rica
+
+[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
+[brown9804](https://github.com/brown9804)
+
+Last updated: 2025-02-13
+
+------------------------------------------
+
+## References 
+
+<details>
+<summary><b>List of References </b> (Click to expand)</summary>
+
+- [TechExcel: Microsoft Fabric with Azure Databricks for Data Analytics (lvl 300 / CSU) lab](https://microsoft.github.io/TechExcel-Fabric-with-Databricks-for-Data-Analytics/)
+- [Databricks Unity Catalog tables available in Microsoft Fabric](https://blog.fabric.microsoft.com/en-us/blog/databricks-unity-catalog-tables-available-in-microsoft-fabric/)
+- [Integrate OneLake with Azure Databricks](https://learn.microsoft.com/en-us/fabric/onelake/onelake-azure-databricks)
+- [Tutorial: Configure Microsoft Fabric mirrored databases from Azure Databricks (Preview)](https://learn.microsoft.com/en-us/fabric/database/mirrored-database/azure-databricks-tutorial)
+- [Integrating Microsoft Fabric with Azure Databricks Delta Tables](https://techcommunity.microsoft.com/blog/fasttrackforazureblog/integrating-microsoft-fabric-with-azure-databricks-delta-tables/3916332)
+- [Data Intelligence End-to-End with Azure Databricks and Microsoft Fabric](https://techcommunity.microsoft.com/blog/azurearchitectureblog/data-intelligence-end-to-end-with-azure-databricks-and-microsoft-fabric/4232621)
+
+</details>
+
+## Overview 
+
+| Resource   | Key Components           | Details                                                                 |
+|------------|--------------------------|-------------------------------------------------------------------------|
+| Databricks | - ETL Tasks<br/> - Data Ingestion              | ETL Tasks: Databricks efficiently handles Extract, Transform, Load (ETL) tasks. It manages the bronze, silver, and gold layers of the medallion architecture: <br> - **Bronze Layer**: Raw data ingestion from various sources. <br> - **Silver Layer**: Data cleaning and transformation. <br> - **Gold Layer**: Aggregated and refined data ready for analysis. <br> Data Ingestion: Supports batch and streaming data ingestion, ensuring real-time data processing capabilities. <br> - **Data Cleaning**: Utilizes Apache Spark's powerful processing engine to clean and transform data at scale. <br> - **Aggregation**: Performs complex aggregations and computations, making data ready for downstream analytics. |
+| Fabric     | - Data Integration <br> - Orchestration <br> - Monitoring and Management | Fabric seamlessly integrates with Databricks, providing a unified interface for managing data workflows. <br> - **Data Integration**: Facilitates the orchestration of data pipelines, ensuring smooth data flow between different stages of processing. <br> - **Monitoring and Management**: Offers robust monitoring and management tools to track data pipeline performance and troubleshoot issues. |
+
+> Here is a [reference of a medallion architecture using only Fabric](https://github.com/MicrosoftCloudEssentials-LearningHub/MS-Fabric-Essentials-Workshop/tree/main/AzurePortal/1_MedallionArch):
+
+<p align="center">
+  <img width="650" alt="image" src="https://github.com/user-attachments/assets/15a7dbfe-524b-4aa9-9b45-3db6bca2dd03" />
+</p>
+
+> If you need to handle `complex data transformations and large-scale data processing`, you can use our combined solution of **Fabric + Databricks**. This powerful combination leverages the strengths of both platforms to provide a robust data processing pipeline. This workshop on [Fabric with Databricks for Data Analytics](https://microsoft.github.io/TechExcel-Fabric-with-Databricks-for-Data-Analytics/) offers a comprehensive step-by-step guide on developing Medallion Architecture using Fabric and Databricks. <br/>
+
+<p align="center">
+  <img width="650" alt="image" src="https://github.com/user-attachments/assets/58431d3b-e294-46fe-89a4-92a046168ec4" />
+</p>
+
+## Integration with Multiple Sources:
+
+> In these two examples of Medallion Archuitecture, `Azure SQL Database` was used as the input source, but the solution is highly flexible and can integrate with multiple data sources, including:
+
+- **Cloud Storage**: Azure Blob Storage, AWS S3, Google Cloud Storage.
+- **Databases**: SQL Server, MySQL, PostgreSQL, Oracle.
+- **Streaming Sources**: Kafka, Event Hubs, IoT Hub.
+- **APIs and Web Services**: REST APIs, SOAP services.
+
+
+<div align="center">
+  <h3 style="color: #4CAF50;">Total Visitors</h3>
+  <img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
+</div>
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/0_MedallionArch_Fabric+Databricks/_docs/MedallionArch-Fabric_Databricks.drawio.png b/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/0_MedallionArch_Fabric+Databricks/_docs/MedallionArch-Fabric_Databricks.drawio.png
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/MSLearn_ExploratoryDataAnalysis.ipynb b/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/MSLearn_ExploratoryDataAnalysis.ipynb
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/1_demos/README.md
@@ -5,15 +5,10 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
-<div align="center">
-  <h3 style="color: #4CAF50;">Total Visitors</h3>
-  <img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
-</div>
-
 > This repository contains a collection of demos showcasing the capabilities of Azure Databricks. Each demo is crafted to highlight specific features and use cases.
 
 ## Prerequisites
@@ -30,3 +25,7 @@ Before you begin, ensure you have the following:
 
 Follow the detailed instructions provided in each demo/showcase.
 
+<div align="center">
+  <h3 style="color: #4CAF50;">Total Visitors</h3>
+  <img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
+</div>
diff --git a/0_Azure/2_AzureAnalytics/3_Databricks/README.md b/0_Azure/2_AzureAnalytics/3_Databricks/README.md
@@ -5,12 +5,12 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-15
+Last updated: 2025-02-13
 
 ----------
 
+## References 
 
-## Wiki 
 - [What is Azure Databricks?](https://learn.microsoft.com/en-us/azure/databricks/introduction/)
 - [Tutorial: Implement Azure Databricks with an Azure Cosmos DB endpoint](https://learn.microsoft.com/en-us/azure/databricks/scenarios/service-endpoint-cosmosdb)
 - [Query databases using JDBC](https://learn.microsoft.com/en-us/azure/databricks/connect/external-systems/jdbc)
diff --git a/0_Azure/2_AzureAnalytics/README.md b/0_Azure/2_AzureAnalytics/README.md
@@ -6,7 +6,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-19
+Last updated: 2025-02-13
 
 ----------
 
@@ -15,6 +15,7 @@ Last updated: 2024-11-19
 - [Microsoft Fabric](./0_Fabric/README.md)
 - [Azure Synapse Analytics](./1_SynapseAnalytics/README.md)
 - [Azure Event Hubs](./2_EventHubs)
+- [Azure Databricks](./3_Databricks)
   
 ## Wiki
 
@@ -24,8 +25,10 @@ Last updated: 2024-11-19
 | Real-time Data Processing | Service | [Azure Stream Analytics](https://azure.microsoft.com/en-us/products/stream-analytics/) | Azure Stream Analytics is a real-time analytics and complex event-processing engine that is designed to analyze and process high volumes of fast streaming data from multiple sources simultaneously. <br/> <br/> For more information: <br/> [Azure Stream Analytics Overview](https://azure.microsoft.com/en-us/products/stream-analytics/) <br/> [What is Azure Stream Analytics?](https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-introduction) <br/> [Azure Stream Analytics Learning documents](https://learn.microsoft.com/en-us/azure/stream-analytics/). |
 | Unified Analytics Platform | Service | [Microsoft Fabric](https://azure.microsoft.com/en-us/blog/introducing-microsoft-fabric-data-analytics-for-the-era-of-ai/) | Microsoft Fabric is an end-to-end, unified analytics platform that integrates technologies like Azure Data Factory, Azure Synapse Analytics, and Power BI into a single product. It covers data movement, processing, ingestion, transformation, real-time event routing, and report building, simplifying analytics requirements for enterprises. <br/> <br/> For more information: <br/> [Microsoft Fabric Overview](https://azure.microsoft.com/en-us/blog/introducing-microsoft-fabric-data-analytics-for-the-era-of-ai/) <br/> [What is Microsoft Fabric?](https://learn.microsoft.com/en-us/fabric/get-started/microsoft-fabric-overview) <br/> [Microsoft Fabric Learning documents](https://learn.microsoft.com/en-us/fabric/). |
 | Real-time Data Streaming | Service | [Azure Event Hubs](./2_EventHubs/) | Azure Event Hubs is a fully managed, real-time data streaming platform that can ingest and process millions of events per second with low latency. It supports multiple protocols, including Apache Kafka, and integrates with various Azure services for building comprehensive data streaming solutions. <br/> <br/> For more information: <br/> [Azure Event Hubs Overview](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about) <br/> [What is Azure Event Hubs?](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about) <br/> Azure Event Hubs Learning documents. |
+| Big Data Analytics | Service | [Azure Databricks](https://azure.microsoft.com/en-us/products/databricks/) | Azure Databricks is a fast, easy, and collaborative Apache Spark-based analytics platform optimized for Azure. It provides an interactive workspace for data engineers, data scientists, and business analysts. <br/> <br/> For more information: <br/> [Azure Databricks Overview](https://azure.microsoft.com/en-us/products/databricks/) <br/> [What is Azure Databricks?](https://learn.microsoft.com/en-us/azure/databricks/scenarios/what-is-azure-databricks) <br/> [Azure Databricks Learning documents](https://learn.microsoft.com/en-us/azure/databricks/). |
+| Data Integration | Service | [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/) | Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale. <br/> <br/> For more information: <br/> [Azure Data Factory Overview](https://azure.microsoft.com/en-us/products/data-factory/) <br/> [What is Azure Data Factory?](https://learn.microsoft.com/en-us/azure/data-factory/introduction) <br/> [Azure Data Factory Learning documents](https://learn.microsoft.com/en-us/azure/data-factory/). |
 
 <div align="center">
   <h3 style="color: #4CAF50;">Total Visitors</h3>
   <img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
-</div>
+</div>
diff --git a/0_Azure/3_AzureAI/9_AzureOpenAI/README.md b/0_Azure/3_AzureAI/9_AzureOpenAI/README.md
@@ -7,7 +7,7 @@ Costa Rica
 [![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
 [brown9804](https://github.com/brown9804)
 
-Last updated: 2024-11-19
+Last updated: 2025-02-13
 
 ----------
 
@@ -17,21 +17,28 @@ Key Features:
 - **Models Available**: The models available through Azure OpenAI Service include GPT-4, GPT-3.5-Turbo, and Embeddings.
 -  **Security and Compliance**: Azure OpenAI Service offers the security and enterprise promise of Azure.
 
-## Content 
-
-<!-- TOC -->
-
-- [Azure OpenAI](#azure-openai)
-    - [Content](#content)
-    - [Example of Use Cases](#example-of-use-cases)
-        - [Text Generation - Azure OpenAI:](#text-generation---azure-openai)
-        - [Sentiment Analysis - Azure OpenAI:](#sentiment-analysis---azure-openai)
-        - [Language Translation - Azure OpenAI:](#language-translation---azure-openai)
-        - [Content Moderation - Azure OpenAI:](#content-moderation---azure-openai)
-        - [Speech Recognition - Azure OpenAI:](#speech-recognition---azure-openai)
-    - [Demos & Tech Talks](./demos/)
-        - [Optimizing Chatbot Efficiency](./demos/1_OptimizingChatbotEfficiency.md)
-<!-- /TOC -->
+
+<details>
+<summary><b>List of References </b> (Click to expand)</summary>
+
+- [Azure Open AI resource - Change log](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/openai/Azure.AI.OpenAI/CHANGELOG.md)
+
+</details>
+
+<details>
+<summary><b>Table of Content </b> (Click to expand)</summary>
+
+- [Example of Use Cases](#example-of-use-cases)
+    - [Text Generation - Azure OpenAI:](#text-generation---azure-openai)
+    - [Sentiment Analysis - Azure OpenAI:](#sentiment-analysis---azure-openai)
+    - [Language Translation - Azure OpenAI:](#language-translation---azure-openai)
+    - [Content Moderation - Azure OpenAI:](#content-moderation---azure-openai)
+    - [Speech Recognition - Azure OpenAI:](#speech-recognition---azure-openai)
+- [Demos & Tech Talks](./demos/)
+    - [Optimizing Chatbot Efficiency](./demos/1_OptimizingChatbotEfficiency.md)
+      
+</details>
+
 
 ## Example of Use Cases
 
