overview

Author: brown9804

0_Azure/2_AzureAnalytics/0_Fabric/demos/28_MirroringERPperfAnalysis.md

@@ -11,6 +11,10 @@ Last updated: 2025-02-28
 
 > Quick technical overview of Mirroring an ERP System Database into Microsoft Fabric
 
+> [!NOTE]
+> `Shortcuts` in Microsoft Fabric are objects that point to other storage locations, either within OneLake or external sources like Azure Data Lake, Amazon S3, and Dataverse. They provide `real-time access` to data without physically copying it, reducing storage overhead and ensuring the latest data is always available. <br/>
+> `Mirroring` in Microsoft Fabric involves creating a synchronized copy of an external database within OneLake. This process `physically stores` the data in Fabric, enabling faster query performance and ensuring data consistency through continuous updates.
+
 <details>
 <summary><b>List of References</b> (Click to expand)</summary>
 
@@ -21,7 +25,10 @@ Last updated: 2025-02-28
 - [Permission model](https://learn.microsoft.com/en-us/fabric/security/permission-model)
 - [Secure data access in Microsoft Fabric](https://learn.microsoft.com/en-us/training/modules/secure-data-access-in-fabric/)
 - [Authorization in SQL database in Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/database/sql/authorization)
-
+- [What is the SQL analytics endpoint for a lakehouse? - Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-sql-analytics-endpoint)
+- [Query the SQL analytics endpoint of your SQL database in Fabric](https://learn.microsoft.com/en-us/fabric/database/sql/query-sql-analytics-endpoint)
+- [SQL database Overview (Preview) - Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/database/sql/overview)
+  
 </details>
 
 
@@ -38,6 +45,7 @@ Last updated: 2025-02-28
 | **Continuous Replication** | Microsoft Fabric continuously replicates data from your ERP system database into Fabric's OneLake. |
 | **Low Latency** | The replication process is designed to be low-latency, ensuring near real-time data availability for analytics. |
 | **Read-Only Access** | The mirrored data is stored in a read-only format, protecting the operational ERP database from performance degradation due to analytics queries. |
+| **Use Cases** | - **Analytics and Reporting:** Perform analytics and generate reports without impacting the operational ERP database. <br> - **Data Engineering:** Use the mirrored data for data engineering tasks, such as data transformation and integration. <br> - **Data Science:** Leverage the mirrored data for data science projects, including machine learning and AI. |
 
 ### Continuous Replication
 
@@ -106,6 +114,150 @@ graph TD
     E -->|Prevents modifications| J[Enhanced Security]
 ```
 
+### Use Cases
+```mermaid
+mindmap
+  root((Use Cases))
+    Analytics and Reporting
+      Benefits
+        Uninterrupted Operations
+        Timely Insights
+        Scalability
+      Examples
+        Financial Reporting
+        Sales Analysis
+        Operational Metrics
+    Data Engineering
+      Benefits
+        Data Transformation
+        Data Integration
+        Data Quality
+      Examples
+        ETL Processes
+        Data Enrichment
+        Data Aggregation
+    Data Science
+      Benefits
+        Predictive Analytics
+        Machine Learning
+        AI Applications
+      Examples
+        Customer Segmentation
+        Demand Forecasting
+        Anomaly Detection
+```
+#### Analytics and Reporting
+
+> The primary goal is to enable business users and analysts to perform data analysis and generate reports without affecting the performance of the operational ERP database.
+
+**Benefits:**
+- **Uninterrupted Operations:** Since the mirrored data is read-only, analytics queries do not interfere with the day-to-day operations of the ERP system.
+- **Timely Insights:** Near real-time replication ensures that the data used for analytics is up-to-date, allowing for timely and accurate decision-making.
+- **Scalability:** The system can handle large volumes of data, making it suitable for enterprises with extensive reporting needs.
+
+> Examples:
+
+| **Example** | **Description** |
+|-------------|-----------------|
+| **Financial Reporting** | Generate financial statements, balance sheets, and profit & loss reports. |
+| **Sales Analysis** | Analyze sales trends, customer behavior, and product performance. |
+| **Operational Metrics** | Monitor key performance indicators (KPIs) such as inventory levels, order fulfillment rates, and production efficiency. |
+
+#### Data Engineering
+
+> To facilitate data transformation, integration, and preparation tasks that are essential for building data pipelines and workflows.
+
+> Benefits:
+  - **Data Transformation:** Convert raw data into a structured format suitable for analysis and reporting.
+  - **Data Integration:** Combine data from multiple sources, including the ERP system, to create a unified data repository.
+  - **Data Quality:** Implement data cleansing and validation processes to ensure the accuracy and reliability of the data.
+
+> Examples:
+
+| **Example** | **Description** |
+|-------------|-----------------|
+| **ETL Processes** | Extract, transform, and load data from the ERP system into a data warehouse or data lake. |
+| **Data Enrichment** | Enhance the ERP data with additional information from external sources, such as market data or customer feedback. |
+| **Data Aggregation** | Summarize and aggregate data to create high-level views and dashboards. |
+
+
+#### **Data Science:**
+- **Purpose:** To leverage the mirrored data for advanced analytics, machine learning, and artificial intelligence projects.
+> Benefits:
+  - **Predictive Analytics:** Use historical data to build models that predict future trends and outcomes.
+  - **Machine Learning:** Train machine learning models on the mirrored data to automate decision-making processes and improve business operations.
+  - **AI Applications:** Develop AI-driven applications that can provide insights, recommendations, and automation.
+
+> Examples:
+
+| **Example** | **Description** |
+|-------------|-----------------|
+| **Customer Segmentation** | Use clustering algorithms to segment customers based on their behavior and preferences. |
+| **Demand Forecasting** | Predict future demand for products and services using time series analysis. |
+| **Anomaly Detection** | Identify unusual patterns or outliers in the data that may indicate fraud, errors, or opportunities. |
+
+## Mirroring Process
+
+```mermaid
+mindmap
+  root((Mirroring Process))
+    Querying Mirrored Data
+    Data Storage
+```
+
+### Data Storage
+
+```mermaid
+graph TD
+    A[SQL Database in Fabric] -->|Stores Data| B[.mdf Files]
+    A -->|Mirrors Data| C[OneLake]
+    C -->|Stores Data| D[Delta Parquet Files]
+    D -->|Optimized for| E[Analytical Queries]
+
+    subgraph Data Storage
+        A --> B
+        A --> C
+        C --> D
+        D --> E
+    end
+```
+
+- **Mirrored Data Storage:** The mirrored data is stored as delta parquet files in OneLake. Parquet is a columnar storage file format optimized for analytical queries, providing efficient data compression and encoding schemes.
+- **Delta Format:** The delta format supports ACID transactions and allows for efficient data updates and deletions. This ensures that the mirrored data remains consistent and reliable.
+- **Storage Management:** OneLake provides a unified storage solution that integrates with various Fabric services, such as Spark, Power BI, and data engineering tools. This integration allows for seamless data management and analytics.
+
+
+### Querying Mirrored Data
+
+```mermaid
+mindmap
+  root((Querying Mirrored Data))
+    SQL Analytics Endpoint
+      Queries Data
+        Delta Parquet Files in OneLake
+          Provides Data
+            Reporting and Analytics
+              Read-Only Mode
+                Protects Operational Data
+              Creates Views
+                Data Presentation
+              Joins Tables
+                Cross-Database Queries
+              Access Tools
+                SSMS
+                VS Code
+                Fabric Portal
+```
+
+- **SQL Analytics Endpoint:** The SQL Analytics Endpoint provides a SQL-based experience for querying mirrored data stored as delta parquet files in OneLake. It allows you to run reporting and analytics queries without impacting the primary workload of the operational ERP database.
+- **Read-Only Mode:** The endpoint operates in read-only mode, ensuring that analytics queries do not modify the mirrored data. This protects the integrity of the operational data and prevents any performance degradation.
+- **Creating Views:** You can create views in your SQL analytics endpoint to shape the data presentation. Views allow you to define how the data should be displayed and can include complex joins, aggregations, and calculations.
+- **Joining Tables:** The SQL analytics endpoint supports joining mirrored tables with other tables in different warehouses or lakehouses in the workspace. This enables you to perform cross-database queries and integrate data from multiple sources for comprehensive analysis.
+- **Access and Querying Tools:** You can access the SQL Analytics Endpoint through various tools such as SQL Server Management Studio, Visual Studio Code, and the Fabric portal. It supports T-SQL language, allowing you to create views, save functions, and apply SQL security.
+
+
+
+