Merge pull request #94 from MicrosoftCloudEssentials-LearningHub/purviewMetadata

brown9804 · web-flow · commit bb69c1c39635 · 2025-04-01T03:08:07.000-06:00
medata purview estimation
diff --git a/0_Azure/5_DataProtectionMng/0_Purview/demos/2_PurviewCostEstimation.md b/0_Azure/5_DataProtectionMng/0_Purview/demos/2_PurviewCostEstimation.md
@@ -28,9 +28,8 @@ Estimating the cost for Azure Purview requires consideration of the following co
 | Managed Virtual Network Charges | Customers using the latest version of Microsoft Purview Managed Virtual Network will be charged at 1/8 vCore hour for the running time of the Managed VNet Integration Runtime, in addition to the charges on scan and ingestion jobs | 
 | Data Transfers and API Calls |  Customers using Microsoft Purview to govern data in other clouds (e.g., AWS, GCP) may incur additional charges due to data transfers and API calls associated with the publishing of metadata into the Microsoft Purview Data Map. This charge varies by region | 
 
-
-The general formula to keep in mind for estimating the cost of Microsoft Purview is:
-
+> [!IMPORTANT]
+> The general formula to keep in mind for estimating the cost of Microsoft Purview is: <br/> 
 > - **Cost of Data Map**: Calculated based on the number of capacity units and the price per capacity unit per hour. <br/>
 > - **Cost of Scanning**: Calculated based on the total duration (in minutes) of all scans in a month, divided by 60 minutes per hour, multiplied by the number of vCores per scan, and the price per vCore per hour. <br/>
 > - **Cost of Resource Set**: Calculated based on the total duration (in hours) of processing resource set data assets in a month, multiplied by the price per vCore per hour.
@@ -104,15 +103,40 @@ $$
 \$299.03 + \$201.60 + \$10.50 = \$511.13
 $$
 
+> [!NOTE]
+> To estimate the number of data assets being analyzed in the given example, we need to consider the total duration of scans and the processing time for resource sets. `The relationship between the amount of data and the scanning time can vary based on the complexity and size of the data assets.` Actual numbers may vary based on specific data characteristics and processing requirements. `For precise estimation, it is recommended to use detailed performance metrics and data characteristics.`
+
 ## Average Scan Frequency for Different Use Cases
 
 | **Scenario**                           | **Description**                                                                                     | **Frequency**            |
 |----------------------------------------|-----------------------------------------------------------------------------------------------------|--------------------------|
-| **Compliance and Regulatory Requirements** | Organizations that need to comply with strict regulatory requirements may perform daily or weekly scans to ensure data is up-to-date and compliant. | Daily or Weekly          |
-| **Data Governance and Management**     | For general data governance and management, organizations may perform weekly or bi-weekly scans to keep track of data changes and maintain data quality. | Weekly or Bi-weekly      |
-| **Data Analytics and Reporting**       | Organizations that rely heavily on data analytics and reporting may perform monthly scans to ensure that the data used for analysis is accurate and up-to-date. | Monthly                  |
+| **Compliance and Regulatory Requirements** | Organizations that need to comply with strict regulatory requirements may perform daily or weekly scans to ensure data is up-to-date and compliant. | Daily or Weekly|
+| **Data Governance and Management**     | For general data governance and management, organizations may perform weekly or bi-weekly scans to keep track of data changes and maintain data quality. | Weekly or Bi-weekly|
+| **Data Analytics and Reporting**       | Organizations that rely heavily on data analytics and reporting may perform monthly scans to ensure that the data used for analysis is accurate and up-to-date. | Monthly|
 | **Ad-hoc Scans**                       | In some cases, organizations may perform ad-hoc scans as needed, based on specific events or requirements. | As Needed                |
 
+## Cost Estimation for Different Metadata Volumes
+
+> [!IMPORTANT]
+>  Microsoft Purview `scans metadata to classify, label, and protect data asset`s. It does `not scan the actual data content but rather the information about the data`. <br/>
+>  `The size of the data itself does not directly` impact the cost of `metadata scanning unless it affects the amount of metadata generated`. The `number of metadata assets and their complexity` are the primary factors influencing costs.
+
+Assumptions: 
+- The number of metadata assets is assumed based on the data volume, with an average size of 1 MB per metadata asset. 
+- The average size of each metadata asset is assumed to be 1 MB. 
+- These estimates are based on the assumption that the governed assets and data management costs are applied for 100 hours per month. Actual costs may vary based on specific agreements with Microsoft, usage patterns, etc. 
+
+| **Data Volume** | **Total Minutes of Scanning** | **Assumed Number of Metadata Assets** | **Average Size per Metadata Asset** | **Total Hours of Processing** | **Total Cost for Data Map** | **Total Cost for Scanning** | **Total Cost for Resource Set** | **Total Monthly Cost** |
+|-----------------|------------------|------------------|----------------|-----------------------|-----------------------------|-----------------------------|-------------------------------|-------------------------|
+| 1 GB            | 5 minutes | 1000 | 1 MB  | 0.1 hour  | $41.10 | $1.67                       | $0.02                         | $42.79                  |
+| 30 GB           | 30 minutes  | 30000  | 1 MB  | 3 hours  | $41.10 | $10.08                      | $0.63                         | $51.81                  |
+| 50 GB   | 50 minutes   | 50000   | 1 MB   | 5 hours   | $41.10   | $16.73                      | $1.05                         | $58.88                  |
+| 100 GB  | 100 minutes   | 100000  | 1 MB  | 10 hours  | $41.10 | $33.47                      | $2.10                         | $76.67                  |
+| 300 GB  | 300 minutes   | 300000   | 1 MB   | 30 hours  | $41.10 | $100.80                     | $6.30                         | $148.20                 |
+
+> [!NOTE]
+> In the case of processing 1 GB of data, the cost structure is primarily influenced by the time consumed by the system to handle the data rather than the actual volume of the data itself. For instance, scanning 1 GB of data takes 5 minutes, and processing it takes 0.1 hours. These time durations directly impact the costs associated with scanning and processing. Some costs, such as the `Total Cost for Data Map`, are fixed and `remain constant regardless of the data volume`, while other costs, like the `Total Cost for Scanning` and `Total Cost for Resource Set`, `vary based on the time taken to process the data`. For example, scanning 1 GB costs $1.67, which is calculated based on the 5 minutes of scanning time. The overall monthly cost is a summation of all these costs, including both fixed and variable components. Therefore, even if the data volume is small, if the system takes longer to process it, the costs could be higher. Efficient processing can reduce costs even for larger volumes of data. `This highlights that the cost is more related to the system's handling time rather than the amount of data being processed.`
+
 ## Additional Considerations
 
 - **Optimize Scan Frequency**: Reduce the frequency of scans to lower the overall cost by customizing Scan Rule Sets to control cost.This enable sto fine-tune the time scans take. Find below the steps to create and customize scan rule sets:
