Skip to content

Commit 9179638

Browse files
brown9804brown9804
authored
Merge 15bef2c into 106c93a 
2 parents 106c93a + 15bef2c commit 9179638

3 files changed

Lines changed: 126 additions & 1 deletion

File tree

0_Azure/2_AzureAnalytics/0_Fabric/demos/30_dynamic_pipeline_nbkparameters.md renamed to 0_Azure/2_AzureAnalytics/0_Fabric/demos/30_DynamicPipeline_nbkparametersADF.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Costa Rica
55
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
66
[brown9804](https://github.com/brown9804)
77

8-
Last updated: 2025-03-03
8+
Last updated: 2025-03-05
99

1010
----------
1111

‎0_Azure/2_AzureAnalytics/0_Fabric/demos/31_FabricActivatorRulePipeline/GeneratesRandomData.ipynb‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"cells":[{"cell_type":"code","source":["# Generates Dummy json file in Files/\n","\n","# Import necessary libraries\n","from pyspark.sql import SparkSession\n","from pyspark.sql.types import *\n","import random\n","from datetime import datetime, timedelta\n","\n","# Initialize Spark session (if not already initialized)\n","spark = SparkSession.builder.appName(\"GenerateRandomData\").getOrCreate()\n","\n","# Function to generate random data\n","def generate_random_data(num_entries):\n"," data = []\n"," for i in range(1, num_entries + 1):\n"," name = f\"User{i}\"\n"," entry = {\n"," \"id\": i,\n"," \"name\": name,\n"," \"age\": random.randint(18, 65),\n"," \"email\": f\"{name.lower()}@example.com\",\n"," \"created_at\": (datetime.now() - timedelta(days=random.randint(0, 365))).strftime(\"%Y-%m-%d %H:%M:%S\")\n"," }\n"," data.append(entry)\n"," return data\n","\n","# Generate 10 random entries\n","random_data = generate_random_data(10)\n","\n","# Define schema for the DataFrame\n","schema = StructType([\n"," StructField(\"id\", IntegerType(), True),\n"," StructField(\"name\", StringType(), True),\n"," StructField(\"age\", IntegerType(), True),\n"," StructField(\"email\", StringType(), True),\n"," StructField(\"created_at\", StringType(), True)\n","])\n","\n","# Create a DataFrame from the random data\n","df_random_data = spark.createDataFrame(random_data, schema=schema)\n","\n","# Write the DataFrame to the Lakehouse in the specified path\n","output_path = \"abfss://{WORKSPACE-NAME}@onelake.dfs.fabric.microsoft.com/raw_Bronze.Lakehouse/Files/random_data\" # Replace {WORKSPACE-NAME}\n","df_random_data.write.format(\"delta\").mode(\"overwrite\").save(output_path)\n","\n","print(f\"Random data has been saved to the Lakehouse at '{output_path}'.\")"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"8d820f25-3c2e-45b3-8a08-af78f0d45e1d"}],"metadata":{"kernel_info":{"name":"synapse_pyspark"},"kernelspec":{"name":"synapse_pyspark","language":"Python","display_name":"Synapse PySpark"},"language_info":{"name":"python"},"microsoft":{"language":"python","language_group":"synapse_pyspark","ms_spell_check":{"ms_spell_check_language":"en"}},"nteract":{"version":"nteract-front-end@1.0.0"},"spark_compute":{"compute_id":"/trident/default","session_options":{"conf":{"spark.synapse.nbs.session.timeout":"1200000"}}},"dependencies":{}},"nbformat":4,"nbformat_minor":5}

‎0_Azure/2_AzureAnalytics/0_Fabric/demos/31_FabricActivatorRulePipeline/README.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Microsoft Fabric: Automating Pipeline Execution with Activator
2+
3+
Costa Rica
4+
5+
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
6+
[brown9804](https://github.com/brown9804)
7+
8+
Last updated: 2025-03-05
9+
10+
----------
11+
12+
> This process shows how to set up Microsoft Fabric Activator to automate workflows by detecting file creation events in a storage system and triggering another pipeline to run.
13+
14+
<details>
15+
<summary><b>List of References </b> (Click to expand)</summary>
16+
17+
</details>
18+
19+
20+
## Set Up the First Pipeline
21+
22+
1. **Create the Pipeline**:
23+
- In [Microsoft Fabric](https://app.fabric.microsoft.com/), create the first pipeline that performs the required tasks.
24+
25+
> [!NOTE]
26+
> This code generates random data with fields such as id, name, age, email, and created_at, organizes it into a PySpark DataFrame, and saves it to a specified Lakehouse path using the Delta format.
27+
28+
https://github.com/user-attachments/assets/95206bf3-83a7-42c1-b501-4879df22ef7d
29+
30+
- Add a `Copy Data` activity as the final step in the pipeline.
31+
32+
2. **Generate the Trigger File**:
33+
- Configure the `Copy Data` activity to create a trigger file in a specific location, such as `Azure Data Lake Storage (ADLS)` or `OneLake`.
34+
- Ensure the file name and path are consistent and predictable (e.g., `trigger_file.json` in a specific folder).
35+
3. **Publish and Test**: Publish the pipeline and test it to ensure the trigger file is created successfully.
36+
37+
https://github.com/user-attachments/assets/798a3b12-c944-459d-9e77-0112b5d82831
38+
39+
## Configure Activator to Detect the Event
40+
41+
> [!TIP]
42+
> Event options:
43+
44+
https://github.com/user-attachments/assets/282fae9b-e1c6-490d-bd23-9ed9bdf6105d
45+
46+
1. **Set Up an Event**:
47+
- Create a new event to monitor the location where the trigger file is created (e.g., ADLS or OneLake). Click on `Real-Time`:
48+
49+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/e1ce1f83-a8f6-4a3c-94dc-749e370d8079" />
50+
51+
- Choose the appropriate event type, such as `File Created`.
52+
53+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/3a21abd7-0ff4-428f-a3a1-5e387314c1f5" />
54+
55+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/94e5556b-5d56-4a42-9edd-83b514e7c953" />
56+
57+
- Add a source:
58+
59+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/9709a690-f3b5-453b-b3d9-c67d4b1a9465" />
60+
61+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/8dcadd23-4abb-47ee-82ca-f3868cb818e1" />
62+
63+
https://github.com/user-attachments/assets/43a9654b-e8d0-44da-80b9-9f528483fa3b
64+
65+
2. **Test Event Detection**:
66+
- Save the event and test it by manually running the first pipeline to ensure Activator detects the file creation.
67+
- Check the **Event Details** screen in Activator to confirm the event is logged.
68+
69+
https://github.com/user-attachments/assets/6b21194c-54b4-49de-9294-1bf78b1e5acd
70+
71+
## Define the Rule in Activator
72+
73+
1. **Setup the Activator**:
74+
75+
https://github.com/user-attachments/assets/7c88e080-d5aa-4920-acd6-94c2e4ae0568
76+
77+
78+
2. **Create a New Rule**:
79+
- In `Activator`, create a rule that responds to the event you just configured.
80+
- Set the condition to match the event details (e.g., file name, path, or metadata).
81+
3. **Set the Action**:
82+
- Configure the rule to trigger the second pipeline.
83+
- Specify the pipeline name and pass any required parameters.
84+
85+
3. **Save and Activate**:
86+
- Save the rule and activate it.
87+
- Ensure the rule is enabled and ready to respond to the event.
88+
89+
90+
91+
92+
### **Step 4: Set Up the Second Pipeline**
93+
1. **Create the Pipeline**:
94+
- In Microsoft Fabric, create the second pipeline that performs the next set of tasks.
95+
- Ensure it is configured to accept external triggers.
96+
97+
2. **Publish the Pipeline**:
98+
- Publish the second pipeline and ensure it is ready to be triggered.
99+
100+
---
101+
102+
### **Step 5: Test the Entire Workflow**
103+
1. **Run the First Pipeline**:
104+
- Execute the first pipeline and verify that the trigger file is created.
105+
106+
2. **Monitor Activator**:
107+
- Check the **Event Details** and **Rule Activation Details** in Activator to ensure the event is detected and the rule is activated.
108+
109+
3. **Verify the Second Pipeline**:
110+
- Confirm that the second pipeline is triggered and runs successfully.
111+
112+
---
113+
114+
## Troubleshooting (If Needed)
115+
- If the second pipeline does not trigger:
116+
1. Double-check the rule configuration in Activator.
117+
2. Review the logs in Activator for any errors or warnings.
118+
119+
120+
<div align="center">
121+
<h3 style="color: #4CAF50;">Total Visitors</h3>
122+
<img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
123+
</div>
124+

0 commit comments

Comments
 (0)