Skip to content

Commit d82b360

Browse files
brown9804brown9804
authored
Merge pull request #84 from MicrosoftCloudEssentials-LearningHub/pipeline-mdf-adf
+ 29_ADF_MonitorChanges.md
2 parents f12ce6c + e49a8db commit d82b360

1 file changed

Lines changed: 118 additions & 0 deletions

File tree

‎0_Azure/2_AzureAnalytics/0_Fabric/demos/29_ADF_MonitorChanges.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# Azure Data Factory: Quick Guide on How to Track Pipeline Modifications
2+
3+
Costa Rica
4+
5+
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
6+
[brown9804](https://github.com/brown9804)
7+
8+
Last updated: 2025-03-03
9+
10+
----------
11+
12+
<details>
13+
<summary><b>List of References </b> (Click to expand)</summary>
14+
15+
- [What is Azure Data Factory?](https://learn.microsoft.com/en-us/azure/data-factory/introduction)
16+
- [Quickstart: Get started with Azure Data Factory](https://learn.microsoft.com/en-us/azure/data-factory/quickstart-get-started)
17+
- [Quickstart: Create a data factory by using the Azure portal](https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory)
18+
19+
</details>
20+
21+
22+
## How to create a Data Factory in Azure
23+
24+
1. **Log in to Azure Portal**: Open your web browser and go to the Azure Portal. Enter your credentials to log in.
25+
2. **Search for Data Factory**: Use the search bar at the top to search for `Data Factory` and select `Data Factory` from the results.
26+
27+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/78b857da-1550-41f4-80bd-ea81fdafc24c" />
28+
29+
3. **Create a New Data Factory**:
30+
- Click on the `+ Create` button.
31+
- In the "Basics" tab, fill in the required fields:
32+
- **Subscription**: Select your Azure subscription.
33+
- **Resource Group**: Select an existing resource group or create a new one.
34+
- **Region**: Choose the region where you want to deploy the Data Factory.
35+
- **Name**: Enter a unique name for your Data Factory.
36+
- **Version**: Select V2 (the latest version).
37+
4. **Configure Git (Optional)**: If you want to configure Git for source control, you can do so in the `Git configuration` tab. This step is optional and can be skipped if not needed.
38+
5. **Review and Create**:
39+
- Click on the `Review + create` button.
40+
- Review your settings and click `Create` once the validation passes.
41+
42+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/3dbdc4a4-2a41-487e-9a2a-628921381dfe" />
43+
44+
6. **Wait for Deployment**: The deployment process will take a few minutes. Once it's complete, you will see a notification.
45+
7. **Access Data Factory**: After the deployment is complete, click on the `Go to resource` button to access your new Data Factory.
46+
8. **Launch Data Factory Studio**: In the Data Factory resource page, click on the `Launch Studio` tile to launch the Data Factory Studio where you can start creating pipelines and other data integration tasks.
47+
48+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/2fae2d31-54d0-40f0-adab-111f5b464cab" />
49+
50+
## Create a pipeline
51+
52+
1. **Log in to Azure Portal**: Open your web browser and go to the Azure Portal. Enter your credentials to log in.
53+
2. **Go to Data Factory**: Use the search bar at the top to search for `Data Factory` and select your Data Factory instance from the list.
54+
3. **Launch Data Factory Studio**: In the Data Factory resource page, click on the `Launch Studio` tile to launch the Data Factory Studio where you can start creating pipelines and other data integration tasks.
55+
56+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/2fae2d31-54d0-40f0-adab-111f5b464cab" />
57+
58+
4. **Create a New Pipeline**:
59+
- Click on the `New` next to `Pipelines` in the tree view.
60+
- Select `Pipeline` from the dropdown menu.
61+
62+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/5f112ab1-5327-49d9-bce6-b8ee98f23267" />
63+
64+
5. **Add Activities to the Pipeline**:
65+
- In the pipeline canvas, click on the `Activities` pane on the left.
66+
- Drag and drop the desired activities (e.g., Copy Data, Data Flow) onto the pipeline canvas.
67+
68+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/fd21dfaf-8e3f-4265-8ff7-1f44098b4827" />
69+
70+
6. **Configure Activities**:
71+
- Click on each activity on the canvas to configure its properties.
72+
- For example, if you are using a Copy Data activity, you will need to specify the source and destination datasets.
73+
74+
7. **Set Up Linked Services**:
75+
- Linked services are used to define the connection information for data sources and destinations.
76+
- Go to the `Manage` tab on the left, then click on `Linked services`.
77+
- Click on the **+ New** button to create a new linked service and configure the connection details.
78+
79+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/a6ca86d0-754d-4b1d-bd27-8f45da00fbe0" />
80+
81+
8. **Create Datasets**:
82+
- Datasets represent the data structures within the data stores.
83+
- Go to the `Author` tab, then click on `Datasets`.
84+
- Click on the `+ (plus) icon` to create a new dataset and configure its properties.
85+
86+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/56ef28cb-8e57-4e70-8322-7ad6777bb886" />
87+
88+
9. **Validate the Pipeline**: Click on the `Validate` button at the top of the pipeline canvas to check for any errors or missing configurations.
89+
10. **Publish the Pipeline**: Once validation is successful, click on the `Publish All` button to save and publish your pipeline.
90+
11. **Trigger the Pipeline**: Click on `Trigger now` to run the pipeline immediately, or configure a trigger for scheduled runs.
91+
92+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/0c42c3e3-57a8-4111-add4-1ea7192b6ac8" />
93+
94+
12. **Monitor Pipeline Runs**: In the `Monitor` tab, you can view the status of pipeline runs, check for any errors, and review the execution details.
95+
96+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/11922b16-2d9e-49d8-b0ab-13605a18018f" />
97+
98+
## How to see who modified a pipeline
99+
100+
1. **Log in to Azure Portal**: Open your web browser and go to the Azure Portal. Enter your credentials to log in.
101+
2. **Go to Azure Data Factory**: Once logged in, use the search bar at the top to search for `Data Factory` and select your Data Factory instance from the list.
102+
3. **Open the Activity Log**:
103+
- In the Data Factory resource page, look for the `Activity log` option in the left-hand menu under the `Monitoring` section.
104+
- Click on `Activity log` to open the log view.
105+
4. **View Activity Log Details**:
106+
- In the Activity Log, you will see a list of events related to your Data Factory.
107+
- You can see columns such as `Operation Name`, `Status`, `Event Initiated By`, `Time`,`Subscription`, and more.
108+
5. **Filter and Search**:
109+
- Use the filters at the top to narrow down the events by time range, resource group, resource, and more.
110+
- You can also use the search bar to find specific events or operations.
111+
6. **Review Event Details**: Click on any event in the list to view more detailed information about that event, including the JSON payload with additional properties.
112+
113+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/07cf4582-6b7b-451e-94e9-9557cdbfd09f">
114+
115+
<div align="center">
116+
<h3 style="color: #4CAF50;">Total Visitors</h3>
117+
<img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
118+
</div>

0 commit comments

Comments
 (0)