Skip to content

Commit 644882c

Browse files
brown9804brown9804
authored
Merge 7d51ecb into 871cabd 
2 parents 871cabd + 7d51ecb commit 644882c

10 files changed

Lines changed: 304 additions & 10 deletions

File tree

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/0_serverlessSQLPool/README.md‎

Lines changed: 212 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,27 +6,232 @@ Costa Rica
66
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
77
[brown9804](https://github.com/brown9804)
88

9-
Last updated: 2025-01-24
9+
Last updated: 2025-01-26
1010

1111
----------
1212

13-
## Wiki
13+
## Content
1414

1515
<details>
16-
<summary><b>List of References </b> (Click to expand)</summary>
16+
<summary><b>Table of Content </b> (Click to expand)</summary>
1717

18+
- [Content](#content)
19+
- [Overview](#overview)
20+
- [Demo](#demo)
21+
- [Set Up a Synapse Workspace](#set-up-a-synapse-workspace)
22+
- [Upload Sample Data to Storage Account](#upload-sample-data-to-storage-account)
23+
- [Create User Database](#create-user-database)
24+
- [Create an External Data Source and File Format](#create-an-external-data-source-and-file-format)
25+
- [Create an External Table](#create-an-external-table)
26+
- [Create Views with Modified Tables/Column Names](#create-views-with-modified-tablescolumn-names)
1827

1928
</details>
2029

21-
## Content
30+
## Overview
2231

23-
<details>
24-
<summary><b>Table of Content </b> (Click to expand)</summary>
32+
> [!IMPORTANT]
33+
> The serverless SQL pool in Azure Synapse Analytics `does not support internal tables. It primarily supports external tables and temporary tables`.
34+
> For `internal tables,` you would need to use a `dedicated SQL pool` in Synapse Analytics, which allows you to `create and manage internal tables with local storage`.
2535
36+
| **Table Type** | **Description** | **Use Cases** |
37+
|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
38+
| **External Tables** | - Reference data stored in external sources like Azure Data Lake Storage, Azure Blob Storage, Azure Cosmos DB, etc. <br> - Data is not physically stored in the SQL pool. <br> - Useful for querying large datasets without loading them into the SQL pool. | - Querying large datasets stored externally. <br> - Performing data analysis on data stored in various formats. |
39+
| **Temporary Tables**| - Created and used within the scope of a session. <br> - Data is stored temporarily and is dropped when the session ends. <br> - Useful for intermediate data processing and transformations. | - Storing intermediate results during complex queries. <br> - Performing temporary data transformations and aggregations. |
2640

27-
</details>
2841

42+
## Demo
43+
44+
### Set Up a Synapse Workspace
45+
46+
1. **Sign in to the Azure Portal**: Go to the Azure Portal and sign in with your Azure account.
47+
2. **Navigate to Your Synapse Workspace**: In the Azure Portal, search for your Synapse workspace or create a new one if you don't have one.
48+
49+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/92a5e451-1868-47e2-b32b-858591c306ee" />
50+
51+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/51e3b091-855d-4481-89e0-623705e3cf2a" />
52+
53+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/7d03bfa8-e1e3-4706-970f-a89c7b8cd904" />
54+
55+
3. **Launch Synapse Studio**: From the Synapse workspace overview, click on the `Open Synapse Studio` button.
56+
57+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/302b1fd8-49a6-427e-93dc-8e952f1667e6" />
58+
59+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/a368036b-c859-47fc-a1c5-d045b6910790" />
60+
61+
### Upload Sample Data to Storage Account
62+
63+
> [!IMPORTANT]
64+
> For this demo, `we'll be using the same storage account created with the synpase workspace`.
65+
> `CREATE EXTERNAL DATA SOURCE is not supported in the master database of the serverless` SQL pool. Instead, you `need to create a user database and perform the operations there`.
66+
67+
1. Create a Container in the Storage Account:
68+
- Go to the Azure portal and navigate to your storage account.
69+
- In the left-hand menu, select `Containers`.
70+
- Click on `+ Container` to create a new container.
71+
- Enter a name for the container, such as `sample-tables-container`.
72+
- Set the `Public access level` to your preference (e.g., Private).
73+
- Click `Create`.
74+
75+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/3060045f-a515-4a27-a4c6-80bd2ba3d9ab" />
76+
77+
2. Navigate to the Container:
78+
- In the Azure portal, go to your storage account.
79+
- Select `Containers` from the left-hand menu.
80+
- Click on the container you created (e.g., `sample-tables-container`).
81+
3. Upload the Sample CSV File:
82+
- Click on the `Upload` button.
83+
- In the upload blade, click on `Browse` to select the sample CSV file from your local machine.
84+
- Choose the file, click `Upload` to upload the file to the container.
85+
86+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/4fa8d1c8-e5a0-4d65-a3b5-83bf7c7118c1" />
87+
88+
### Create User Database
89+
90+
1. First, create a new database in your Synapse workspace.
91+
92+
```sql
93+
CREATE DATABASE {User Database Name};
94+
```
95+
96+
<img width="884" alt="image" src="https://github.com/user-attachments/assets/f24a4d45-6bee-46ea-bccc-b5f893044c01" />
97+
98+
2. Switch to the User Database: Use the newly created database.
99+
100+
### Create an External Data Source and File Format
101+
102+
1. Integrate this task with the previous step by establishing the external data source within the user database.
103+
104+
```sql
105+
USE {User Database Name};
106+
107+
CREATE EXTERNAL DATA SOURCE {Data Source Name}
108+
WITH (
109+
LOCATION = 'https://<your-storage-account>.dfs.core.windows.net/<your-container>/'
110+
);
111+
```
112+
113+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/13f22e3b-7c68-4ed2-8a56-2752a5033869" />
114+
115+
116+
2. **Create an External File Format**: As part of the same flow, define the format of the CSV file.
117+
118+
```sql
119+
CREATE EXTERNAL FILE FORMAT {File Format Name}
120+
WITH (
121+
FORMAT_TYPE = DELIMITEDTEXT,
122+
FORMAT_OPTIONS (
123+
FIELD_TERMINATOR = ',',
124+
STRING_DELIMITER = '"',
125+
FIRST_ROW = 2
126+
)
127+
);
128+
```
129+
130+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/40a4c6a2-bc51-49ba-9361-bec6eb0edb9e" />
131+
132+
133+
### Create an External Table
134+
135+
1. Create an external table that references the sample data.
136+
137+
```sql
138+
CREATE EXTERNAL TABLE [Table Name] (
139+
[Employee ID] INT,
140+
[Employee Name] NVARCHAR(50),
141+
[Hire Date] DATE
142+
)
143+
WITH (
144+
LOCATION = 'employee data with spaces.csv',
145+
DATA_SOURCE = {Data Source Name},
146+
FILE_FORMAT = {File Format Name}
147+
);
148+
149+
CREATE EXTERNAL TABLE [Table Name] (
150+
[Product ID] INT,
151+
[Product Name] NVARCHAR(50),
152+
[Release Date] DATE
153+
)
154+
WITH (
155+
LOCATION = 'product data with spaces.csv',
156+
DATA_SOURCE = {Data Source Name},
157+
FILE_FORMAT = {File Format Name}
158+
);
159+
```
160+
161+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/841cc8b7-4c2d-4c9a-975c-fec4bb3326d8" />
162+
163+
2. Confirm the existence of the tables
164+
165+
```sql
166+
SELECT * FROM sys.tables
167+
WHERE name IN ('Product Data', 'Employee Data');
168+
```
169+
170+
<img width="240" alt="image" src="https://github.com/user-attachments/assets/9eb9e379-30a7-42b8-966d-8f8c96b31395" />
171+
172+
173+
3. **Query the External Table**: You can now query the external table to see the sample data.
174+
175+
```sql
176+
SELECT * FROM {Table Name};
177+
```
178+
179+
### Create Views with Modified Tables/Column Names
180+
181+
> This script is designed to dynamically create views for each table in a database, renaming columns to remove spaces. It starts by creating a temporary table to store the SQL statements and assigns a unique row number to each statement. The script then loops through these statements, executing each one in turn. Finally, it cleans up by dropping the temporary table.
182+
> 1. **Temporary Table Creation**: A temporary table `#CreateViewStatements` is created to store the dynamic SQL statements and their corresponding row numbers. <br/>
183+
> 2. **Inserting SQL Statements**: The script generates SQL statements to create views for each table in the database. It uses the `INFORMATION_SCHEMA.COLUMNS` to get the table and column names, renaming columns to remove spaces. These statements, along with a row number, are inserted into the temporary table. <br/>
184+
> 3. **Variable Declaration**: Variables are declared to hold the current SQL statement, the current row number, and the maximum row number. <br/>
185+
> 4. **Getting Maximum Row Number**: The script retrieves the maximum row number from the temporary table to determine how many statements need to be executed. <br/>
186+
> 5. **Executing SQL Statements**: A loop iterates through each row in the temporary table, retrieves the SQL statement, executes it, and increments the row number until all statements are executed. <br/>
187+
> 6. **Cleanup**: The temporary table is dropped to clean up after the script has finished executing. <br/>
188+
189+
```sql
190+
-- Create a temporary table to store the dynamic SQL statements
191+
CREATE TABLE #CreateViewStatements (SQLStatement NVARCHAR(MAX), RowNum INT);
192+
193+
-- Insert dynamic SQL statements for each table with a row number
194+
INSERT INTO #CreateViewStatements (SQLStatement, RowNum)
195+
SELECT
196+
'CREATE VIEW ' + QUOTENAME(REPLACE(TABLE_NAME, ' ', '_')) + ' AS SELECT ' +
197+
STRING_AGG('[' + COLUMN_NAME + '] AS [' + REPLACE(COLUMN_NAME, ' ', '') + ']', ', ') +
198+
' FROM ' + QUOTENAME(TABLE_NAME),
199+
ROW_NUMBER() OVER (ORDER BY TABLE_NAME)
200+
FROM INFORMATION_SCHEMA.COLUMNS
201+
GROUP BY TABLE_NAME;
202+
203+
-- Declare variables to hold the SQL statement and row number
204+
DECLARE @sql NVARCHAR(MAX);
205+
DECLARE @rowNum INT = 1;
206+
DECLARE @maxRowNum INT;
207+
208+
-- Get the maximum row number
209+
SELECT @maxRowNum = MAX(RowNum) FROM #CreateViewStatements;
210+
211+
-- Loop through the temporary table and execute each SQL statement
212+
WHILE @rowNum <= @maxRowNum
213+
BEGIN
214+
-- Get the next SQL statement
215+
SELECT @sql = SQLStatement FROM #CreateViewStatements WHERE RowNum = @rowNum;
216+
217+
-- Execute the SQL statement
218+
EXEC sp_executesql @sql;
219+
220+
-- Increment the row number
221+
SET @rowNum = @rowNum + 1;
222+
END;
223+
224+
-- Drop the temporary table
225+
DROP TABLE #CreateViewStatements;
226+
```
227+
228+
229+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/2654dcb8-d843-4000-9b32-b36f14f2a7a8" />
230+
231+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/6f270523-adf1-4ca3-b80c-b5decd624459" />
29232

233+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/7a5a26ff-9a5f-4c98-a337-b64ce0cc1699" />
234+
30235
<div align="center">
31236
<h3 style="color: #4CAF50;">Total Visitors</h3>
32237
<img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/0_serverlessSQLPool/_samples-files/employee data with spaces.csv‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Employee ID, Employee Name, Hire Date
2+
101, Alice Johnson, 2024-05-15
3+
102, Bob Smith, 2024-06-20
4+
103, Charlie Brown, 2024-07-25

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/0_serverlessSQLPool/_samples-files/product data with spaces.csv‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
ProductID, Product Name, Release Date
2+
201, Gadget Pro, 2023-11-01
3+
202, Widget Plus, 2023-12-15
4+
203, Device Max, 2024-01-10

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/0_serverlessSQLPool/src/0_create-userdb.sql‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
CREATE DATABASE MyUserDatabase

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/0_serverlessSQLPool/src/1_use-userdb.sql‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
USE MyUserDatabase;
2+
3+
CREATE EXTERNAL DATA SOURCE MyDataSourceNameTest
4+
WITH (
5+
LOCATION = 'https://brownteststorage.dfs.core.windows.net/sample-tables-container/'
6+
);

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/0_serverlessSQLPool/src/2_create-external-fileformat.sql‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
CREATE EXTERNAL FILE FORMAT MyFileFormat
2+
WITH (
3+
FORMAT_TYPE = DELIMITEDTEXT,
4+
FORMAT_OPTIONS (
5+
FIELD_TERMINATOR = ',',
6+
STRING_DELIMITER = '"',
7+
FIRST_ROW = 2
8+
)
9+
);

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/0_serverlessSQLPool/src/3_create-external-tables.sql‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
CREATE EXTERNAL TABLE [Employee Data Test Brown] (
2+
[Employee ID] INT,
3+
[Employee Name] NVARCHAR(50),
4+
[Hire Date] DATE
5+
)
6+
WITH (
7+
LOCATION = 'employee data with spaces.csv',
8+
DATA_SOURCE = MyDataSourceNameTest,
9+
FILE_FORMAT = MyFileFormat
10+
);
11+
12+
CREATE EXTERNAL TABLE [Product Data Test Brown] (
13+
[Product ID] INT,
14+
[Product Name] NVARCHAR(50),
15+
[Release Date] DATE
16+
)
17+
WITH (
18+
LOCATION = 'product data with spaces.csv',
19+
DATA_SOURCE = MyDataSourceNameTest,
20+
FILE_FORMAT = MyFileFormat
21+
);

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/0_serverlessSQLPool/src/4_query-tables.sql‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
-- Confirm the existence of the tables
2+
SELECT * FROM sys.tables
3+
WHERE name IN ('Product Data', 'Employee Data');
4+
5+
USE MyUserDatabase;
6+
-- SELECT * FROM [dbo].[Product Data];
7+
SELECT * FROM [dbo].[Employee Data Test Brown];
8+

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/0_serverlessSQLPool/src/5_dynamically-remove-spaces.sql‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
-- Create a temporary table to store the dynamic SQL statements
2+
CREATE TABLE #CreateViewStatements (SQLStatement NVARCHAR(MAX), RowNum INT);
3+
4+
-- Insert dynamic SQL statements for each table with a row number
5+
INSERT INTO #CreateViewStatements (SQLStatement, RowNum)
6+
SELECT
7+
'CREATE VIEW ' + QUOTENAME(REPLACE(TABLE_NAME, ' ', '_')) + ' AS SELECT ' +
8+
STRING_AGG('[' + COLUMN_NAME + '] AS [' + REPLACE(COLUMN_NAME, ' ', '') + ']', ', ') +
9+
' FROM ' + QUOTENAME(TABLE_NAME),
10+
ROW_NUMBER() OVER (ORDER BY TABLE_NAME)
11+
FROM INFORMATION_SCHEMA.COLUMNS
12+
GROUP BY TABLE_NAME;
13+
14+
-- Declare variables to hold the SQL statement and row number
15+
DECLARE @sql NVARCHAR(MAX);
16+
DECLARE @rowNum INT = 1;
17+
DECLARE @maxRowNum INT;
18+
19+
-- Get the maximum row number
20+
SELECT @maxRowNum = MAX(RowNum) FROM #CreateViewStatements;
21+
22+
-- Loop through the temporary table and execute each SQL statement
23+
WHILE @rowNum <= @maxRowNum
24+
BEGIN
25+
-- Get the next SQL statement
26+
SELECT @sql = SQLStatement FROM #CreateViewStatements WHERE RowNum = @rowNum;
27+
28+
-- Execute the SQL statement
29+
EXEC sp_executesql @sql;
30+
31+
-- Increment the row number
32+
SET @rowNum = @rowNum + 1;
33+
END;
34+
35+
-- Drop the temporary table
36+
DROP TABLE #CreateViewStatements;

‎0_Azure/2_AzureAnalytics/1_SynapseAnalytics/demos/2_synapse_views_dynamically_remove_space/1_dedicatedSQLPoolStoreProc/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Costa Rica
66
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
77
[brown9804](https://github.com/brown9804)
88

9-
Last updated: 2025-01-24
9+
Last updated: 2025-01-26
1010

1111
----------
1212

@@ -17,7 +17,7 @@ Last updated: 2025-01-24
1717

1818
- [Content](#content)
1919
- [Demo](#demo)
20-
- [Set Up a Dedicated SQL Pool](#set-up-a-dedicated-sql-pool)
20+
- [Set Up a Synapse Workspace](#set-up-a-synapse-workspace)
2121
- [Create a Dedicated SQL Pool](#create-a-dedicated-sql-pool)
2222
- [Create Tables with Spaces in Names and Columns](#create-tables-with-spaces-in-names-and-columns)
2323
- [Create Views with Modified Tables/Column Names](#create-views-with-modified-tablescolumn-names)
@@ -26,7 +26,7 @@ Last updated: 2025-01-24
2626

2727
## Demo
2828

29-
### Set Up a Dedicated SQL Pool
29+
### Set Up a Synapse Workspace
3030

3131
1. **Sign in to the Azure Portal**: Go to the Azure Portal and sign in with your Azure account.
3232
2. **Navigate to Your Synapse Workspace**: In the Azure Portal, search for your Synapse workspace or create a new one if you don't have one.

0 commit comments

Comments
 (0)