harden Fabric mirroring automation and align docs with Microsoft Foundry

Author: Mike Swantek

CHANGELOG.md

@@ -2,7 +2,25 @@
 
 All notable changes to this project will be documented in this file.
 
-## [Unreleased] - 2026-03-06
+## [2026-03-20]
+### Added
+- Read-only PostgreSQL mirroring preflight script for validating runner prerequisites before mirror setup
+- PostgreSQL mirroring follow-up wrapper to run preflight, preparation, and mirror creation as a deliberate post-deployment flow
+- Shared AI Search helper module for OneLake indexing scripts to centralize public network access toggles and tokenized REST calls
+
+### Changed
+- PostgreSQL mirroring guidance now treats mirroring as a follow-up step after `azd up`, with clearer public-access versus private-network paths
+- Postprovision now restores only PostgreSQL mirroring readiness preparation instead of attempting full mirror creation during the main deployment run
+- Fabric connection and workspace automation now resolve more values from deployment outputs, azd environment values, and deployed resources when transient hook context is incomplete
+- PostgreSQL mirroring scripts now support explicit connection-mode outputs, stronger credential handling, clearer network-path failures, and gateway-aware Fabric connection creation
+- Purview collection and Fabric datasource registration scripts now derive default names and deployment context more reliably from outputs and environment values
+- Post-deployment and mirroring documentation consolidated the mirror workflow into a single primary runbook and clarified when mirroring should be deferred
+
+### Removed
+- Temporary PostgreSQL mirroring prep wrapper that toggled public access as a separate script
+- Fabric connection probe debug script and the redundant PostgreSQL mirroring opt-in guide
+
+## [2026-03-18]
 ### Added
 - Parameter to override Log Analytics workspace resource ID and output mapping for automation scripts
 - Optional `SKIP_PURVIEW_INTEGRATION` guard for Purview automation scripts (used by hooks when Purview is disabled)
@@ -12,17 +30,18 @@ All notable changes to this project will be documented in this file.
 - Preprovision error output simplified with concise failure reason and optional verbose diagnostics
 - Main parameter file reordered into required/optional/defaulted sections with clearer comments
 - OneLake indexing scripts prefer outputs, include AAD-only auth, and handle transient 409 run conflicts
+- Post-deployment steps now include Fabric mirroring checklist items and Key Vault networking guidance for retrieving the `fabric_user` password
 
-### Fixed
-- Power BI headers initialization in Log Analytics linkage script to resolve workspace ID lookups
+### Removed
+- Log Analytics linkage script `scripts/automationScripts/FabricPurviewAutomation/connect_log_analytics.ps1`
 
 ## [1.3] - 2025-12-09
 ### Added
 - Microsoft Fabric integration with automatic capacity creation and management
 - Microsoft Purview integration for governance and data cataloging
 - OneLake indexing pipeline connecting Fabric lakehouses to AI Search
 - Comprehensive post-provision automation (22 hooks for Fabric/Purview/Search setup)
-- New documentation: `deploy_app_from_foundry.md` for publishing apps from AI Foundry
+- New documentation: `deploy_app_from_foundry.md` for publishing apps from Microsoft Foundry
 - New documentation: `TRANSPARENCY_FAQ.md` for responsible AI transparency
 - New documentation: `NewUserGuide.md` for first-time users
 - Header icons matching GSA standard format

README.md

@@ -55,13 +55,13 @@ hooks:
       continueOnError: false
 
     # Stage 5: Purview Collection Creation
-    - run: "$env:SKIP_PURVIEW_INTEGRATION='true'; ./scripts/automationScripts/FabricPurviewAutomation/create_purview_collection.ps1"
+    - run: ./scripts/automationScripts/FabricPurviewAutomation/create_purview_collection.ps1
       interactive: false
       shell: pwsh
       continueOnError: false
 
     # Stage 6: Register Fabric as Purview Data Source
-    - run: "$env:SKIP_PURVIEW_INTEGRATION='true'; ./scripts/automationScripts/FabricWorkspace/CreateWorkspace/register_fabric_datasource.ps1"
+    - run: ./scripts/automationScripts/FabricWorkspace/CreateWorkspace/register_fabric_datasource.ps1
       interactive: false
       shell: pwsh
       continueOnError: false
@@ -72,14 +72,8 @@ hooks:
       shell: pwsh
       continueOnError: false
 
-    # Stage 7.4: Prepare PostgreSQL for Fabric mirroring (server params + role)
-    - run: ./scripts/automationScripts/FabricWorkspace/Mirror/run_postgresql_mirroring_prep_with_public_access.ps1
-      interactive: false
-      shell: pwsh
-      continueOnError: false
-
-    # Stage 7.5: Create PostgreSQL Mirrored Database (if PostgreSQL is provisioned)
-    - run: ./scripts/automationScripts/FabricWorkspace/Mirror/create_postgresql_mirror.ps1
+    # Stage 7.5: Prepare PostgreSQL server for Fabric mirroring readiness
+    - run: ./scripts/automationScripts/FabricWorkspace/mirror/prepare_postgresql_for_mirroring.ps1
       interactive: false
       shell: pwsh
       continueOnError: false
@@ -139,7 +133,7 @@ hooks:
       continueOnError: false
 
     # Stage 17: Trigger Purview Scan (if Purview enabled)
-    - run: "$env:SKIP_PURVIEW_INTEGRATION='true'; ./scripts/automationScripts/FabricPurviewAutomation/trigger_purview_scan_for_fabric_workspace.ps1"
+    - run: ./scripts/automationScripts/FabricPurviewAutomation/trigger_purview_scan_for_fabric_workspace.ps1
       interactive: false
       shell: pwsh
       continueOnError: false

azure.yaml

@@ -10,8 +10,7 @@ After running `azd up` or `azd provision` followed by `azd hooks run postprovisi
 |-----------|---------------|----------------|
 | Fabric Capacity | Azure Portal → Microsoft Fabric capacities | **Active** (not Paused) |
 | Fabric Workspace | [app.fabric.microsoft.com](https://app.fabric.microsoft.com) | Workspace visible with 3 lakehouses |
-| PostgreSQL Mirroring (if enabled) | Fabric → Workspace → Connections/Mirror | Connection saved and mirror running |
-| AI Foundry Project | [ai.azure.com](https://ai.azure.com) | Project accessible, models deployed |
+| Microsoft Foundry project | [ai.azure.com](https://ai.azure.com) | Project accessible, models deployed |
 | AI Search Index | Azure Portal → AI Search → Indexes | `onelake-index` exists with documents |
 | Purview Scan | Purview Portal → Data Map → Sources | Fabric data source registered |
 
@@ -48,35 +47,46 @@ az fabric capacity resume --capacity-name <capacity-name> --resource-group <rg-n
 5. Open the **bronze** lakehouse and verify the `Files/documents` folder structure exists
 6. In the workspace, check each lakehouse (**bronze**, **silver**, **gold**) and confirm the **Sensitivity label** matches the value set in the parameter file.
 
-### PostgreSQL Mirroring (if enabled)
+### Optional PostgreSQL Mirroring Follow-Up
 
-Use these short steps to verify the automatic Fabric connection and mirroring flow. For full details and troubleshooting, see [PostgreSQL mirroring](./postgresql_mirroring.md).
+Use these short steps to verify the PostgreSQL mirroring follow-up flow. For full details and troubleshooting, see [PostgreSQL mirroring](./postgresql_mirroring.md).
+
+Mirroring in the current branch is a separate follow-up activity. Fabric connection creation and mirrored database creation are not part of `azd up`.
+
+For post-deployment verification, the important distinction is simple:
+
+- If you did not intentionally run the mirroring follow-up, treat mirroring as deferred and do not use it as a deployment success criterion.
+- If you did run the mirroring follow-up, verify the Fabric connection and mirrored database from the workspace.
+
+If you need to complete mirroring after deployment, use the dedicated steps in [PostgreSQL mirroring](./postgresql_mirroring.md).
+
+The PostgreSQL server's **Fabric Mirroring** page only covers the source-server prerequisite preparation. It does not replace the Fabric workspace connection and mirrored database creation steps.
 
-0. In **Azure Portal** → **Key Vault** → your vault → **Networking**, set **Public access** to **Allow public access from specific virtual networks and IP addresses**, add your client IP, then **Apply**. This lets you read the PostgreSQL connection password from the vault.
-   After you retrieve the secret, remove your IP and **Apply** again to re-lock the vault.
 1. Check the resolved mirroring identity instead of hardcoding it:
    - `azd env get-value postgreSqlMirrorConnectionModeOut`
    - `azd env get-value postgreSqlMirrorConnectionUserNameOut`
    - `azd env get-value postgreSqlMirrorConnectionSecretNameOut`
-2. Run `pwsh ./scripts/automationScripts/FabricWorkspace/Mirror/create_postgresql_mirror.ps1` if Stage 7.5 did not already complete it.
-3. Verify `azd env get-value fabricPostgresConnectionId` now returns a Fabric connection ID.
-4. In Fabric, confirm the PostgreSQL connection exists under **Connections** and that the mirrored database is running.
-5. If your PostgreSQL source requires a Fabric VNet gateway, set `azd env set-value fabricPostgresGatewayId "<gateway-id>"` and rerun the script.
+2. If you have not run the separate mirroring follow-up, stop here for this test cycle.
+   - The deployment can still be considered successful for Fabric workspace, PostgreSQL server, and Purview automation.
+   - PostgreSQL mirroring remains a documented follow-up item, not a same-run success criterion.
+3. If you want mirroring now, follow the current runbook in [PostgreSQL mirroring](./postgresql_mirroring.md).
+4. After the follow-up completes, verify `azd env get-value fabricPostgresConnectionId` returns a Fabric connection ID.
+5. In Fabric, confirm the PostgreSQL connection exists under **Connections** and that the mirrored database is running.
 
 ---
 
-## 3. Verify AI Foundry Project
+## 3. Verify Microsoft Foundry Project
 
 1. Navigate to [ai.azure.com](https://ai.azure.com)
-2. Sign in and select your AI Foundry project
+2. Sign in and select your Microsoft Foundry project
 3. Verify:
    - **Models** — Check that GPT-4o and text-embedding-ada-002 (or configured models) are deployed
    - **Connections** — AI Search connection should be listed
    - **Playground** — Test the chat playground with a sample query
 
 ### Testing AI Search Connection in Playground
 
-1. In AI Foundry, go to **Playgrounds** → **Chat**
+1. In Microsoft Foundry, go to **Playgrounds** → **Chat**
 2. Click **Add your data**
 3. Select your AI Search index (`onelake-index`)
 4. Ask a question about your indexed documents
@@ -113,6 +123,16 @@ If no documents appear, check:
 3. Verify the Fabric data source is registered (e.g., `Fabric-Workspace-<id>`)
 4. Check **Scans** to see if the initial scan completed
 
+If `purviewCollectionName` is left empty in [infra/main.bicepparam](../infra/main.bicepparam), the automation now uses `collection-<AZURE_ENV_NAME>`.
+
+If you need to rerun the Purview steps after provisioning:
+
+```powershell
+pwsh ./scripts/automationScripts/FabricPurviewAutomation/create_purview_collection.ps1
+pwsh ./scripts/automationScripts/FabricWorkspace/CreateWorkspace/register_fabric_datasource.ps1
+pwsh ./scripts/automationScripts/FabricPurviewAutomation/trigger_purview_scan_for_fabric_workspace.ps1
+```
+
 ### Data Lineage
 
 1. In Purview, go to **Data Catalog** → **Browse**
@@ -125,23 +145,23 @@ If no documents appear, check:
 
 When `networkIsolation` is set to `true`:
 
-### Check AI Foundry Network Settings
+### Check Microsoft Foundry Network Settings
 
-1. Go to **Azure Portal** → **Azure AI Foundry** → your account
+1. Go to **Azure Portal** → **Microsoft Foundry** → your account
 2. Click **Settings** → **Networking**
 3. Verify:
    - **Public network access**: Disabled (if fully isolated)
    - **Private endpoints**: Active connections listed
 
-   ![Image showing the Azure Portal for AI Foundry and the settings blade](../img/provisioning/checkNetworkIsolation1.png)
+   ![Image showing the Azure Portal for Microsoft Foundry and the settings blade](../img/provisioning/checkNetworkIsolation1.png)
 
 4. Open the **Workspace managed outbound access** tab to see private endpoints
 
    ![Image showing managed outbound access](../img/provisioning/checkNetworkIsolation2.png)
 
 ### Test Isolation
 
-When accessing AI Foundry from outside the virtual network, you should see an access denied message:
+When accessing Microsoft Foundry from outside the virtual network, you should see an access denied message:
 
 ![Image showing access denied from public network](../img/provisioning/checkNetworkIsolation4.png)
 
@@ -170,7 +190,7 @@ For network-isolated deployments, use Azure Bastion to access resources:
    ![Image showing bastion login](../img/provisioning/checkNetworkIsolation8.png)
 
 5. Once connected, open **Edge browser** and navigate to:
-   - [ai.azure.com](https://ai.azure.com) — AI Foundry
+   - [ai.azure.com](https://ai.azure.com) — Microsoft Foundry
    - [app.fabric.microsoft.com](https://app.fabric.microsoft.com) — Fabric
 
 6. Complete MFA if prompted
@@ -195,9 +215,9 @@ az resource show --ids /subscriptions/<sub>/resourceGroups/<rg>/providers/Micros
 az fabric capacity resume --capacity-name <name> --resource-group <rg>
 ```
 
-### AI Search Connection Fails in AI Foundry Playground
+### AI Search Connection Fails in Microsoft Foundry Playground
 
-Verify RBAC roles are assigned to the AI Foundry identities:
+Verify RBAC roles are assigned to the Microsoft Foundry identities:
 
 ```bash
 # Get the AI Search resource ID
@@ -208,7 +228,7 @@ az role assignment list --scope $SEARCH_ID --output table
 ```
 
 Required roles on the AI Search service:
-- **Search Service Contributor** — For the AI Foundry account and project managed identities
+- **Search Service Contributor** — For the Microsoft Foundry account and project managed identities
 - **Search Index Data Contributor** — For read/write access to index data
 - **Search Index Data Reader** — For read access to index data
 
@@ -266,7 +286,7 @@ pwsh ./scripts/automationScripts/<path-to-script>.ps1
 Once verification is complete:
 
 1. **Upload documents** to the bronze lakehouse for indexing
-2. **Test the AI Foundry playground** with your indexed content
+2. **Test the Microsoft Foundry playground** with your indexed content
 3. **Configure additional models** if needed
-4. **[Deploy your app](./deploy_app_from_foundry.md)** from the AI Foundry playground
+4. **[Deploy your app](./deploy_app_from_foundry.md)** from the Microsoft Foundry playground
 5. **Review governance** in Microsoft Purview