post release updates for process steps and code break fixes and hardening

Author: Mike Swantek

README.md

@@ -65,6 +65,8 @@ For the first attempt, the lowest-risk path is to keep Fabric and Purview disabl
 | PostgreSQL mirroring | PostgreSQL enabled in the deployment with `postgreSqlNetworkIsolation = false`, then follow the post-deploy mirror steps | Database deploys, but mirroring is not completed |
 | Private networking | `networkIsolation = true` and enough deployment time for private endpoint provisioning | Deployment takes longer and is harder to troubleshoot if other prerequisites are not already stable |
 
+> **Purview note:** If you enable Purview integration, the identity running `azd` must have **Purview Collection Admin** (or equivalent) on the target collection. To ensure a collection is created or resolved, set `purviewCollectionName` so automation captures `PURVIEW_COLLECTION_ID` and the scan can be assigned.
+
 ### Choose Your Starting Path
 
 | Goal | Recommended path |
@@ -110,7 +112,7 @@ Follow the deployment guide to deploy this solution to your own Azure subscripti
 
 1. Run `azd auth login` and confirm the target subscription with `az account show`
 2. Create a new environment and set `AZURE_SUBSCRIPTION_ID` and `AZURE_LOCATION`
-3. Review `infra/main.bicepparam`, especially `principalId`, `aiSearchAdditionalAccessObjectIds`, `fabricCapacityPreset`, `fabricWorkspacePreset`, `fabricCapacityAdmins`, `purviewAccountResourceId`, `networkIsolation`, `postgreSqlNetworkIsolation`, and `postgreSqlAllowAzureServices`
+3. Review `infra/main.bicepparam`, especially `principalId`, `aiSearchAdditionalAccessObjectIds`, `fabricCapacityPreset`, `fabricWorkspacePreset`, `fabricCapacityAdmins`, `purviewAccountResourceId`, `purviewCollectionName`, `networkIsolation`, `postgreSqlNetworkIsolation`, and `postgreSqlAllowAzureServices`
 4. Run `azd up`
 5. Follow [docs/post_deployment_steps.md](./docs/post_deployment_steps.md) to verify the deployment
 

docs/deploymentguide.md

@@ -37,6 +37,8 @@ To deploy this solution accelerator, ensure you have access to an [Azure subscri
 | **Microsoft Fabric** | Access to create F8 capacity and workspace, OR existing Fabric capacity ID |
 | **Microsoft Purview** | Existing tenant-level Purview account resource ID |
 
+> **Purview requirement:** The identity running `azd` must have **Purview Collection Admin** (or equivalent) on the target collection. If the collection cannot be created or resolved, scan automation will skip collection assignment.
+
 ### Region Availability
 
 Check [Azure Products by Region](https://azure.microsoft.com/en-us/explore/global-infrastructure/products-by-region/) to ensure the following services are available in your target region:
@@ -164,6 +166,7 @@ Edit `infra/main.bicepparam` or set environment variables:
 | Parameter | Description | Example |
 |-----------|-------------|---------|
 | `purviewAccountResourceId` | Resource ID of existing Purview account | `/subscriptions/.../Microsoft.Purview/accounts/...` |
+| `purviewCollectionName` | Optional. Purview collection name to create or resolve for scans. If blank, scripts create `collection-<env name>`. | `ai-prod-collection` |
 | `fabricCapacityPreset` | Fabric capacity preset: `create`, `byo`, or `none` | `create` |
 | `fabricWorkspacePreset` | Fabric workspace preset: `create`, `byo`, or `none` | `create` |
 | `fabricCapacitySku` | Fabric capacity SKU (only used when `fabricCapacityPreset=create`) | `F8` (default) |
@@ -177,6 +180,11 @@ Edit `infra/main.bicepparam` or set environment variables:
 # (Edit infra/main.bicepparam)
 # param purviewAccountResourceId = "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Purview/accounts/<account-name>"
 
+# Example: Set Purview collection (optional)
+# (Edit infra/main.bicepparam)
+# param purviewCollectionName = "ai-prod-collection"
+# Leave blank to auto-create collection-<env name>
+
 # Example: Disable all Fabric automation
 # (Edit infra/main.bicepparam)
 # var fabricCapacityPreset = 'none'

docs/post_deployment_steps.md

@@ -149,6 +149,8 @@ If the connection fails, verify RBAC roles are assigned (see Troubleshooting sec
 
 If `purviewCollectionName` is left empty in [infra/main.bicepparam](../infra/main.bicepparam), the automation now uses `collection-<AZURE_ENV_NAME>`.
 
+If the identity running `azd` does not have **Purview Collection Admin** (or equivalent) on the target collection, the Purview scripts will warn and skip collection, datasource, and scan steps. Grant the role, then rerun the Purview scripts.
+
 If you need to rerun the Purview steps after provisioning:
 
 ```powershell

docs/postgresql_mirroring.md

@@ -34,7 +34,7 @@ This is the most common flow when you run `azd up` from a non-VNet machine. The
 
 ### Public Access Enabled
 
-Use this path when the PostgreSQL server has `publicNetworkAccess=Enabled`. In this repo, that corresponds to `postgreSqlNetworkIsolation = false`.
+Follow this path when the PostgreSQL server has `publicNetworkAccess=Enabled`. In this repo, that corresponds to `postgreSqlNetworkIsolation = false`.
 
 Recommended deployment settings for this path:
 
@@ -74,7 +74,7 @@ If the database or login fails, confirm `postgreSqlAllowAzureServices = true` (o
 
 ### Private Network or Private Endpoint
 
-Use this path when the PostgreSQL server is private-only or Fabric cannot reach it over public networking.
+Follow this path when the PostgreSQL server is private-only or Fabric cannot reach it over public networking.
 
 You must supply a Fabric VNet gateway ID for the connection flow in this mode. The repo may add a gateway option in a future update, but today you need to bring your own gateway and set `fabricPostgresGatewayId` before creating the connection.
 
@@ -85,62 +85,40 @@ You must supply a Fabric VNet gateway ID for the connection flow in this mode. T
 
 ### What to Do First
 
-If you just need the mirror working with the fewest steps, follow the **Public Access Enabled** flow above.
+If you want the shortest path to a working mirror, follow the **Public Access Enabled** flow above.
 
 If you are intentionally staying private for now, skip mirror creation for this provisioning test and continue validating the rest of the deployment.
 
 ## Recommended Repo Flow
 
-In this repo, mirroring should be treated as a deliberate follow-up step after the main deployment completes.
+In this repo, mirroring is prepared during postprovision and only needs a short manual follow-up after the main deployment completes.
 
 That means:
 
-1. `azd up` deploys the infrastructure and core postprovision automation.
+1. `azd up` deploys the infrastructure and runs the mirroring prep automation.
 2. PostgreSQL mirroring is not a required same-run success criterion.
-3. If you want mirroring, run it afterward from a runner that can actually reach PostgreSQL, Key Vault, and Fabric.
+3. The only required follow-up is a short manual Fabric registration step (see below).
 
 The cleanest sequence is:
 
 1. Run `azd up`.
 2. Validate the deployment with [post_deployment_steps.md](./post_deployment_steps.md).
-3. Connect to the deployed VM or another runner with PostgreSQL network reachability.
-4. Run the mirroring follow-up flow, or use the manual steps above if you are not on the VNet.
+3. Temporarily enable Key Vault public access (if needed) to retrieve the Fabric user secret.
+4. Register the PostgreSQL connection in Fabric and create the mirror.
 5. Verify the Fabric connection and mirrored database.
 
 Running from the deployed VM is usually the least fragile option because it avoids local DNS, firewall, VPN, and endpoint-security issues.
 
-### Follow-Up Wrapper
+### Manual Follow-Up (Required)
 
-If you want the repo-managed sequence, run:
+After `azd up`, no additional scripts are required. Complete these manual steps:
 
-```powershell
-pwsh -NoProfile -File .\scripts\automationScripts\FabricWorkspace\mirror\run_postgresql_mirroring_followup.ps1
-```
-
-That wrapper runs, in order:
-
-1. `test_postgresql_mirroring_prereqs.ps1`
-2. `prepare_postgresql_for_mirroring.ps1`
-3. `create_postgresql_mirror.ps1`
-
-### Preflight First
-
-Before attempting mirroring from a VM or any other runner, use the read-only preflight:
-
-```powershell
-pwsh -NoProfile -File .\scripts\automationScripts\FabricWorkspace\mirror\test_postgresql_mirroring_prereqs.ps1
-```
-
-It checks the things that usually break the flow:
-
-1. `az` and `azd` availability
-2. Azure sign-in
-3. required `azd` environment values
-4. DNS resolution for PostgreSQL
-5. TCP connectivity to PostgreSQL on `5432`
-6. Fabric token acquisition
+1. Temporarily enable Key Vault public access (if it is private).
+2. Retrieve the Fabric user secret from Key Vault.
+3. Register the PostgreSQL connection in Fabric.
+4. Create the mirror in Fabric and validate sync.
 
-If preflight fails, fix the runner first instead of continuing into SQL prep or Fabric connection creation.
+If you need to troubleshoot connectivity from a runner, use the preflight script as a diagnostic, but it is no longer required for the normal flow.
 
 ## Automation status
 

infra/main.bicepparam

@@ -1,4 +1,4 @@
-using './main.bicep'
+using './main.bicep'
 
 // ========================================
 // REQUIRED INPUTS
@@ -205,7 +205,7 @@ param containerAppsList = [
 ]
 
 param vmAdminPassword = readEnvironmentVariable('VM_ADMIN_PASSWORD', '$(secretOrRandomPassword)')
-param vmSize = 'Standard_D2s_v3'
+param vmSize = 'Standard_D2s_v4'
 
 // ========================================
 // FABRIC CAPACITY PARAMETERS
@@ -244,9 +244,8 @@ param fabricWorkspaceName = '' // optional (helpful for naming/UX)
 // Fabric capacity SKU.
 param fabricCapacitySku = 'F8'
 
-// Fabric capacity admin members (email addresses or object IDs).
-var fabricAdminValue = readEnvironmentVariable('fabricCapacityAdmins', '')
-param fabricCapacityAdmins = empty(fabricAdminValue) ? [] : split(fabricAdminValue, ',')
+// Fabric capacity admin members (UPN emails preferred).
+param fabricCapacityAdmins = []
 
 // ========================================
 // PURVIEW PARAMETERS (Optional)

scripts/automationScripts/FabricPurviewAutomation/create_purview_collection.ps1

@@ -114,6 +114,16 @@ function Resolve-PurviewFromResourceId([string]$resourceId) {
   }
 }
 
+function Test-PurviewCollectionAdmin([string]$endpoint, [hashtable]$headers) {
+  try {
+    Invoke-SecureRestMethod -Uri "$endpoint/account/collections?api-version=2019-11-01-preview" -Headers $headers -Method Get -ErrorAction Stop | Out-Null
+    return $true
+  } catch {
+    Warn "Purview collection access check failed. Ensure the current identity has Purview Collection Admin on the target collection."
+    return $false
+  }
+}
+
 function Get-DefaultPurviewCollectionName() {
   $environmentName = $env:AZURE_ENV_NAME
   if (-not $environmentName) { $environmentName = Get-AzdEnvValue -key 'AZURE_ENV_NAME' }
@@ -198,6 +208,11 @@ if (-not $purviewToken) { Fail 'Failed to acquire Purview access token' }
 $purviewHeaders = New-SecureHeaders -Token $purviewToken
 
 $endpoint = "https://$purviewAccountName.purview.azure.com"
+if (-not (Test-PurviewCollectionAdmin -endpoint $endpoint -headers $purviewHeaders)) {
+  Warn "Skipping Purview collection setup due to missing collection permissions."
+  Clear-SensitiveVariables -VariableNames @("accessToken", "fabricToken", "purviewToken", "powerBIToken", "storageToken")
+  exit 0
+}
 # Check existing collections
 $allCollections = Invoke-SecureRestMethod -Uri "$endpoint/account/collections?api-version=2019-11-01-preview" -Headers $purviewHeaders -Method Get -ErrorAction Stop
 $existing = $null

scripts/automationScripts/FabricPurviewAutomation/trigger_purview_scan_for_fabric_workspace.ps1

@@ -64,6 +64,16 @@ function Resolve-PurviewFromResourceId([string]$resourceId) {
   }
 }
 
+function Test-PurviewCollectionAdmin([string]$endpoint, [hashtable]$headers) {
+  try {
+    Invoke-SecureRestMethod -Uri "$endpoint/account/collections?api-version=2019-11-01-preview" -Headers $headers -Method Get -ErrorAction Stop | Out-Null
+    return $true
+  } catch {
+    Warn "Purview collection access check failed. Ensure the current identity has Purview Collection Admin on the target collection."
+    return $false
+  }
+}
+
 # Resolve Purview account name
 $PurviewAccountName = $env:PURVIEW_ACCOUNT_NAME
 $PurviewSubscriptionId = $env:PURVIEW_SUBSCRIPTION_ID
@@ -166,6 +176,13 @@ if (-not $purviewToken) { Fail "Failed to acquire Purview access token" }
 
 $endpoint = "https://$PurviewAccountName.purview.azure.com"
 
+$purviewHeaders = New-SecureHeaders -Token $purviewToken
+if (-not (Test-PurviewCollectionAdmin -endpoint $endpoint -headers $purviewHeaders)) {
+  Warn "Skipping Purview scan trigger due to missing collection permissions."
+  Clear-SensitiveVariables -VariableNames @("accessToken", "fabricToken", "purviewToken", "powerBIToken", "storageToken")
+  exit 0
+}
+
 # Determine Purview datasource name. If a previous script created it, fabric_datasource.env in the temp directory will contain FABRIC_DATASOURCE_NAME. If missing or empty, skip scan creation.
 $datasourceName = 'Fabric'
 $tempDir = [IO.Path]::GetTempPath()
@@ -313,7 +330,7 @@ for ($attempt = 1; $attempt -le $maxCreateAttempts; $attempt++) {
   Warn "Scan create/update failed (HTTP $code): $respBody"
   if ($collectionId) {
     try {
-      Warn "Retrying scan create/update without collection assignment..."
+      Warn "Retrying scan create/update without collection assignment (collection reference may be missing or invalid)..."
       $payloadNoCollection = $payload.PSObject.Copy()
       if ($payloadNoCollection.properties -and $payloadNoCollection.properties.PSObject.Properties.Name -contains 'collection') {
         $payloadNoCollection.properties.PSObject.Properties.Remove('collection')
@@ -356,7 +373,9 @@ if (-not $createSucceeded -and -not $scanExists) {
   if ($lastCreateStatus -or $lastCreateBody) {
     Warn "Final scan create/update response (HTTP $lastCreateStatus): $lastCreateBody"
   }
-  Fail "Could not create or retrieve scan definition after $maxCreateAttempts attempts."
+  Warn "Could not create or retrieve scan definition after $maxCreateAttempts attempts. Continuing without a Purview scan run."
+  Clear-SensitiveVariables -VariableNames @("accessToken", "fabricToken", "purviewToken", "powerBIToken", "storageToken")
+  exit 0
 }
 
 # Trigger a run with retries

scripts/automationScripts/FabricWorkspace/CreateWorkspace/register_fabric_datasource.ps1

@@ -164,6 +164,16 @@ function Get-DefaultPurviewCollectionName() {
   return "collection-$($environmentName.Trim())"
 }
 
+function Test-PurviewCollectionAdmin([string]$endpoint, [hashtable]$headers) {
+  try {
+    Invoke-SecureRestMethod -Uri "$endpoint/account/collections?api-version=2019-11-01-preview" -Headers $headers -Method Get -ErrorAction Stop | Out-Null
+    return $true
+  } catch {
+    Warn "Purview collection access check failed. Ensure the current identity has Purview Collection Admin on the target collection."
+    return $false
+  }
+}
+
 # Resolve Purview account and collection name from outputs/env/azd
 $outputs = $null
 if ($env:AZURE_OUTPUTS_JSON) {
@@ -317,6 +327,12 @@ try {
 # Create secure headers
 $purviewHeaders = New-SecureHeaders -Token $purviewToken
 
+if (-not (Test-PurviewCollectionAdmin -endpoint $endpoint -headers $purviewHeaders)) {
+  Warn "Skipping Purview datasource registration due to missing collection permissions."
+  Clear-SensitiveVariables -VariableNames @('purviewToken')
+  exit 0
+}
+
 # Debug: print the identity running this script
 try {
   $acctName = & az account show --query name -o tsv 2>$null