Merge pull request #105 from microsoft/feature/fabric-private-link-cleanup

feat: fabric automation updates

Author: aniaroramsft

README.md

@@ -121,15 +121,17 @@ Follow the deployment guide to deploy this solution to your own Azure subscripti
   | Requirement | Details |
   |-------------|---------|
   | **Azure Subscription** | Owner or Contributor + User Access Administrator permissions |
-  | **Microsoft Fabric** | Access to create capacity, workspace (or existing Fabric capacity ID) |
+  | **Microsoft Fabric** | Optional. Either access to create capacity/workspace, or provide existing Fabric capacity/workspace IDs, or disable Fabric automation |
   | **Microsoft Purview** | Existing tenant-level Purview account (or ability to create one) |
   | **Azure CLI** | Version 2.61.0 or later |
   | **Azure Developer CLI** | Version 1.15.0 or later |
   | **Quota** | Sufficient Azure OpenAI quota ([check here](./docs/quota_check.md)) |
 
-  > **Note:** If you enable Fabric capacity deployment, you must supply at least one valid Fabric capacity admin principal (Entra user UPN email or object ID) via `fabricCapacityAdmins`.
+  > **Note:** Fabric automation is optional. To disable all Fabric automation, set `fabricCapacityPreset = 'none'` and `fabricWorkspacePreset = 'none'` in `infra/main.bicepparam`.
 
-  > **Note:** If you enable Fabric provisioning, the user running `azd` must have the **Fabric Administrator** role (or equivalent Fabric/Power BI tenant admin permissions) to call the required admin APIs.
+  > **Note:** If you enable Fabric capacity deployment (`fabricCapacityPreset='create'`), you must supply at least one valid Fabric capacity admin principal (Entra user UPN email or object ID) via `fabricCapacityAdmins`.
+
+  > **Note:** If you enable Fabric provisioning (`fabricWorkspacePreset='create'`), the user running `azd` must have the **Fabric Administrator** role (or equivalent Fabric/Power BI tenant admin permissions) to call the required admin APIs.
 
 </details>
 
@@ -141,7 +143,7 @@ Follow the deployment guide to deploy this solution to your own Azure subscripti
   | Azure AI Foundry | Standard | [Pricing](https://azure.microsoft.com/pricing/details/machine-learning/) |
   | Azure OpenAI | Pay-per-token | [Pricing](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) |
   | Azure AI Search | Standard | [Pricing](https://azure.microsoft.com/pricing/details/search/) |
-  | Microsoft Fabric | F8 Capacity | [Pricing](https://azure.microsoft.com/pricing/details/microsoft-fabric/) |
+  | Microsoft Fabric | F8 Capacity (if enabled) | [Pricing](https://azure.microsoft.com/pricing/details/microsoft-fabric/) |
   | Virtual Network + Bastion | Standard | [Pricing](https://azure.microsoft.com/pricing/details/azure-bastion/) |
 
   >  **Cost Optimization:** Fabric capacity can be paused when not in use. Use `az fabric capacity suspend` to stop billing.
@@ -171,8 +173,6 @@ After deployment, you'll have a complete, enterprise-ready platform that unifies
 | **Governance** | Microsoft Purview with cataloging, scans, and DSPM | Track data lineage, enforce policies, and maintain compliance visibility |
 | **Security** | Private endpoints, managed identities, RBAC, network isolation | Zero public internet exposure—all traffic stays on the Microsoft backbone |
 
-> 💡 **Note:** When Microsoft Fabric automation supports private link provisioning, the entire solution will operate with full network isolation end-to-end.
-
 <br/>
 
 ### Key Features

azure.yaml

@@ -15,9 +15,18 @@ metadata:
 # Pre/Post-provision automation hooks
 hooks:
   preprovision:
+    # Integrated preprovision:
+    # - Runs AI Landing Zone preprovision to generate deploy/ files and Template Specs
+    # - Ensures our wrapper points to deploy/main.bicep (Template Spec-based) to avoid ARM 4MB template limit
+    # On Windows, `shell: sh` may not be available; the PowerShell script is a fallback.    
     - shell: sh
-      run: ./submodules/ai-landing-zone/bicep/scripts/preprovision.sh
-      interactive: true
+      run: ./scripts/preprovision-integrated.sh
+      interactive: false
+      continueOnError: true
+
+    - shell: pwsh
+      run: ./scripts/preprovision-integrated.ps1
+      interactive: false
       continueOnError: false
 
   postprovision:
@@ -45,18 +54,6 @@ hooks:
       shell: pwsh
       continueOnError: false
 
-    # Stage 3.3: Create Private Link Service Resource (prerequisite for private endpoint)
-    - run: ./scripts/automationScripts/FabricWorkspace/SecureWorkspace/create_fabric_private_link_service.ps1
-      interactive: false
-      shell: pwsh
-      continueOnError: false
-    
-    # Stage 3.7: Setup Workspace Private Endpoint (for secure VNet access)
-    - run: ./scripts/automationScripts/FabricWorkspace/SecureWorkspace/setup_workspace_private_endpoint.ps1
-      interactive: false
-      shell: pwsh
-      continueOnError: false
-    
     # Stage 4: Assign Workspace to Domain
     - run: ./scripts/automationScripts/FabricWorkspace/CreateWorkspace/assign_workspace_to_domain.ps1
       interactive: false

docs/DeploymentGuide.md

@@ -152,14 +152,22 @@ Edit `infra/main.bicepparam` or set environment variables:
 | Parameter | Description | Example |
 |-----------|-------------|---------|
 | `purviewAccountResourceId` | Resource ID of existing Purview account | `/subscriptions/.../Microsoft.Purview/accounts/...` |
-| `aiSearchAdditionalAccessObjectId` | Array of ObjectId's to apply RBAC role for Search Access | `["user@contoso.com"]`  |
-| `fabricCapacitySku` | Fabric capacity SKU | `F8` (default) |
-| `fabricCapacityAdmins` | Fabric capacity admin principals (UPN emails or Entra object IDs) | `["user@contoso.com"]` |
-| `desiredFabricWorkspaceName` | Name for Fabric workspace | `workspace-myenv` |
+| `aiSearchAdditionalAccessObjectIds` | Array of Entra object IDs to grant Search roles | `["00000000-0000-0000-0000-000000000000"]` |
+| `fabricCapacityMode` | Fabric capacity mode: `create`, `byo`, or `none` | `create` |
+| `fabricWorkspaceMode` | Fabric workspace mode: `create`, `byo`, or `none` | `create` |
+| `fabricCapacitySku` | Fabric capacity SKU (only used when `fabricCapacityMode=create`) | `F8` (default) |
+| `fabricCapacityAdmins` | Fabric capacity admin principals (UPN emails or Entra object IDs) (required when `fabricCapacityMode=create`) | `["user@contoso.com"]` |
+| `fabricCapacityResourceId` | Existing Fabric capacity ARM resource ID (required when `fabricCapacityMode=byo`) | `/subscriptions/.../providers/Microsoft.Fabric/capacities/...` |
+| `fabricWorkspaceId` | Existing Fabric workspace ID (GUID) (required when `fabricWorkspaceMode=byo`) | `00000000-0000-0000-0000-000000000000` |
+| `fabricWorkspaceName` | Existing Fabric workspace name (used when `fabricWorkspaceMode=byo`) | `my-existing-workspace` |
 
 ```bash
 # Example: Set Purview account
 azd env set purviewAccountResourceId "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Purview/accounts/<account-name>"
+
+# Example: Disable all Fabric automation
+azd env set fabricCapacityMode none
+azd env set fabricWorkspaceMode none
 ```
 
 </details>

docs/PARAMETER_GUIDE.md

@@ -1,6 +1,16 @@
 # Parameter Guide for AI Landing Zone Deployment
 
-This guide explains every parameter in `infra/main.parameters.json` and how to customize your deployment.
+This guide focuses on configuration concepts for the **AI Landing Zone**.
+
+> **Important**: This repository deploys using Bicep parameter files, not `infra/main.parameters.json`.
+>
+> - Primary parameters file: `infra/main.bicepparam`
+> - AI Landing Zone submodule parameters file (if you deploy it directly): `submodules/ai-landing-zone/bicep/infra/main.bicepparam`
+>
+> **Fabric options in this repo** are configured in `infra/main.bicepparam` via:
+> - `fabricCapacityPreset` (`create` | `byo` | `none`)
+> - `fabricWorkspacePreset` (`create` | `byo` | `none`)
+> - BYO inputs: `fabricCapacityResourceId`, `fabricWorkspaceId`, `fabricWorkspaceName`
 
 ## Table of Contents
 1. [Basic Parameters](#basic-parameters)

docs/automation-outputs-mapping.md

@@ -20,9 +20,13 @@ The postprovision automation scripts consume deployment outputs via the `AZURE_O
 
 | Bicep Output | Script Variable | Used By | Purpose |
 |-------------|-----------------|---------|---------|
+| `fabricCapacityModeOut` | `fabricCapacityMode` | Multiple Fabric scripts | Whether capacity is `create`, `byo`, or `none` |
+| `fabricWorkspaceModeOut` | `fabricWorkspaceMode` | Multiple Fabric scripts | Whether workspace is `create`, `byo`, or `none` |
 | `fabricCapacityId` | `FABRIC_CAPACITY_ID` | `ensure_active_capacity.ps1` | ARM resource ID of Fabric capacity |
-| `fabricCapacityResourceId` | `fabricCapacityId` | `create_fabric_workspace.ps1` | Resource ID for capacity assignment |
-| `desiredFabricWorkspaceName` | `FABRIC_WORKSPACE_NAME` | Multiple Fabric scripts | Target workspace name |
+| `fabricCapacityResourceIdOut` | `fabricCapacityId` | `create_fabric_workspace.ps1` | Resource ID for capacity assignment |
+| `fabricWorkspaceIdOut` | `FABRIC_WORKSPACE_ID` | Multiple Fabric scripts | Existing or created Fabric workspace ID |
+| `fabricWorkspaceNameOut` | `FABRIC_WORKSPACE_NAME` | Multiple Fabric scripts | Target workspace name |
+| `desiredFabricWorkspaceName` | `FABRIC_WORKSPACE_NAME` | Multiple Fabric scripts | Back-compat alias for `fabricWorkspaceName` |
 | `desiredFabricDomainName` | `domainName` | `create_fabric_domain.ps1` | Target domain name |
 | `fabricCapacityName` | - | - | Display name (optional) |
 
@@ -79,7 +83,11 @@ When `azd up` completes, it sets:
 ```bash
 export AZURE_OUTPUTS_JSON='{
   "fabricCapacityId": {"type":"String","value":"/subscriptions/.../fabricCapacities/fabric-xyz"},
-  "desiredFabricWorkspaceName": {"type":"String","value":"ai-workspace"},
+  "fabricCapacityModeOut": {"type":"String","value":"create"},
+  "fabricWorkspaceModeOut": {"type":"String","value":"create"},
+  "fabricWorkspaceNameOut": {"type":"String","value":"workspace-myenv"},
+  "fabricWorkspaceIdOut": {"type":"String","value":""},
+  "desiredFabricWorkspaceName": {"type":"String","value":"workspace-myenv"},
   "aiSearchName": {"type":"String","value":"search-xyz"},
   "aiSearchResourceGroup": {"type":"String","value":"rg-ai-landing-zone"},
   ...
@@ -93,7 +101,10 @@ Scripts parse this JSON:
 if (-not $WorkspaceName -and $env:AZURE_OUTPUTS_JSON) {
   try { 
     $out = $env:AZURE_OUTPUTS_JSON | ConvertFrom-Json
-    $WorkspaceName = $out.desiredFabricWorkspaceName.value 
+    $WorkspaceName = $out.fabricWorkspaceNameOut.value
+    if (-not $WorkspaceName) {
+      $WorkspaceName = $out.desiredFabricWorkspaceName.value
+    }
   } catch {}
 }
 ```
@@ -116,13 +127,15 @@ azd env get-values
 
 # View specific output
 azd env get-value fabricCapacityId
+azd env get-value fabricCapacityModeOut
+azd env get-value fabricWorkspaceModeOut
 azd env get-value aiSearchName
 ```
 
 ## Related Files
 
-- **Infrastructure**: `/infra/main-orchestrator.bicep` (lines 313-349)
-- **Parameters**: `/infra/main-orchestrator.bicepparam`
+- **Infrastructure**: `/infra/main.bicep`
+- **Parameters**: `/infra/main.bicepparam`
 - **Automation Workflow**: `/azure.yaml` (postprovision hooks)
 - **Scripts**: `/scripts/automationScripts/`
 

docs/examples/azure.yaml.private-networking.sample

@@ -20,19 +20,7 @@ hooks:
       interactive: false
       continueOnError: false
 
-    # Step 1: Create or update the Fabric private link service resource in Azure
-    - shell: pwsh
-      run: ./scripts/automationScripts/FabricWorkspace/SecureWorkspace/create_fabric_private_link_service.ps1
-      interactive: false
-      continueOnError: false
-
-    # Step 2: Create/refresh the private endpoint and DNS configuration in the target VNet
-    - shell: pwsh
-      run: ./scripts/automationScripts/FabricWorkspace/SecureWorkspace/setup_workspace_private_endpoint.ps1
-      interactive: false
-      continueOnError: false
-
-    # Step 3: Lock down the Fabric workspace so that only private endpoints are allowed
+    # Step 1: Lock down the Fabric workspace so that only private endpoints are allowed
     - shell: pwsh
       run: ./scripts/automationScripts/FabricWorkspace/SecureWorkspace/enable_fabric_workspace_inbound_protection.ps1
       interactive: false

docs/fabric-onelake-private-networking.md

@@ -38,7 +38,7 @@ When deploying AI Search, AI Foundry, and Purview within a VNet (as configured i
 │  │  │  Lakehouse   │  │  Lakehouse   │  │  Lakehouse   │     │ │
 │  │  └──────────────┘  └──────────────┘  └──────────────┘     │ │
 │  │                                                              │ │
-│  │  Private Link Resource: privateLinkServicesForFabric        │ │
+│  │  Private Link Resource: Service-managed (no ARM RP)         │ │
 │  └─────────────────────────────────────────────────────────────┘ │
 └───────────────────────────────────────────────────────────────────┘
 ```
@@ -47,12 +47,12 @@ When deploying AI Search, AI Foundry, and Purview within a VNet (as configured i
 
 ### 1. Fabric Workspace Private Links
 
-Microsoft Fabric supports **workspace-level private links** that enable secure, private connectivity from Azure VNets to specific Fabric workspaces and their OneLake lakehouses.
+Microsoft Fabric supports **workspace-level private links** that enable secure, private connectivity from Azure VNets to specific Fabric workspaces and their OneLake lakehouses. This is a **service-managed** experience: there is currently **no customer-facing ARM/Bicep resource provider** to deploy or manage these objects.
 
 > **Important:** As of November 2025 Azure AI Search cannot complete a shared private link where `group-id = "workspace"`. Our automation detects the failure message `Cannot create private endpoint for requested type 'workspace'` and skips the shared private link stage so OneLake indexers continue to work over public endpoints. Follow the steps in [Phase 2](#phase-2-configure-shared-private-link-from-ai-search-automated) to re-run the script when Microsoft enables the feature, and keep the workspace communication policy in **Allow** mode until the link can be provisioned.
 
-- **Resource Provider**: `Microsoft.Fabric/privateLinkServicesForFabric`
-- **Target Subresource**: `workspace` (workspace-specific) or `tenant` (tenant-wide)
+- **Resource Provider**: _Not available in ARM/Bicep; service-managed only_
+- **Target Subresource**: `workspace` (workspace-specific) or `tenant` (tenant-wide) — managed by the service
 - **Workspace FQDN Format**: `https://{workspaceid}.z{xy}.blob.fabric.microsoft.com`
   - `{workspaceid}` = Workspace GUID without dashes
   - `{xy}` = First two characters of workspace GUID
@@ -83,7 +83,7 @@ Private DNS zones are required to resolve Fabric workspace FQDNs to private IPs:
 
 ### Phase 1: Enable Fabric Workspace Private Link (Manual - Post-Deployment)
 
-> **Note**: Fabric workspace private links cannot be configured via ARM/Bicep as of October 2025. This must be done manually after workspace creation.
+> **Note**: Fabric workspace private links cannot be configured via ARM/Bicep. This must be done manually after workspace creation in the Fabric portal.
 
 1. **Create Fabric Workspace** (via postprovision script: `create_fabric_workspace.ps1`)
 
@@ -95,79 +95,17 @@ Private DNS zones are required to resolve Fabric workspace FQDNs to private IPs:
 
 > **Note**: Once Microsoft enables workspace-targeted shared private links, the connection from AI Search should auto-approve because both resources live in the same subscription/tenant. Until then, the script will exit with a warning and no shared private link is created.
 
-### Phase 2: Configure Shared Private Link from AI Search (Automated)
+### Phase 2: Configure Shared Private Link from AI Search (Not yet supported)
 
-This is handled by the Bicep infrastructure in **Stage 7: Fabric Private Networking** and the **`setup_fabric_private_link.ps1`** postprovision script.
-
-**Resources created (when supported)**:
-1. Private DNS zones for Fabric endpoints
-2. DNS zone virtual network links
-3. Shared private link from AI Search to Fabric workspace (via PowerShell script)
+Workspace-targeted shared private links from AI Search are not supported today; the automation path is disabled/no-op to avoid failures. Keep workspace communication policy in **Allow** mode until Microsoft enables the feature, then re-enable private access manually.
 
 **RBAC tip:** Add Azure AD group object IDs to the `aiSearchAdditionalAccessObjectIds` parameter (or `azd env set aiSearchAdditionalAccessObjectIds "<objectId>"`) so interactive users inherit the same Search roles that the automation assigns to managed identities.
 
-**Key Benefits of Automatic Approval**:
-- ✅ **No manual approval needed** - Connection is auto-approved because both resources are in the same subscription/tenant
-- ✅ **Consistent with other private endpoints** - Works like Storage, Cosmos DB, AI Search private endpoints
-- ✅ **Faster deployment** - No waiting for manual approval step
-- ✅ **Production-ready** - Fully automated end-to-end
-
-**Bicep Configuration** (Stage 7):
-```bicep
-// Private DNS zones created
-resource analysisDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01'
-resource capacityDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01'
-resource powerQueryDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01'
-
-// VNet links for DNS resolution
-resource analysisVnetLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01'
-```
-
-**PowerShell Script** (`setup_fabric_private_link.ps1`):
-```powershell
-# Step 1: Automatically creates shared private link with same-subscription auto-approval
-az search shared-private-link-resource create \
-  --resource-group <rg-name> \
-  --service-name <search-name> \
-  --name fabric-workspace-link \
-  --group-id workspace \
-  --resource-id <fabric-workspace-resource-id>
-
-# Connection status will be "Approved" automatically (2-3 minutes provisioning time) once Azure supports workspace shared private links
-
-# Step 2: Configure workspace to deny public access (allow only private link connections)
-$policyBody = @{
-  inbound = @{
-    publicAccessRules = @{
-      defaultAction = "Deny"
-    }
-  }
-} | ConvertTo-Json
-
-Invoke-RestMethod `
-  -Uri "https://api.fabric.microsoft.com/v1/workspaces/$workspaceId/networking/communicationPolicy" `
-  -Headers $headers `
-  -Method Put `
-  -Body $policyBody `
-  -ContentType 'application/json'
-
-# Policy takes effect in up to 30 minutes
-```
-
-**What Gets Automated (once the platform supports workspace shared private links)**:
-1. ✅ Shared private link creation (AI Search → Fabric)
-2. ✅ Automatic approval (same subscription/tenant)
-3. ✅ Workspace communication policy (deny public access)
-4. ✅ Verification of connection status
-
-**Current Behavior (End of 2025)**:
-- ⚠️ Shared private link creation fails with `Cannot create private endpoint for requested type 'workspace'`
-- ⚠️ Script logs a warning and skips the shared private link stage
-- ✅ Workspace remains in **Allow** mode so indexing continues over public endpoints
-- ✅ You can re-run the script after Microsoft releases support; no additional changes required
-
-**Remaining Manual Step** (one-time):
-- Enable workspace-level private link in Fabric portal (required before shared private link can be created)
+**Current Behavior**:
+- ⚠️ Workspace shared private link creation fails with `Cannot create private endpoint for requested type 'workspace'`.
+- ⚠️ Scripts that tried to deploy `Microsoft.Fabric/privateLinkServicesForFabric` are now disabled and act as no-ops.
+- ✅ Keep workspace in **Allow** mode so indexing continues over public endpoints until Microsoft delivers a supported path.
+- ✅ You can re-run the connectivity scripts later if the platform exposes a supported resource provider.
 
 ### Phase 3: Configure OneLake Data Source (Automated Script)