ci: fix deployment pipeline, integrate deployment pipeline with test automation pipeline

[Harsh-Microsoft]

Purpose

This pull request introduces significant updates to the CI/CD pipeline and test automation workflows, focusing on improving deployment, resource management, and test execution. Key changes include transitioning from Windows-based runners to Ubuntu, enhancing resource cleanup processes, and simplifying test automation by removing unnecessary steps.

CI/CD Pipeline Enhancements:

• Renamed the workflow from "Validate Deployment" to "Deploy-Test-Cleanup Pipeline" and switched the runner from windows-latest to ubuntu-latest. Added environment variables (GPT_CAPACITY and TEXT_EMBEDDING_CAPACITY) to dynamically configure resource capacities. [1] [2]
• Replaced PowerShell-based installation commands for Azure CLI, Kubernetes CLI, and Helm with Bash-based equivalents for better compatibility with Ubuntu runners.
• Enhanced deployment validation with retry logic, improved error handling, and added steps to extract and validate the Web App URL and increase TPM for Azure OpenAI models.
• Introduced a cleanup job to delete resource groups and purge Azure OpenAI and Document Intelligence resources after deployment, ensuring proper resource management. [1] [2]

Test Automation Workflow Simplifications:

• Changed the workflow trigger from push and schedule to workflow_call, allowing the test automation workflow to be invoked directly with inputs like DKM_URL. Removed Azure CLI login and AKS start/stop steps to streamline the workflow. [1] [2] [3]
• Added steps to open the provided URL and introduced a sleep interval before running tests to ensure readiness.

Deployment Script Fix:

• Corrected quota check logic in Deployment/checkquota.ps1 to properly handle cases where quota information for a model is unavailable, preventing unnecessary continuation of the loop.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information

[github-actions]

🎉 This PR is included in version 1.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀