Skip to content

Commit e98f0b2

Browse files
fix: update GSSA, appinspect and minor updates (#476)
### Description - Appinspect CLI action bump to v2.11 - Java version for pre-commit updated - don't require GSSA execution for docs only changes. ### Checklist - [x] `README.md` has been updated or is not required - [ ] push trigger tests - [ ] manual release test - [ ] automated releases test - [ ] pull request trigger tests - [ ] schedule trigger tests - [ ] workflow errors/warnings reviewed and addressed ### Testing done splunk/splunk-add-on-for-microsoft-office-365#932 splunk/splunk-add-on-for-okta-identity-cloud#420
2 parents efced57 + d854681 commit e98f0b2

File tree

2 files changed

+168
-14
lines changed

2 files changed

+168
-14
lines changed

.github/workflows/reusable-build-test-release.yml

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -468,6 +468,10 @@ jobs:
468468
- uses: actions/setup-python@v5
469469
with:
470470
python-version: ${{ env.PYTHON_VERSION }}
471+
- uses: actions/setup-java@v5
472+
with:
473+
distribution: 'temurin'
474+
java-version: '17'
471475
- uses: pre-commit/action@v3.0.1
472476

473477
review_secrets:
@@ -799,7 +803,7 @@ jobs:
799803
name: package-splunkbase
800804
path: build/package/
801805
- name: Scan
802-
uses: splunk/appinspect-cli-action@v2.10
806+
uses: splunk/appinspect-cli-action@v2.11
803807
with:
804808
app_path: build/package/
805809
included_tags: ${{ matrix.tags }}
@@ -2932,8 +2936,8 @@ jobs:
29322936
run: |
29332937
# GS Scorecard: must have run for PRs to main (result doesn't matter, but skipped = blocked)
29342938
GS_RESULT=$(echo "$NEEDS" | jq -r '.["run-gs-scorecard"].result')
2935-
if [[ "${{ github.event_name }}" == "pull_request" ]] && [[ "${{ github.base_ref }}" == "main" ]] && [[ "$GS_RESULT" == "skipped" ]]; then
2936-
echo "::error::GS Scorecard is required for PRs to main. Add the 'execute_gs_scorecard' label and re-run the workflow."
2939+
if [[ "${{ github.event_name }}" == "pull_request" ]] && [[ "${{ github.base_ref }}" == "main" ]] && [[ "$GS_RESULT" == "skipped" ]] && [[ "${{ needs.check-docs-changes.outputs.docs-only }}" != "true" ]]; then
2940+
echo "::error::GS Scorecard is required for merging PRs to main. To unblock merging please add the 'execute_gs_scorecard' label and re-run the workflow."
29372941
echo "## GS Scorecard Required" >> "$GITHUB_STEP_SUMMARY"
29382942
echo "Add the \`execute_gs_scorecard\` label to this PR and re-run the workflow. GS Scorecard must complete before merging to main (result does not need to pass)." >> "$GITHUB_STEP_SUMMARY"
29392943
echo "run-publish=false" >> "$GITHUB_OUTPUT"

README.md

Lines changed: 161 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,11 @@
99
* [Spec reusable-build-test-release](#spec-reusable-build-test-release)
1010
* [Workflow Inputs](#workflow-inputs)
1111
* [General troubleshooting](#general-troubleshooting)
12+
* [[Job] validate-custom-version](#job-validate-custom-version)
13+
* [[Job] check-splunktafunctionaltests-exists](#job-check-splunktafunctionaltests-exists)
14+
* [[Job] check-docs-changes](#job-check-docs-changes)
1215
* [[Job] setup-workflow](#job-setup-workflow)
16+
* [[Job] validate-pr-title](#job-validate-pr-title)
1317
* [[Job] meta](#job-meta)
1418
* [[Job] fossa-scan](#job-fossa-scan)
1519
* [[Job] fossa-test](#job-fossa-test)
@@ -24,10 +28,13 @@
2428
* [[Job] setup](#job-setup)
2529
* [[Job] test-unit-python3](#job-test-unit-python3)
2630
* [[Job] run-btool-check](#job-run-btool-check)
31+
* [[Job] run-spl2-tests](#job-run-spl2-tests)
2732
* [[Job] run-knowledge-tests](#job-run-knowledge-tests)
2833
* [[Job] run-ui-tests](#job-run-ui-tests-)
2934
* [[Job] run-modinput-tests](#job-run-modinput-tests-)
3035
* [[Job] run-ucc-modinput-tests](#job-run-ucc-modinput-tests-)
36+
* [[Job] run-upgrade-tests](#job-run-upgrade-tests)
37+
* [[Job] run-scripted-input-tests-full-matrix](#job-run-scripted-input-tests-full-matrix)
3138
* [[Job] pre-publish](#job-pre-publish)
3239
* [[Job] publish](#job-publish)
3340
* [Vendor Addon Matrix tests](#vendor-addon-matrix-tests)
@@ -168,21 +175,58 @@ gitGraph
168175

169176
# Spec reusable-build-test-release
170177
## Workflow Inputs
171-
* marker - list of markers used to paralelize modinput tests
172-
* ucc-modinput-marker - list of markers used to paralelize ucc modinput tests
173-
* ui_marker - list of markers used to paralelize ui tests
174-
* custom-version - version used for release on manual workflow trigger
175-
* execute-tests-on-push-to-release - enable tests on release branch - default false
176-
* k8s-enfironment - k8s environment for testing
177-
* k8s-manifests-branch - k8s-manifests branch for testing
178-
* scripted-inputs-os-list - list of OSes used for scripted inputs tests
178+
* `marker` - list of markers used to parallelize modinput tests
179+
* `ucc-modinput-marker` - list of markers used to parallelize ucc modinput tests
180+
* `ui_marker` - list of markers used to parallelize ui tests
181+
* `custom-version` - version used for release on manual workflow trigger (format: `x.x.x`)
182+
* `execute-tests-on-push-to-release` - enable tests on push to `release/*` branch - default `false`
183+
* `k8s-environment` - k8s environment for testing, choices: `production` (default) or `staging`
184+
* `k8s-manifests-branch` - k8s-manifests branch for testing, default `v4.0`
185+
* `scripted-inputs-os-list` - list of OSes used for scripted inputs tests (default includes ubuntu 16.04–24.04 and redhat 8.4–9.5)
186+
* `upgrade-tests-ta-versions` - list of TA versions (format `X.X.X`) used as starting points for upgrade tests; e.g. `['7.6.0', '7.7.0']`
187+
* `wfe-run-on-splunk-latest` - when `true` forces WFE tests to run only on the latest Splunk version; when `false` runs on all supported Splunk versions required for release; default `false`
188+
* `python-version` - Python version used for testing, default `3.9`
189+
* `gs-image-version` - version of the GS Scorecard Docker image, default `1.1`
190+
* `gs-version` - version of the GS Scorecard tool, default `0.3`
179191

180192
## General troubleshooting
181193

182194
* For each stage there are logs which provides list of failures or link the test report for the stage or more details like error code regarding what caused the stage to fail.
183195
* Check if there is any similar issue reported to GitHub repo for the action by other users.
184196
* If you are not sure what to do, please use `go/addon/help`.
185197

198+
## [Job] validate-custom-version
199+
200+
**Description:**
201+
202+
- Runs only when `custom-version` input is provided (manual `workflow_dispatch` trigger).
203+
- Validates that the value matches semantic version format `X.X.X` and that the version tag does not already exist in the repository.
204+
205+
**Pass/fail behaviour:**
206+
207+
- Fails if the version format is invalid or the tag already exists, preventing duplicate releases.
208+
209+
**Artifacts:**
210+
211+
- No additional artifacts.
212+
213+
214+
## [Job] check-splunktafunctionaltests-exists
215+
216+
**Description:**
217+
218+
- Checks whether the deprecated `splunktafunctionaltests` package is referenced in `poetry.lock` or `dev_deps/requirements_dev.txt`.
219+
- This package must **not** be used for modinput tests. See the [Slack notice](https://splunk.slack.com/archives/C081JT7R69Z/p1754662758743839) for context.
220+
221+
**Pass/fail behaviour:**
222+
223+
- Fails with a warning if `splunktafunctionaltests` is detected, blocking the pipeline until the dependency is removed.
224+
225+
**Artifacts:**
226+
227+
- No additional artifacts.
228+
229+
186230
## [Job] check-docs-changes
187231

188232
**Description:**
@@ -206,14 +250,47 @@ gitGraph
206250
* triggering event is workflow_dispatch (used to create custom release version)
207251
* schedule event (controlled from [here](https://github.com/splunk/addonfactory-repository-template/blob/main/tools/jinja_parameters.yml))
208252
* To trigger specific test type
209-
* add to PR one or multiple labels, available choices can be found [here](https://github.com/splunk/addonfactory-workflow-addon-release/blob/4f3fa4d779b6ec7649f0dc6b973eb4d68e5fcc48/.github/workflows/reusable-build-test-release.yml#L153)
210-
* there is no need to add labels when PR's target branch is `main`
253+
* add to PR one or multiple of the following labels:
254+
* `execute_unit` - unit tests
255+
* `execute_knowledge` - knowledge / KO tests
256+
* `execute_spl2` - SPL2 tests
257+
* `execute_ui` - UI tests
258+
* `execute_modinput_functional` - modinput functional tests
259+
* `execute_ucc_modinput_functional` - UCC modinput functional tests
260+
* `execute_scripted_inputs` - scripted inputs tests
261+
* `execute_upgrade` - upgrade tests (not run on push to `main` or scheduled runs)
262+
* `execute_gs_scorecard` - Gold Standard Scorecard (always runs on push to `main`)
263+
* `execute_all_tests` - all available test types
264+
* `use_labels` - when combined with a PR to `main`, enables label-based selective test execution instead of running all tests
265+
* `exit-first` - stop pytest execution on the first failure (`-x` flag)
266+
* there is no need to add labels when PR's target branch is `main` (all tests except upgrade tests run by default)
267+
268+
## [Job] validate-pr-title
269+
270+
**Description:**
271+
272+
- Runs only on pull request events.
273+
- Validates the PR title conforms to the [Conventional Commits](https://www.conventionalcommits.org/) specification, which is required for semantic release to correctly determine the next version number.
274+
- Also validates that a single commit PR has a matching commit message.
275+
276+
**Action used:** https://github.com/amannn/action-semantic-pull-request
277+
278+
**Pass/fail behaviour:**
279+
280+
- Fails if the PR title does not follow conventional commit format (e.g. `feat:`, `fix:`, `chore:`, etc.) or if it is marked as WIP.
281+
282+
**Artifacts:**
283+
284+
- No additional artifacts.
285+
211286

212287
## [Job] meta
213288

214289
**Description:**
215290

216291
- Determines which Splunk and SC4S versions to run tests with.
292+
- Outputs matrices for supported and latest Splunk versions, SC4S versions, and vendor matrices for modinput/UI tests.
293+
- On schedule events, always uses latest Splunk only. On PRs to `main` or push to `main`/`develop`, uses the full supported matrix (unless overridden by `wfe-run-on-splunk-latest` input).
217294

218295
## [Job] fossa-scan
219296

@@ -551,7 +628,7 @@ appinspect-api-html-report-self-service
551628
- `GH_TOKEN_ADMIN` and `SA_GH_USER_NAME` for GitHub access
552629
- `SPL_COM_USER` and `SPL_COM_PASSWORD` for AppInspect integration
553630

554-
- Check that the Docker image version specified in `GS_SCORECARD_VERSION` environment variable exists in the ECR registry.
631+
- Check that the Docker image version specified via the `gs-image-version` workflow input (`GS_IMAGE_VERSION` env var, default `1.1`) exists in the ECR registry. The GS Scorecard tool version is controlled separately via `gs-version` input (`GS_VERSION` env var, default `0.3`).
555632

556633
- Review the job logs for specific error messages from the GS Scorecard tool.
557634

@@ -617,6 +694,25 @@ gs-scorecard-report (gs_scorecard.html)
617694
btool-output.txt
618695
```
619696

697+
## [Job] run-spl2-tests
698+
699+
**Description:**
700+
701+
- Executes SPL2 tests for add-ons that ship SPL2 modules under `package/default/data/spl2/`.
702+
- Runs inside the `ghcr.io/splunk/spl2-testing-base:latest` container using the `spl2_tests_run` CLI.
703+
- Triggered when `package/default/data/spl2/` directory is detected in the repository.
704+
705+
**Pass/fail behaviour:**
706+
707+
- Fails if any SPL2 test case fails.
708+
709+
**Artifacts:**
710+
711+
```
712+
Junit XML test report
713+
```
714+
715+
620716
## [Job] run-knowledge-tests
621717

622718
**Description:**
@@ -776,6 +872,60 @@ splunk-add-on-ucc-modinput-test-functional.log
776872
Junit XML file
777873
```
778874

875+
## [Job] run-upgrade-tests
876+
877+
**Description:**
878+
879+
- Executes upgrade tests to verify that the TA can be upgraded from a previous version to the current build without issues.
880+
- Runs a matrix of Splunk versions × vendor versions × TA versions supplied via the `upgrade-tests-ta-versions` workflow input.
881+
- Uses the Argo Workflow Engine (WFE) on Kubernetes to run tests, same infrastructure as modinput and UI tests.
882+
- **Not** triggered on PRs to `main`, push to `main`, scheduled runs, or custom-version releases. Only triggered by the `execute_upgrade` label on a PR or by the `execute_all_tests` label on non-main PRs.
883+
884+
**Pass/fail behaviour:**
885+
886+
- Fails if any upgrade test scenario fails.
887+
888+
**Troubleshooting steps for failures if any:**
889+
890+
- Ensure `upgrade-tests-ta-versions` input is set with the correct list of prior TA versions (e.g. `['7.6.0', '7.7.0']`).
891+
- Review Argo workflow logs and test result XML artifacts for failure details.
892+
- Verify AWS credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION`) and k8s environment are properly configured.
893+
894+
**Artifacts:**
895+
896+
```
897+
Junit XML file
898+
argo-logs
899+
```
900+
901+
902+
## [Job] run-scripted-input-tests-full-matrix
903+
904+
**Description:**
905+
906+
- Executes scripted input tests across a matrix of Splunk versions × OS images.
907+
- The OS list is controlled by the `scripted-inputs-os-list` workflow input (default: Ubuntu 16.04–24.04 and Red Hat 8.4–9.5).
908+
- Tests are located under `tests/scripted_inputs/` and run on Kubernetes via the Argo Workflow Engine.
909+
- Uses `--hostname=spl --os-name=<os> --os-version=<version> -m script_input` test arguments.
910+
911+
**Pass/fail behaviour:**
912+
913+
- Fails if any scripted input test fails on any OS/Splunk combination.
914+
915+
**Troubleshooting steps for failures if any:**
916+
917+
- Review the test result XML and Argo logs artifacts for the specific OS/Splunk combination that failed.
918+
- Verify that credentials and AWS secrets are correctly configured.
919+
- Test can be run locally using the same OS Docker image listed in `scripted-inputs-os-list`.
920+
921+
**Artifacts:**
922+
923+
```
924+
Junit XML file
925+
argo-logs
926+
```
927+
928+
779929
## [Job] pre-publish
780930

781931
**Description:**

0 commit comments

Comments
 (0)