You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
echo "::error::GS Scorecard is required for merging PRs to main. To unblock merging please add the 'execute_gs_scorecard' label and re-run the workflow."
echo "Add the \`execute_gs_scorecard\` label to this PR and re-run the workflow. GS Scorecard must complete before merging to main (result does not need to pass)." >> "$GITHUB_STEP_SUMMARY"
* k8s-manifests-branch - k8s-manifests branch for testing
178
-
* scripted-inputs-os-list - list of OSes used for scripted inputs tests
178
+
*`marker` - list of markers used to parallelize modinput tests
179
+
*`ucc-modinput-marker` - list of markers used to parallelize ucc modinput tests
180
+
*`ui_marker` - list of markers used to parallelize ui tests
181
+
*`custom-version` - version used for release on manual workflow trigger (format: `x.x.x`)
182
+
*`execute-tests-on-push-to-release` - enable tests on push to `release/*` branch - default `false`
183
+
*`k8s-environment` - k8s environment for testing, choices: `production` (default) or `staging`
184
+
*`k8s-manifests-branch` - k8s-manifests branch for testing, default `v4.0`
185
+
*`scripted-inputs-os-list` - list of OSes used for scripted inputs tests (default includes ubuntu 16.04–24.04 and redhat 8.4–9.5)
186
+
*`upgrade-tests-ta-versions` - list of TA versions (format `X.X.X`) used as starting points for upgrade tests; e.g. `['7.6.0', '7.7.0']`
187
+
*`wfe-run-on-splunk-latest` - when `true` forces WFE tests to run only on the latest Splunk version; when `false` runs on all supported Splunk versions required for release; default `false`
188
+
*`python-version` - Python version used for testing, default `3.9`
189
+
*`gs-image-version` - version of the GS Scorecard Docker image, default `1.1`
190
+
*`gs-version` - version of the GS Scorecard tool, default `0.3`
179
191
180
192
## General troubleshooting
181
193
182
194
* For each stage there are logs which provides list of failures or link the test report for the stage or more details like error code regarding what caused the stage to fail.
183
195
* Check if there is any similar issue reported to GitHub repo for the action by other users.
184
196
* If you are not sure what to do, please use `go/addon/help`.
185
197
198
+
## [Job] validate-custom-version
199
+
200
+
**Description:**
201
+
202
+
- Runs only when `custom-version` input is provided (manual `workflow_dispatch` trigger).
203
+
- Validates that the value matches semantic version format `X.X.X` and that the version tag does not already exist in the repository.
204
+
205
+
**Pass/fail behaviour:**
206
+
207
+
- Fails if the version format is invalid or the tag already exists, preventing duplicate releases.
208
+
209
+
**Artifacts:**
210
+
211
+
- No additional artifacts.
212
+
213
+
214
+
## [Job] check-splunktafunctionaltests-exists
215
+
216
+
**Description:**
217
+
218
+
- Checks whether the deprecated `splunktafunctionaltests` package is referenced in `poetry.lock` or `dev_deps/requirements_dev.txt`.
219
+
- This package must **not** be used for modinput tests. See the [Slack notice](https://splunk.slack.com/archives/C081JT7R69Z/p1754662758743839) for context.
220
+
221
+
**Pass/fail behaviour:**
222
+
223
+
- Fails with a warning if `splunktafunctionaltests` is detected, blocking the pipeline until the dependency is removed.
224
+
225
+
**Artifacts:**
226
+
227
+
- No additional artifacts.
228
+
229
+
186
230
## [Job] check-docs-changes
187
231
188
232
**Description:**
@@ -206,14 +250,47 @@ gitGraph
206
250
* triggering event is workflow_dispatch (used to create custom release version)
207
251
* schedule event (controlled from [here](https://github.com/splunk/addonfactory-repository-template/blob/main/tools/jinja_parameters.yml))
208
252
* To trigger specific test type
209
-
* add to PR one or multiple labels, available choices can be found [here](https://github.com/splunk/addonfactory-workflow-addon-release/blob/4f3fa4d779b6ec7649f0dc6b973eb4d68e5fcc48/.github/workflows/reusable-build-test-release.yml#L153)
210
-
* there is no need to add labels when PR's target branch is `main`
253
+
* add to PR one or multiple of the following labels:
*`execute_upgrade` - upgrade tests (not run on push to `main` or scheduled runs)
262
+
*`execute_gs_scorecard` - Gold Standard Scorecard (always runs on push to `main`)
263
+
*`execute_all_tests` - all available test types
264
+
*`use_labels` - when combined with a PR to `main`, enables label-based selective test execution instead of running all tests
265
+
*`exit-first` - stop pytest execution on the first failure (`-x` flag)
266
+
* there is no need to add labels when PR's target branch is `main` (all tests except upgrade tests run by default)
267
+
268
+
## [Job] validate-pr-title
269
+
270
+
**Description:**
271
+
272
+
- Runs only on pull request events.
273
+
- Validates the PR title conforms to the [Conventional Commits](https://www.conventionalcommits.org/) specification, which is required for semantic release to correctly determine the next version number.
274
+
- Also validates that a single commit PR has a matching commit message.
- Fails if the PR title does not follow conventional commit format (e.g. `feat:`, `fix:`, `chore:`, etc.) or if it is marked as WIP.
281
+
282
+
**Artifacts:**
283
+
284
+
- No additional artifacts.
285
+
211
286
212
287
## [Job] meta
213
288
214
289
**Description:**
215
290
216
291
- Determines which Splunk and SC4S versions to run tests with.
292
+
- Outputs matrices for supported and latest Splunk versions, SC4S versions, and vendor matrices for modinput/UI tests.
293
+
- On schedule events, always uses latest Splunk only. On PRs to `main` or push to `main`/`develop`, uses the full supported matrix (unless overridden by `wfe-run-on-splunk-latest` input).
-`GH_TOKEN_ADMIN` and `SA_GH_USER_NAME` for GitHub access
552
629
-`SPL_COM_USER` and `SPL_COM_PASSWORD` for AppInspect integration
553
630
554
-
- Check that the Docker image version specified in `GS_SCORECARD_VERSION` environment variable exists in the ECR registry.
631
+
- Check that the Docker image version specified via the `gs-image-version` workflow input (`GS_IMAGE_VERSION` env var, default `1.1`) exists in the ECR registry. The GS Scorecard tool version is controlled separately via `gs-version` input (`GS_VERSION` env var, default `0.3`).
555
632
556
633
- Review the job logs for specific error messages from the GS Scorecard tool.
- Executes upgrade tests to verify that the TA can be upgraded from a previous version to the current build without issues.
880
+
- Runs a matrix of Splunk versions × vendor versions × TA versions supplied via the `upgrade-tests-ta-versions` workflow input.
881
+
- Uses the Argo Workflow Engine (WFE) on Kubernetes to run tests, same infrastructure as modinput and UI tests.
882
+
-**Not** triggered on PRs to `main`, push to `main`, scheduled runs, or custom-version releases. Only triggered by the `execute_upgrade` label on a PR or by the `execute_all_tests` label on non-main PRs.
883
+
884
+
**Pass/fail behaviour:**
885
+
886
+
- Fails if any upgrade test scenario fails.
887
+
888
+
**Troubleshooting steps for failures if any:**
889
+
890
+
- Ensure `upgrade-tests-ta-versions` input is set with the correct list of prior TA versions (e.g. `['7.6.0', '7.7.0']`).
891
+
- Review Argo workflow logs and test result XML artifacts for failure details.
892
+
- Verify AWS credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION`) and k8s environment are properly configured.
893
+
894
+
**Artifacts:**
895
+
896
+
```
897
+
Junit XML file
898
+
argo-logs
899
+
```
900
+
901
+
902
+
## [Job] run-scripted-input-tests-full-matrix
903
+
904
+
**Description:**
905
+
906
+
- Executes scripted input tests across a matrix of Splunk versions × OS images.
907
+
- The OS list is controlled by the `scripted-inputs-os-list` workflow input (default: Ubuntu 16.04–24.04 and Red Hat 8.4–9.5).
908
+
- Tests are located under `tests/scripted_inputs/` and run on Kubernetes via the Argo Workflow Engine.
909
+
- Uses `--hostname=spl --os-name=<os> --os-version=<version> -m script_input` test arguments.
910
+
911
+
**Pass/fail behaviour:**
912
+
913
+
- Fails if any scripted input test fails on any OS/Splunk combination.
914
+
915
+
**Troubleshooting steps for failures if any:**
916
+
917
+
- Review the test result XML and Argo logs artifacts for the specific OS/Splunk combination that failed.
918
+
- Verify that credentials and AWS secrets are correctly configured.
919
+
- Test can be run locally using the same OS Docker image listed in `scripted-inputs-os-list`.
0 commit comments