Organically manage test case hash changes and implementation data

[jeeyyy]

Problem Statement

Currently the unique id (hashes) for the test cases are generated using https://www.npmjs.com/package/object-hash.

This hash is dependent on 3 arguments: See code

• code
• lang
• rule id

Given an example like below:

<p>Some text</p>

• code - is what is within the backticks.
• html - is the lang (defaults to html).
• rule id - is the rule id under which the example is listed.

Given all the implementations are dependent on the test case id, any change in one of the above arguments will change the test case id, thus making the implementation redundant.

Objective

Is to conserve the implementation data from the past test cases change, without having to mandate all implementers to submit new data immediately, when the hash computation arguments have to be changed.

Solution 1

Unfortunately given the nature of work, the hash computation arguments has to be revisited often.

Given a scenario say we have 2 implementations (IMPL_1 & IMPL_2)for a given Rule (RULE_1).

• IMPL_1 (last updated 1st Jan 2020)
• IMPL_2 (last updated 2nd Jan 2020)

It would be nice to preserve the implementation data if the hash computation arguments are changed say on 15th Jan 2020.

One way to solve for this would be to preserve all previous ids of the test cases in the metadata that gets generated see testscases.json.

Introducing a key value pair like below:

   {
      "testcaseId": "9a417788dfd68b83820b01deb71e427f8d8edc3a",
      "previousTestcaseIds": [
        "abc",
        "xyz"
      ],
      "testcaseTitle": "Passed Example 1",
      "url": "https://act-rules.github.io/testcases/5f99a7/9a417788dfd68b83820b01deb71e427f8d8edc3a.html",
      "relativePath": "testcases/5f99a7/9a417788dfd68b83820b01deb71e427f8d8edc3a.html",
      "expected": "passed",
      "ruleId": "5f99a7",
      "ruleName": "`aria-*` attribute is defined in WAI-ARIA",
      "rulePage": "https://act-rules.github.io/rules/5f99a7",
      "ruleAccessibilityRequirements": null
    },

My preserving the previous id's all older not up to date implementations can be reconciled, meaning if a current implementation does not exist for testcaseId, an older implementation data that is available for one of the id's from previousTestcaseIds can be used. This will also allow to show on the website if an implementation is out of date perhaps.

Note: This requires change in how the test cases meta data are generated and in the implementation mapper logic & also in the website.

Some related issues/ discussions:

• A significant number test files have unrelated accessibility violations act-rules.github.io#1165

[jeeyyy]

@WilcoFiers I believe when we discussed this, we thought of an idea to introduce timestamps/ Date to preserve older implementations, but when writing the proposal, I thought managing older hashes for a test case would be a better solution. Thoughts welcome.

Asides looking for input from other contributors too. Thank you