[CI] Let's not run quite that many tests

[Yoric]

We're running tests on 3 versions of Python x 3 OSes.

That takes lots of time. We should probably reduce that amount.