Systematic testing

[AntoinePrv]

Tests typically use the xt::xarray, sometimes xt::xtensor, but it turns there are some subtle differences between all available containers/expressions/views that leads to error when switching one for another.

A few issues have been caught only when reported by users, who then have to wait for another release to get the fix. For instance

• xt::variance() bug? xtensor-r#85
• problem calculating quantile on pytensor xtensor-python#286

Tests should be refactor and templated to systematically run with all

• xt::xtensor_fixed
• xt::xtensor
• xt::xarray
• A view
• An expression
• 1D, ND
• The Python equivalent
• The R equivalent
• The Julia equivalent

Since this may generate a lot of tests, we would need to have strategy to not run/compile all of them at the same time (perhaps that would be only in CI, in parallel).

I am unsure if testing with project in other repositories (language bindings) should be done here or there.
Depending if the fix needs to be in xtensor or in the other repository, there will always be a case or PR deadlock.

[JohanMabille]

A way to cover most of the use cases from the bindings is to test:

• with column-major order
• with a tensor whose shape is a vector of signed integers

This should catch 99% of the issues. Other issues would be in the downstream projects themselves.

[tdegeus]

I concur. I too have had my annoying small bugs or missing templates:

• xt::pad not working for pyarray input #2643
• Average: debugging bug #2569
• Bugfix: calling argmax(a, 0) with a.dimension() == 1 resulted in argmin(a, 0) #2561

That is also why I proposed

• Run tests on all xtensor objects #2511

What I was planning to do is to write a Python module with most of the xtensor's functions with types xtensor_fixed, xtensor, xarray, pytensor, and pyarray and then test every function against NumPy based on random input. That would cover at least part of what you list above. The only reason that I was hesitating is maintainability. How would we ensure that all possible functions are included?

[JohanMabille]

How would we ensure that all possible functions are included?

I'm not sure that we can; but that should not be a blocker. Also I'm not sure we need to tests with pytensor and pyarray (or that means the tests should leave in another repo), as long as we tests on "containers" with similar properties (like the signed int shapes).

[AntoinePrv]

@JohanMabille remember we also had that pytensor issue due to the fact that the shape was a view on the Python data.

[tdegeus]

An initial proposition for the Python binding : xtensor-stack/xtensor-python#288 . It would be great if we can discuss this a bit.

Furthermore:

• I'm a bit worried about some the timings... It seems that some of NumPy's function might be really quite faster, or there might be some huge blocker with allocation?
• The test is finding a first bug (that I stared debugging in Average: debugging bug #2569 )