owned this note
owned this note
Published
Linked with GitHub
# Scipy test suite inside Pyodide status
[toc]
## Test failures ##
From [2023-04-26 build log](https://github.com/lesteve/scipy-tests-pyodide/actions/runs/4805202066/jobs/8551386890) (after [Pyodide scipy 1.10.1 update](https://github.com/pyodide/pyodide/commit/24957acc139208a0a8113d185ae55dc9288f75f9))
### scipy.interpolate.tests 2 failures ###
- [ ] TestSplder.test_kink probably a floating point exception thing?
Failed: DID NOT RAISE <class 'ValueError'>
- [ ] TestNdPPoly.test_integrate_2d genuine one
AssertionError:
Not equal to tolerance rtol=1e-05, atol=1e-05
[(0, 1), (0, 1)]
Mismatched elements: 1 / 1 (100%)
Max absolute difference: 14.48024922
Max relative difference: 0.88911848
x: array(1.805825)
y: array(16.286074)
### scipy.optimize.tests 2 failures ###
- [ ] TestLinprogSimplexNoPresolve.test_bounds_infeasible_2 maybe floating point exception?
Failed: DID NOT WARN. No warnings of type (<class 'RuntimeWarning'>,) were emitted.
The list of emitted warnings is: [ DeprecationWarning("\`method='simplex'\` is deprecated and will be removed in SciPy 1.11.0. Please use one of the HiGHS solvers (e.g. \`method='highs'\`) in new code.")].
- [ ] TestFSolve.test_reentrant_Dfunc genuine one, maybe an issue with
reentrant and minpack, maybe same root cause as memory corruptions below?
AssertionError:
Arrays are not almost equal to 6 decimals
Mismatched elements: 4 / 4 (100%)
Max absolute difference: 1.
Max relative difference: 1.
x: array([2., 0., 2., 0.])
y: array([1., 1., 1., 1.])
### scipy.special.tests 3 failures ###
- [ ] TestExp1.test_branch_cut maybe float128 support issue since this is checking 0 and -0?
assert sc.exp1(complex(-1, 0)).imag == (
-sc.exp1(complex(-1, -0.0)).imag
)
- [ ] test_add_round_up and test_add_round_down not precise enough roughly 50_000 / 100_000 failures. Maybe lack of floating point support for controling rounding see https://github.com/WebAssembly/design/issues/1384?
### scipy.stats.tests 166 failures ###
- [ ] a few errors with test_stats_boost_ufunc, issue with float128 support?
AssertionError:
Not equal to tolerance rtol=6.16298e-33, atol=0
Mismatched elements: 1 / 1 (100%)
Max absolute difference: 5.62434056e-17
Max relative difference: 1.12486811e-16
x: array(0.5, dtype=float128)
y: array(0.5)
- [ ] test_continuous_basic.py::test_methods_with_lists[studentized_range-args95-ppf] brentq failed to converge
- [ ] test_continuous_basic.py::test_methods_with_lists[studentized_range-args95-isf] brentq failed to converge
- [ ] TestStudentizedRange.test_cdf_against_tables genuine difference genuine difference
- [ ] TestStudentizedRange.test_ppf_against_tables genuine difference in result ...
```
AssertionError:
Not equal to tolerance rtol=0.0001, atol=0
Mismatched elements: 1 / 1 (100%)
Max absolute difference: 0.84668791
Max relative difference: 0.89125043
x: array(0.103312)
y: array(0.95)
```
- [ ] plenty of TestStudentizedRange.test_cdf_against_mp genuine difference
- [ ] test_distributions.py::TestStudentizedRange::test_pdf_integration genuine difference
- [ ] test_distributions.py::TestStudentizedRange::test_cdf_against_r[r_case_result0] genuine difference
- [ ] test_distributions.py::TestStudentizedRange::test_cdf_against_r[r_case_result1] genuine difference
- [ ] test_distributions.py::TestStudentizedRange::test_infinite_df genuine difference
- [ ] test_distributions.py::TestStudentizedRange::test_clipping genuine difference
```
AssertionError:
Not equal to tolerance rtol=1e-07, atol=1e-10
Mismatched elements: 1 / 1 (100%)
Max absolute difference: 1.
Max relative difference: inf
x: array(1.)
y: array(0)
```
- [ ] test_hypotests.py::TestTukeyHSD::test_compare_sas[equal size sample] failed to converge brentq
- [ ] test_hypotests.py::TestTukeyHSD::test_compare_sas[unequal sample size] failed to converge brentq
- [ ] test_hypotests.py::TestTukeyHSD::test_compare_sas[extreme sample size differences] failed to converge brentq
- [ ] test_hypotests.py::TestTukeyHSD::test_compare_matlab[equal size sample] failed to converge brentq
- [ ] test_hypotests.py::TestTukeyHSD::test_compare_matlab[unequal size sample] failed to converge brentq
- [ ] test_hypotests.py::TestTukeyHSD::test_compare_r failed to converge brentq
- [ ] test_hypotests.py::TestTukeyHSD::test_engineering_stat_handbook failed to converge brentq
- [ ] test_hypotests.py::TestTukeyHSD::test_rand_symm failed to converge brentq
- [ ] test_hypotests.py::TestTukeyHSD::test_2_args_ttest genuine difference
- [ ] test_kdeoth.py::test_kde_1d returns unexpected nan
- [ ] test_kdeoth.py::test_kde_1d_weighted returns unexpected nan
- [ ] test_kdeoth.py::test_kde_2d returns unexpected nan
- [ ] test_kdeoth.py::test_kde_2d_weighted returns unexpected nan
- [ ] test_multivariate.py::TestCovariance::test_mvn_with_covariance_cdf[Precision-size1] genuine difference
- [ ] test_multivariate.py::TestMultivariateNormal::test_logcdf_default_values weird complex expected but got float with nan
```
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0
x and y nan location mismatch:
x: array([-71.138756+0.j , 44.505249+0.j , -21.370506+3.141593j,
40.381878+3.141593j, -43.079 +0.j ])
y: array([-71.138756, 44.505249, nan, nan, -43.079 ])
```
- [ ] test_multivariate.py::TestMultivariateNormal::test_broadcasting genuine difference genuine difference
- [ ] test_multivariate.py::TestMultivariateNormal::test_normal_1D genuine difference
- [ ] test_multivariate.py::TestMultivariateNormal::test_R_values genuine difference
- [ ] test_multivariate.py::TestMultivariateNormal::test_cdf_with_lower_limit_arrays
```
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0
Mismatched elements: 12 / 12 (100%)
Max absolute difference: 9.02229076e+30
Max relative difference: inf
x: array([[ 1.429774e-33, 3.752467e+21, -3.665021e-02],
[-2.194558e+12, -3.310962e-32, -2.226312e+05],
[-1.076296e-01, 2.651718e+11, -9.022291e+30],
[-3.928867e+11, 4.416284e+09, -4.572021e+13]])
y: array([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]])
```
- [ ] test_multivariate.py::TestMultivariateNormal::test_cdf_with_lower_limit_consistency genuine difference
```
AssertionError:
Not equal to tolerance rtol=0.0001, atol=0
Mismatched elements: 2 / 2 (100%)
Max absolute difference: 0.00197334
Max relative difference: 1.
x: array([-1.392729e-36, 6.393774e-37])
y: array([-9.969363e-32, 1.973338e-03])
```
- [ ] test_qmc.py::TestMultivariateNormalQMC::test_validations exception not raised maybe floating point exception?
- [ ] test_qmc.py::TestMultivariateNormalQMC::test_MultivariateNormalQMCDegenerate returns unexpected nan?
- [ ] test_stats.py::TestKSTwoSamples::test_some_code_paths maybe floating point exception
- [ ] test_stats.py::TestGeometricStandardDeviation::test_raises_value_error_zero_entry exception not raised maybe floating point exception?
- [ ] test_stats.py::TestGeometricStandardDeviation::test_raises_value_error_negative_entry exception not raised maybe floating point exception?
- [ ] test_stats.py::TestGeometricStandardDeviation::test_raises_value_error_inf_entry exception not raised maybe floating point exception?
- [ ] test_stats.py::TestBrunnerMunzel::test_brunnermunzel_normal_dist no warning maybe floating point exception?
- [ ] test_moments[skewnorm-arg93-True-True-False], returns nan (or expects nan) not 100% sure
AssertionError:
Arrays are not almost equal to 10 decimals
skewnorm - 2ndt moment
x and y nan location mismatch:
x: array(nan)
y: array(0.4008284495)
- [ ] ttest_methods_with_lists[studentized_range-args94-ppf] and ttest_methods_with_lists[studentized_range-args94-isf]
r = _zeros._brentq(f, a, b, xtol, rtol, maxiter, args, full_output, disp)
RuntimeError: Failed to converge after 100 iterations.
### Notes
np.float128 on my machine has resolution 1e-18 so this is likely 80-bit float (and not 128-bit float)
```
I[25]: np.finfo(np.float128)
O[25]: finfo(resolution=1e-18, min=-1.189731495357231765e+4932, max=1.189731495357231765e+4932, dtype=float128)
```
On the other hand, it seems like this is really float128 for Pyodide:
```
> np.finfo(np.float128)
ffo(resolution=1.000000000000000055967309976241902e-33, min=-1.189731495357231765085759326628007e+4932, max=1.189731495357231765085759326628007e+4932, dtype=float128)
```
Could this be the reason for some test failures?
## Pyodide fatal errors ##
See skipped tests in [conftest.py](https://github.com/lesteve/scipy-tests-pyodide/blob/main/conftest.py)
### Signature mismatch ###
- [ ] scipy.linalg test_blas.+test_complex_dotu
```py
from scipy.linalg import _fblas
_fblas.cdotu([3j, -4, 3-4j], [2, 3, 1])
```
stack-trace with debug symbols:
```
RuntimeError: null function or function signature mismatch
at f2py_rout__fblas_cdotu (001e2656:0x13468)
at fortran_call (0003e3ba:0x6642)
at method_call_trampoline (pyodide.asm.js:9750:33)
at _PyObject_MakeTpCall (pyodide.asm.wasm:0x1309ec)
at call_function (pyodide.asm.wasm:0x1ecb7f)
at _PyEval_EvalFrameDefault (pyodide.asm.wasm:0x1ea7d6)
at _PyEval_Vector (pyodide.asm.wasm:0x1e4de9)
at PyEval_EvalCode (pyodide.asm.wasm:0x1e41b5)
at builtin_eval (pyodide.asm.wasm:0x1e1558)
at cfunction_vectorcall_FASTCALL (pyodide.asm.wasm:0x16c465)
```
called with 6 arguments:
```
call_indirect (param i32 i32 i32 i32 i32 i32) (result i32)
```
But function has 8 arguments:
```
(func $f2pywrapcdotu_ (;227;) (export "f2pywrapcdotu_") (param $var0 i32) (param $var1 i32) (param $var2 i32) (param $var3 i32) (param $var4 i32) (param $var5 i32) (param $var6 i32) (param $var7 i32) (result i32)
```
- [ ] scipy.linalg test_cython_blas.+complex
- [ ] scipy.linalg test_lapack.py.+larfg_larf
https://github.com/pyodide/pyodide/issues/3379
- [ ] scipy/sparse test_propack
https://github.com/pyodide/pyodide/issues/3380 and a PR
https://github.com/pyodide/pyodide/pull/3381
### Memory corruption ###
- [ ] scipy.optimize test_minpack.py.+test_reentrant_func There is a
patch for minpack but I think this is because f2c can not deal with recursive
keyword (because it is not Fortran 77):
packages/scipy/patches/0003-fix-fotran-files-minpack.patch
This reproduces the memory corruption:
```py
import numpy as np
from numpy.testing import assert_, assert_array_almost_equal
from scipy import optimize
def pressure_network(flow_rates, Qtot, k):
P = k * flow_rates**2
F = np.hstack((P[1:] - P[0], flow_rates.sum() - Qtot))
return F
def test_pressure_network_no_gradient():
# fsolve without gradient, equal pipes -> equal flows.
k = np.full(4, 0.5)
Qtot = 4
initial_guess = np.array([2., 0., 2., 0.])
final_flows, info, ier, mesg = optimize.fsolve(
pressure_network, initial_guess, args=(Qtot, k),
full_output=True)
# print('no gradient', info, ier, mesg)
# assert_array_almost_equal(final_flows, np.ones(4))
# assert_(ier == 1, mesg)
def func(*args):
test_pressure_network_no_gradient()
return pressure_network(*args)
def test_reentrant_func():
# fsolve without gradient, equal pipes -> equal flows.
k = np.full(4, 0.5)
Qtot = 4
initial_guess = np.array([2., 0., 2., 0.])
final_flows, info, ier, mesg = optimize.fsolve(
func, initial_guess, args=(Qtot, k),
full_output=True)
# assert_array_almost_equal(final_flows, np.ones(4))
# assert_(ier == 1, mesg)
# print('reentrant', info, ier, mesg)
for i in range(50):
print(i)
test_reentrant_func()
# print('no gradient', info, ier, mesg)
```
- [ ] scipy.stats test_resampling.+TestMonteCarloHypothesisTest.+test_against_anderson.+logistic
memory_corruption_msg. It seems like method='hybr' the default use minpack so
minpack may somehow sometimes corrupt memory?
https://docs.scipy.org/doc/scipy/reference/optimize.root-hybr.html#optimize-root-hybr
Seems like this does not converge, maybe because precision is not high
enough? only for logistic (not norm), maybe stats function through boost not precise
enough? Is this a scipy.optimize or a scipy.stats issue, or both? This
reproduces the issue most of the time:
```py
import numpy as np
from scipy import stats
from scipy.optimize import root
dist_name = 'logistic'
def fun(i):
def inner(a):
rng = np.random.default_rng(394295467)
x = stats.tukeylambda.rvs(a, size=100, random_state=rng)
expected = stats.anderson(x, dist_name)
ret = expected.statistic - expected.critical_values[i]
print(ret)
return ret
return inner
for i in range(5):
sol = root(fun(i), x0=0)
print(sol)
```
## Expected failures due to Pyodide limitations
See xfailed tests in [conftest.py](https://github.com/lesteve/scipy-tests-pyodide/blob/main/conftest.py).
## Other issues
Some tests need built extension, that are not built for now in Pyodide (see
this
[patch](https://github.com/pyodide/pyodide/blob/main/packages/scipy/patches/0010-skip-fortran-fails-to-link.patch)
for more details). These show up as tests collection error below.