TST: instability with scipy 1.13.0 on macOS arm64 (a suspect and spurious warning in fitting test)

[neutrinoceros]

With scipy 1.13.0 on macOS arm64, I'm seeing the following failure

pytest "astropy/modeling/tests/test_quantities_fitting.py::test_fitting_with_initial_values[TRFLSQFitter]"

Pytest log

========================= test session starts =========================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.5.0

Running tests with Astropy version 7.0.0.dev41+gdcf60561fc.
Running tests in astropy/modeling/tests/test_quantities_fitting.py::test_fitting_with_initial_values[TRFLSQFitter].

Date: 2024-04-22T10:36:02

Platform: macOS-14.4.1-arm64-arm-64bit

Executable: /Users/clm/.pyenv/versions/astropy.dev/bin/python

Full Python Version:
3.12.2 (main, Feb 22 2024, 09:53:39) [Clang 15.0.0 (clang-1500.1.0.2.5)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions:
Numpy: 1.26.4
Scipy: 1.13.0
Matplotlib: not available
h5py: not available
Pandas: not available
PyERFA: 2.0.1.4
Cython: not available
Scikit-image: not available
asdf-astropy: not available
pyarrow: not available

Using Astropy options: remote_data: none.

CI: undefined
ARCH_ON_CI: undefined
IS_CRON: undefined

rootdir: /Users/clm/dev/astropy-project/coordinated/astropy
configfile: pyproject.toml
plugins: astropy-0.11.0, cov-5.0.0, hypothesis-6.100.1, remotedata-0.4.1, filter-subpackage-0.2.0, doctestplus-1.2.1, astropy-header-0.2.2, arraydiff-0.6.1, xdist-3.5.0, mock-3.14.0
collected 1 item

astropy/modeling/tests/test_quantities_fitting.py F             [100%]

============================== FAILURES ===============================
___________ test_fitting_with_initial_values[TRFLSQFitter] ____________

fitter = <astropy.modeling.fitting.TRFLSQFitter object at 0x117907fb0>

    @pytest.mark.skipif(not HAS_SCIPY, reason="requires scipy")
    @pytest.mark.parametrize("fitter", fitters)
    def test_fitting_with_initial_values(fitter):
        fitter = fitter()

        x, y = _fake_gaussian_data()

        # Fit the data using a Gaussian with units
        g_init = models.Gaussian1D(amplitude=1.0 * u.mJy, mean=3 * u.cm, stddev=2 * u.mm)
>       g = fitter(g_init, x, y)

astropy/modeling/tests/test_quantities_fitting.py:114:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
astropy/modeling/fitting.py:286: in wrapper
    model_new = func(self, model, xdata, ydata, **kwargs)
astropy/modeling/fitting.py:1366: in __call__
    init_values, fitparams, cov_x = self._run_fitter(
astropy/modeling/fitting.py:1525: in _run_fitter
    self.fit_info = optimize.least_squares(
/Users/clm/.pyenv/versions/astropy.dev/lib/python3.12/site-packages/scipy/optimize/_lsq/least_squares.py:941: in least_squares
    result = trf(fun_wrapped, jac_wrapped, x0, f0, J0, lb, ub, ftol, xtol,
/Users/clm/.pyenv/versions/astropy.dev/lib/python3.12/site-packages/scipy/optimize/_lsq/trf.py:123: in trf
    return trf_bounds(
/Users/clm/.pyenv/versions/astropy.dev/lib/python3.12/site-packages/scipy/optimize/_lsq/trf.py:334: in trf_bounds
    step, step_h, predicted_reduction = select_step(
/Users/clm/.pyenv/versions/astropy.dev/lib/python3.12/site-packages/scipy/optimize/_lsq/trf.py:134: in select_step
    p_stride, hits = step_size_to_bound(x, p, lb, ub)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

x = array([ 0.91889636,  6.53513562, 11.59407301])
s = array([nan, nan, nan])
lb = array([          -inf,           -inf, 1.17549435e-38])
ub = array([inf, inf, inf])

    def step_size_to_bound(x, s, lb, ub):
        """Compute a min_step size required to reach a bound.

        The function computes a positive scalar t, such that x + s * t is on
        the bound.

        Returns
        -------
        step : float
            Computed step. Non-negative value.
        hits : ndarray of int with shape of x
            Each element indicates whether a corresponding variable reaches the
            bound:

                 *  0 - the bound was not hit.
                 * -1 - the lower bound was hit.
                 *  1 - the upper bound was hit.
        """
        non_zero = np.nonzero(s)
        s_non_zero = s[non_zero]
        steps = np.empty_like(x)
        steps.fill(np.inf)
        with np.errstate(over='ignore'):
            steps[non_zero] = np.maximum((lb - x)[non_zero] / s_non_zero,
                                         (ub - x)[non_zero] / s_non_zero)
        min_step = np.min(steps)
>       return min_step, np.equal(steps, min_step) * np.sign(s).astype(int)
E       RuntimeWarning: invalid value encountered in cast

/Users/clm/.pyenv/versions/astropy.dev/lib/python3.12/site-packages/scipy/optimize/_lsq/common.py:398: RuntimeWarning
======================= short test summary info =======================
FAILED astropy/modeling/tests/test_quantities_fitting.py::test_fitting_with_initial_values[TRFLSQFitter] - RuntimeWarning: invalid value encountered in cast
========================== 1 failed in 3.28s ==========================

This appears to be due to a problem in scipy's build process: it goes away if I build scipy 1.13.0 from source instead of picking up a wheel from PyPI. I reported it upstream (scipy/scipy#20531) but at the moment my reproducer is too close to the actual astropy test and not nearly minimal enough for anyone to understand what's going on in a reasonable amount of time. At this stage I cannot exclude that something in my own system is broken, and that building from source is merely a functional workaround, but I can't be sure. In any case, I expect the solution to this would be external to astropy, but I wanted to raise awareness on this kirk while I'm figuring it out.

If anyone could try to reproduce on another arm64 machine, it would help.

[neutrinoceros]

If anyone could try to reproduce on another arm64 machine, it would help.

Reproduced in CI: https://github.com/astropy/astropy/actions/runs/8781878745/job/24094808203?pr=16321

[neutrinoceros]

upstream patch incoming: scipy/scipy#20569

[pllim]

I think this is gone now?