RFC: rm __array__, add __buffer__

[ev-br]

Try getting rid of __array__, replace with the buffer protocol's __buffer__.

cross-ref #67, #69

This does not seem to fix gh-102. scipy tests pass locally.
Replacing __array__ with __buffer__ makes code marginally simpler. Effectively bumps the python requirement to 3.12+ though.
So maybe this is a temp solution until dlpack matures.
¯\_(ツ)_/¯

[ev-br]

The matching SciPy PR: scipy/scipy#22189

[ev-br]

The current behavior: np.asarray(...) just works on python 3.12+, and errors out otherwise.
I'm going to merge this for the upcoming release, unless there are suggestions to the contrary.

On python 3.12:

In [3]: import sys; print(sys.version)
3.12.4 | packaged by conda-forge | (main, Jun 17 2024, 10:23:07) [GCC 12.3.0]

In [4]: np.array(xp.ones(3))
Out[4]: array([1., 1., 1.])

In [5]: np.ones(3) + xp.ones(3)     # can't do anything about
Out[5]: array([2., 2., 2.])

On python 3.11:

In [1]: import sys; print(sys.version)
3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:27:40) [GCC 11.3.0]

In [2]: import array_api_strict as xp

In [3]: import numpy as np

In [4]: np.array(xp.ones(3))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 np.array(xp.ones(3))

File ~/repos/array-api-strict/array_api_strict/_array_object.py:170, in Array.__array__(self, *args, **kwds)
    168 def __array__(self, *args, **kwds):
    169     # a stub for python < 3.12; otherwise numpy silently produces object arrays
--> 170     raise TypeError(
    171         f"Interoperation with NumPy requires python >= 3.12. Please upgrade."
    172     )

TypeError: Interoperation with NumPy requires python >= 3.12. Please upgrade.

In [5]: xp.ones(3) + np.ones(3)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 xp.ones(3) + np.ones(3)

File ~/repos/array-api-strict/array_api_strict/_array_object.py:170, in Array.__array__(self, *args, **kwds)
    168 def __array__(self, *args, **kwds):
    169     # a stub for python < 3.12; otherwise numpy silently produces object arrays
--> 170     raise TypeError(
    171         f"Interoperation with NumPy requires python >= 3.12. Please upgrade."
    172     )

TypeError: Interoperation with NumPy requires python >= 3.12. Please upgrade.

[ev-br]

PSA: The timeline for this PR is "land in a couple of month's time". The current thinking is that short-term we work on the 2024.12 support in version 2.3, let the dust settle, and then merge this PR. Downstream testing with array-api-strict will then need to use python >= 3.12.

[ev-br]

Branch protection strikes again: could you please flip testing on python 3.10 and 3.11 to optional @kgryte @rgommers ?

[ev-br]

Alternatively, we can make a 2.4 release with what is now in main (#156), and postpone the __array__ story to a dedicated release.

Locally, this branch causes no problems for SciPy (scipy/scipy#23144 adds small fixes from local testing), and scikit-learn/scikit-learn#31517 fixes scikit-learn.

[betatim]

Should we remove this comment as well or edit it to mention that it is referring to __array__ - because "this was implemented ..." will be weird. Not sure we need to keep it/combine it with the comment below for __buffer__, so delete?

[ev-br]

Indeed! Rephrased.

[betatim]

For my education: why not use return self._array.__buffer__(flags) - aka delegate to numpy for this? And the same for __release_buffer__

[ev-br]

This is a great idea, actually. I'm going to make this change and rerun scipy and scikit-learn tests just to be sure.

The __release_buffer__ story is a bit mysterious:

In [11]: np.asarray(xp.arange(3))
Out[11]: array([0, 1, 2])

In [12]: np.sin(xp.arange(3))
Exception ignored in: Array([0, 1, 2], dtype=array_api_strict.int64)
Traceback (most recent call last):
  File "/home/br/repos/array-api-strict/array_api_strict/_array_object.py", line 173, in __release_buffer__
    self._array.__release_buffer__()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'numpy.ndarray' object has no attribute '__release_buffer__'
Out[12]: array([0.        , 0.84147098, 0.90929743])

In [13]: np.__version__
Out[13]: '1.26.4'

Same behavior on numpy-dev, so it's not something that got fixed in the meantime. OTOH, calling numpy ufuncs on array-api-strict objects is not really correct without an explicit prior np.asarray call, so probably it's not something we need to worry about.

[ev-br]

Per discussion with @seberg on numpy's slack,

Normally all you need is
def __buffer__(self, flags):
return self.data.__buffer__(flags)
and nothing else, in particular no __release_buffer__

this PR now does precisely that: use Tim's suggestion for self._array.__buffer__(flags) and does not define __release_buffer__ at all.

SciPy and scikit-learn tests still pass locally.

[lucascolley]

Alternatively, we can make a 2.4 release with what is now in main (#156), and postpone the array story to a dedicated release.

either way sounds fine to me. 2 separate releases is obviously 'nicer' but happy to trust your judgement as things seem promisingly green!