Type UDT ops like built-in ops; fix UDT literals, casts and comparisons - #591

test_udt_layouts_under_one_name_get_their_own_jit_c_names set
jit_c_control to 'on' and compiled kernels for both layouts, so on a host
whose JIT cannot compile (no C compiler, or a broken one) it failed with
JitError instead of skipping. CI always installs a compiler, so CI never
saw it.

It now asks _enable_jit_for_udt, which probes with a real compile once
per process, and skips the kernel part when the answer is no. The C name
checks and the register_new check need no compiler, so they come first
and run everywhere.

Behavior fix. SuiteSparse exports the iso value of an array UDT with
shape (1, *subarray shape), and normalize_values broadcast it to
shape + values.shape, keeping that axis of length 1. With the
suitesparse backend, for an iso Vector or Matrix of an array UDT such
as INT64[4] (as v[:] = [0, 1, 2, 3] makes):

- Vector.to_dense: TypeError "unsupported operand type(s) for +: 'int'
  and 'tuple'" on main and before this commit (it passed its size as
  an int); now an array of shape (size, 4).
- Matrix.to_dense: shape (nrows, ncols, 1, 4); now (nrows, ncols, 4).
- Matrix.to_csr, to_csc, to_dcsr and to_dcsc: values of shape
  (nvals, 1, 4); now (nvals, 4).
- Matrix.to_dicts, which uses to_dcsr or to_dcsc: each value as
  [[0, 1, 2, 3]]; now [0, 1, 2, 3].

Vector.to_coo, Matrix.to_coo, the vanilla backend and records were
already right. normalize_values now broadcasts to
shape + values.shape[1:], and Vector.to_dense passes (size,).

CI found it through #652's test_udt_mixed_jit_c_name_is_its_own (17
jobs, e.g. job 122633175811): SuiteSparse before 10.5 makes a vector
built from equal values iso, so that test's constant inputs gave an
iso result, and to_dense raised. With 10.5 the vectors are not iso.

…rse 7

Behavior fix. register_anonymous passed a dtype's itemsize to
GrB_Type_new without a check. SuiteSparse 8 and later refuse a size of
0, but SuiteSparse 7 takes it as 1. lookup_dtype auto-registers what it
does not know, so on SuiteSparse 7 a str became the UDT "<U0", which
holds nothing.

- register_anonymous of str, bytes, np.dtype("U"), an empty record or
  a subarray of length 0: InvalidValue with no message on SuiteSparse
  8 and later, a UDT on 7; now ValueError "dtype must not have a size
  of 0" on every version.
- lookup_dtype(str): ValueError "Unknown dtype" on 8 and later, the
  "<U0" UDT on 7; now ValueError "Unknown dtype" on every version.
- Scalar.from_value("x"), Scalar + "x" and v.apply(binary.plus,
  right="x"): on SuiteSparse 7, ValueError "got more than 0 values to
  unpack" from cffi; now the TypeError they raise on 8 and later.

CI found it in wave 2 through #654's test_bad_ewise case (s1, "x"),
which expects the TypeError, in its one job with
python-suitesparse-graphblas 7.4.2 (122632718294).

SuiteSparse:GraphBLAS 10.5.2 fixed the bug _value_select_in_own_type
works around (DrTimothyAldenDavis/GraphBLAS#463): select with a VALUE*
op of another type than the input's now casts both to the op's type.
Its MAINT note said to delete the function once every supported version
has the fix, but that would change two results that
test_select_value_thunk_of_another_type pins: with the function
bypassed on 10.5.2, an INT64 thunk of 2**53 + 1 beside FP32 or FC32
values of 2**53 rounds to 2**53 in the op's type and keeps that element.
The other 61 cases of the test match. The note now says so, and gives
the versions with the bug as 8.0.2 through 10.5.1. Comments only.

…own type

Binding theta to an IndexBinaryOp of no type, op(theta), took the op's type
from theta alone, and the inputs were cast to it. Now op(theta) returns a
BinaryOp whose type is chosen each time it runs, as any other BinaryOp's is.
Theta is typed by its value when it is given, as apply types an IndexUnaryOp's
thunk: a Python int is INT64, a float FP64 and a bool BOOL, and a numpy scalar
or a Scalar keeps its type. A numpy array is typed as its array layout, as
other array literals are. The default theta, False, changes no type.

An op registered without is_udt runs at the type that holds both inputs and
theta. One with is_udt=True is compiled for the types of x, y and theta as
they are, and SuiteSparse gets theta's type as the op's theta_type, which
GxB_IndexBinaryOp_new takes apart from x's and y's. So theta is never
converted. The compiled ops are cached per (x, y, theta) types, and
op[X, Y, T] names all three; op[X, Y] and op[T] give theta y's type, as
before. A Semiring built from op(theta) is polymorphic in the same way.
op[T](theta) and op(theta, dtype=T) still bind at T. IndexBinaryOp has not
been released, so nothing is deprecated.

Behavior changes (before -> after):

- A.ewise_mult(B, op()) on INT64: a BOOL op, so every value became True ->
  INT64, the values op computes.
- u.ewise_mult(w, op(2)) on FP64: computed in INT64 (1.5 + 2 * 2.25 gave 5)
  -> FP64 (6.0). op(np.int8(2)): computed in INT8, where 300.0 became 127
  and wrapped -> FP64.
- A Semiring from op(0.5) with INT64, FP32 or INT8 matrices: KeyError ->
  computed in FP64, with or without is_udt.
- The SuiteSparse user guide's argmax (an is_udt multiplier that makes the
  pair (k, A(i, k)), theta unused) on a matrix of pairs: iop(), iop(False),
  iop(0) and iop(0.0) were ops on BOOL, INT64 or FP64 values, which failed
  to compile for the pairs -> all four run, compiled for (pair, BOOL, BOOL),
  (pair, BOOL, INT64) and (pair, BOOL, FP64).
- An is_udt op on INT64 data: op(0.5) or an FP32 Scalar theta made the op
  FP64 or FP32 and cast the inputs to it -> the inputs stay INT64, theta keeps
  its type, and Numba types the result (FP64 for x + y + theta).
- An is_udt op on int32[3] data: op(np.array([10, 20, 30], np.int32)) raised
  "Cannot infer a dtype for theta" -> theta is an int32[3] value; an int64
  array is an int64[3] one. op(5) gave GrB_DOMAIN_MISMATCH -> theta is an
  INT64. op((10, 20, 30)) still raises TypeError, now saying to give theta
  as a Scalar of its UDT; an op without is_udt raises KeyError for an array
  theta, as for a UDT Scalar.
- A JIT op (indexbinary.ss.register_new) is registered for its x, y and theta
  types. op(theta) took the one registered at theta's type T for x, y and
  theta, and cast the inputs to T -> it takes the one for the inputs' types
  and theta's, without casting, as a JIT IndexUnaryOp takes its thunk: with
  an op registered at FP64, op(10.0) on FP64 data runs as before, op(10.0)
  on FP32 data ran in FP64 -> KeyError naming (FP32, FP32, FP64), and op(10)
  raised KeyError when bound -> when it runs. op.types is keyed
  (x, y, theta), so registering the same x and theta types for another y no
  longer raises "already defined", and op["FP64", "FP32"] is x and y, no
  longer x and theta: op["FP64", "FP64", "FP32"] is the op with an FP32
  theta.
- A bound op pickles with all its types: op["INT64", "FP64"](1) came back as
  op["INT64"](1), whose y is INT64.
- op(theta).type: the bound type -> AttributeError, as for any untyped op;
  op(theta)[T] is op[T](theta). repr(op(2)): binary.f[INT64] ->
  indexbinary.f(2). op(theta)(A & B) works, as for other BinaryOps.
- A Scalar theta is copied when given, so a later change to it does not
  reach the op, as when the op bound it at once.

find_opclass now reads an opclass attribute on an OpBase subclass, so
op(theta) is still a BinaryOp there. The user guide's examples drop the
op[float](0.5) that hid the old typing. The comments on IndexUnaryOp's and
SelectOp's UDT caches now say they are keyed by (dtype, thunk_type).

…the field

Behavior fix. Where a literal for a record UDT is typed (a lifted op such as
+, ==, a user's op, an IndexUnaryOp thunk, an ewise_union default, and ==
on a Scalar), a part for an array field must now broadcast to the field's
shape, or ValueError names both shapes. An array UDT's element takes the
shape it broadcasts to with a literal, as numpy does, so INT8[3] plus
np.array([[1, 300, 3]]) is INT64[1, 3]; that is unchanged. A record's field
keeps its shape, so for the record the typing gave up, and the literal was
converted into the field as numpy assigns it: the leading axis was dropped
and 300 wrapped to 44, silently, even beside a user's op, which refuses a
literal that does not fit. Stores (r[0] = ...) still convert as numpy
assigns, as they do for an array UDT.

With r a Vector of a record with an int8[3] field a and an int64 field b,
holding ([1, 2, 3], 0) (before -> after):

- r + (np.array([[1, 300, 3]]), 0): ([2, 46, 6], 0) -> ValueError
  "array([[  1, 300,   3]]) has shape (1, 3), which does not fit field
  'a' of R, of shape (3,): ...".
- r.apply(user_op, right=(np.array([[1, 300, 3]]), 0)), with user_op
  returning y: ([1, 44, 3], 0) -> the same ValueError.
- r == (np.array([[1, 300, 3]]), 0): False -> the same ValueError; with
  np.array([[1, 2, 3]], np.int8) it was True.
- r + ([[1, 300, 3]], 0), and the other paths with it: numpy's "setting an
  array element with a sequence" ValueError -> this ValueError.
- r + ([1, 2], 0): numpy's "could not broadcast input array from shape (2,)
  into shape (3,)" -> this ValueError.
- r[0] = (np.array([[1, 300, 3]]), 0): ([1, 44, 3], 0), unchanged.

Literal typing had arms no test reached: a comparison with a record literal
of the wrong length (alone and nested beside an int beyond its leaf), a part
holding numpy values that numpy cannot make an array of or that is not
numbers, and text a bytes field cannot encode. An array of records now pins
the UdfParseError a lifted op gives when Numba cannot compile it, and the
NoJITWarning is checked to stay silent for a pair it warned about.

Two arms of the monoid identity are excluded from coverage: the binary op
compiles first, and it refuses what would reach them.

indexbinary.ss.register_new compiled every type signature of one op under
the op's own name. SuiteSparse finds a compiled kernel in memory by its op's
name and the types in the kernel's encoding, and an ewise or mxm kernel's
encoding has x's, y's and z's types but not theta's. So two signatures that
differ only in theta's type ran whichever kernel was compiled first. Now each
signature's C function is compiled as name_N_x_M_y_theta, with N and M the
lengths of the x and y parts (built-in types by name, UDTs by their C name):
the C definition is the user's with the function renamed. A name SuiteSparse
would cut at 127 characters is the op name's first 100 and a digest of the
whole. The op's jit_c_name and jit_c_source show the C name and the source
SuiteSparse compiles; jit_c_definition is still the definition given.
JIT IndexUnaryOps and SelectOps are not affected: the thunk's type is in
their kernels' encodings.

Behavior change (before -> after): with add_theta registered at FP64 for x,
y and theta, and then at an FP32 theta with
(*z) = (*x) + (*y) + 2 * (double)(*theta), the FP32 signature on FP64 data
with theta 10 gave [2, 2, 4, 0] (the FP64 kernel, reading the float as a
double) -> [22, 22, 24, 20]. op.jit_c_name: None ->
add_theta_4_FP64_4_FP64_FP32.

The ss.register_new functions of unary, binary, indexunary (and select) gave
SuiteSparse the op's last name as its C name, and indexbinary's the last name
and types. SuiteSparse finds a compiled kernel in memory by its op's name,
and checks an op's definition only when it loads a kernel from disk. So, in
one process, an op registered again under its name with another definition
(after delattr, as when a notebook cell is run again), or an op of the same
last name in another namespace (ss.f and ss.nested.f), ran the kernel
compiled first.

Now every op registered from C source is compiled under a C name made of its
last name, the types for an IndexBinaryOp (whose theta type is not in its
kernels' encoding), and a 12-hex digest of these with the full dotted name
and the definition: f_3a1c90e2d4b7, or add_theta_4_FP64_4_FP64_4_FP32_...
for an IndexBinaryOp. A name SuiteSparse would cut at 127 characters is the
last name's first 100 characters and a 16-hex digest. The function is renamed
in the definition wherever it is named, except as a struct member (after "."
or "->"). The same name and definition give the same C name, so a kernel is
compiled once for them. jit_c_name and jit_c_source, None for these ops
before (except IndexBinaryOp, from the previous commit), show the C name and
the renamed source; jit_c_definition is still the definition given. The
helper is ss._utils._jit_c_source.

Behavior changes (before -> after), in one process:

- binary.ss.register_new("f", x + y) and ("nested.f", x - y), both FP64:
  nested.f computed x + y -> x - y.
- unary.ss.register_new("u", x * x), delattr(unary.ss, "u"), then
  register_new("u", x + 10): u computed x * x -> x + 10.

SuiteSparse binds theta without checking that it holds a value: it reads
the Scalar's value array. Binding an empty Scalar crashed the interpreter,
and op(s) with no type, which binds a copy of s, ran with theta 0. Both now
raise EmptyObject, as apply does for an empty IndexUnaryOp thunk, in every
bind: typed or not, is_udt or not, a JIT op's too.

Behavior change (before -> after), with s = Scalar(int):

- op[int](s) and op(s, dtype=int): segmentation fault -> EmptyObject.
- op(s): an op that ran with theta 0 -> EmptyObject, at the bind. With s a
  C scalar (is_cscalar=True): segmentation fault -> EmptyObject.

A record literal's date or duration field takes the field's type only
when its value converts into the field and back unchanged; otherwise it
keeps its own type, so == compares it whole. A numpy date did this
already, but a Python datetime.date, datetime.datetime or
datetime.timedelta, or a date given as text, always took the field's type.
So it was cut to the field's unit before == compared it: noon compared
equal to its day in a field of days, and 5 s and 1 us equal to 5 s in a
field of seconds. Main gave the same wrong answers (it gave them for numpy
dates too). Now such a value is first made the numpy value it stands for
(a datetime in microseconds, a timedelta in microseconds, a date as text
in the unit it is written to), and is typed as that numpy value is.

Behavior change (before -> after), with v a Vector of a record of an
M8[D] and an m8[s] field holding (2020-01-01, 5 s), and s = v[0].new():

- v == (datetime(2020, 1, 1, 12), np.timedelta64(5, "s")), and with
  "2020-01-01T12:00" for the date: True -> False; the same for s ==.
- v == (np.datetime64("2020-01-01"), timedelta(seconds=5, microseconds=1)):
  True -> False.
- v.apply(user_op, right=(datetime(2020, 1, 1, 12), np.timedelta64(5,
  "s"))): stored the day, silently -> ValueError "does not fit", as for
  np.datetime64("2020-01-01T12:00").
- v + (datetime(2020, 1, 1, 12), ...): UdfParseError from Numba ->
  KeyError naming datetime64[D] and datetime64[us].
- A datetime at midnight, a date, and text such as "2020-01-01" still
  compare equal and fit the field.

… does

A UDT that is a bare datetime64 or timedelta64 (not a record or an array)
converted a date or duration literal into its own unit before == compared
it, so noon compared equal to its day in a UDT of days. Main raised
TypeError here for a Vector, and gave True for a Scalar; "Compare UDTs that
are neither records nor arrays in eq and ne" made the Vector give True too.
A numpy duration in another unit was a KeyError instead, even when it was
the same duration.

Now such a literal is typed as in a record's date field: the UDT itself
when it converts into the UDT and back unchanged, else numpy's type for it
(a datetime is in microseconds, a timedelta too, a date as text in the unit
it is written to). Under eq and ne, a UDT of dates or durations pairs with
one in another unit, as text of one kind pairs at another width, and they
compare as numpy compares them.

Behavior change (before -> after), with day a Vector of an M8[D] UDT
holding 2020-01-01 and five one of an m8[s] UDT holding 5 s:

- day == np.datetime64("2020-01-01T12:00"), datetime(2020, 1, 1, 12) or
  "2020-01-01T12:00": True -> False; != the opposite; the same for
  day[0].new() ==.
- five == np.timedelta64(5000, "ms"): KeyError -> True; 5001 ms: KeyError
  -> False. five == timedelta(seconds=5, microseconds=1): True -> False.
- binary.eq(five & a Vector of m8[ms] holding 5000 ms), and five.isequal
  of it: KeyError -> True.
- day.apply(user_op, right=datetime(2020, 1, 1, 12)): stored the day,
  silently -> ValueError "does not fit".
- five.apply(user_op, right=np.timedelta64(5000, "ms")): converted with a
  DeprecationWarning -> converts with none, as in a record's field.
- day == timedelta(5): numpy's ValueError -> binary.eq's KeyError.

A literal operand beside a UDT became an element of that UDT even when
the op was typed and takes that operand in another type: op[A, INT64]
takes its thunk, or its second input, as an INT64. Apply and select pass
a UDT thunk or operand to SuiteSparse as a pointer to its bytes, so the op
read the bytes of the A element as an int64: silently wrong answers.
Main did the same. A Scalar of the op's type gave the right answer.

Now a typed op's own input type on the literal's side decides: its first
input type for a left operand, else its second input or thunk type.
GraphBLAS converts a value to a built-in input type, so the literal is
typed as beside a built-in typed op (its own type: 2.5 for an INT64 thunk
is converted to 2, as for an op typed on built-in types); a UDT input type
takes only a literal that fits it. _literal_operand takes left=True from
apply's left operand and from ewise_union's left default.

Behavior change (before -> after), with v a Vector of a record A of two
float64 fields holding (1.0, 2.0) and (3.0, 4.0), and user ops typed
[A, INT64] (or [INT64, A]) that compare or add field a with the other:

- v.apply(iu[A, INT64], 2): [False, False] -> [False, True], as with
  Scalar.from_value(2); the same for thunks 2.5 and np.int8(2).
- v.select(se[A, INT64], 2), on a Vector or Matrix: empty -> index 1.
- v.apply(f[A, INT64], right=2) and v.apply(g[INT64, A], left=2): about
  4.6e18 for each element -> [3.0, 5.0].
- With B a record of two int64 fields: v.apply(h[A, B], right=(2, 3)):
  about 4.6e18 -> [3.0, 5.0]; v.apply(iu_rec[A, B], (2, 0)): [False,
  False] -> [False, True], and with (2.5, 0): [False, False] -> ValueError
  "does not fit".
- Scalar(A).ewise_mult(2, f[A, INT64]): GraphBLAS's DomainMismatch -> 3.0.

The accumulator is typed on the object's dtype, and for a built-in Scalar
GraphBLAS converts the value to that type before it accumulates: an INT8
Scalar of -1 with s += 0.5 (Scalar.__iadd__ is s(binary.plus) << 0.5)
adds 0 and stays -1, as v(accum=binary.plus) << 0.5 does for a Vector of
either kind. A UDT Scalar used the untyped op instead (_scalar_accum_op),
so it computed -1 + 0.5 in FP64 and truncated the sum to 0. That was meant
to match a Vector's v += 0.5, but that is v << v + 0.5, not an accumulate.
Main gave the built-in answer.

Now a UDT Scalar accumulates with the typed op, and the value is converted
to the Scalar's type first, as storing it would convert it: a literal by
Scalar.from_value, a Scalar or an element of another UDT by the store cast.
A built-in Scalar into a UDT Scalar still raises DomainMismatch, as a store
of it does, into a Vector too.

Behavior change (before -> after), with s an INT8[3] UDT Scalar of
[-1, -1, -1]:

- s += 0.5 and s(accum=binary.plus) << 0.5: [0, 0, 0] -> [-1, -1, -1]
  (an INT8 Scalar gives -1; a Vector of the UDT gives [-1, -1, -1]).
- s(accum=binary.plus) << (0.5, 1.5, 2.5): [0, 0, 1] -> [-1, 0, 1], as for
  a Vector of the UDT.
- s(accum=binary.plus) << v[1], with v a Vector of an FP64[3] UDT holding
  [0.5, 1.5, 2.5]: [0, 0, 1] -> [-1, 0, 1], as with the same value in a
  Scalar of that UDT, which was cast first already.

udt.rst says a numpy array beside a UDT pairs as an array UDT of its own
shape, and array UDTs pair when their shapes broadcast. A numpy array
paired only when it broadcast to the UDT's shape, or had that shape apart
from leading 1s. Any other array, even one whose shape broadcasts with
the UDT's, was converted into the UDT, where numpy raised "could not
broadcast", while the same values as a UDT operand worked. Main raised
too (a KeyError for the UDT pair). Now such an array is typed in its own
shape and pairs as an array UDT operand does; the result has the shape
numpy's would. An array that broadcasts to the UDT's shape is typed in the
UDT's shape, as before, which a literal that must become an element needs.

Behavior change (before -> after), with v a Vector of an INT8[3] UDT
holding [1, 2, 3] and u one of an INT8[1] UDT holding [1]:

- v + np.array([[1, 2, 3], [4, 5, 6]]), and apply(binary.plus) with it on
  either side: ValueError "could not broadcast input array from shape
  (2,3) into shape (3,)" -> INT64[2, 3] [[2, 4, 6], [5, 7, 9]], as with
  the same values in an INT64[2, 3] UDT.
- v == np.array([[1, 2, 3], [1, 2, 3]]), on a Vector or a Scalar: the same
  ValueError -> True.
- u + np.array([1, 2, 3]), and u + (1, 2, 3): ValueError -> INT64[3]
  [2, 3, 4].
- v + [[1, 2, 3], [4, 5, 6]]: numpy's "setting an array element with a
  sequence" ValueError -> INT64[2, 3], as for the array.
- v.ewise_union(v, binary.plus, a 2-by-3 array, 0): numpy's ValueError ->
  ValueError "does not fit".

OpBase.__getitem__ reads is_positional for a type the op does not have,
and IndexBinaryOp had none, so asking an IndexBinaryOp (registered without
is_udt) for a type it does not take raised AttributeError. Main did the
same; IndexBinaryOp has not been released. It is now False, as on a bound
IndexBinaryOp and every other op class.

Behavior change (before -> after), with op an IndexBinaryOp of x + y +
theta and R a record UDT:

- R in op: AttributeError -> False, as for an IndexUnaryOp.
- op[R] and op(Scalar.from_value((1.0, 2), R), dtype=R): AttributeError ->
  KeyError "... does not work with R".

reduce turns a BinaryOp that is part of a Monoid into that Monoid, and
refuses any other BinaryOp with a TypeError naming it. It reads the op's
monoid for that, and the error message its parent's; neither a bound
IndexBinaryOp nor IndexBinaryOp had the attribute, so reduce raised
AttributeError, for ibo[T](theta) on main and for the new untyped
ibo(theta) too. Both now have monoid = None, as a BinaryOp of no Monoid
has. IndexBinaryOp has not been released.

Behavior change (before -> after), with v an INT64 Vector and ibo an
IndexBinaryOp of x + y + theta:

- v.reduce(ibo(1)) and v.reduce(ibo[int](1)): AttributeError
  "'_BoundIndexBinaryOp' object has no attribute 'monoid'" -> TypeError
  "Bad type for argument `op` in Vector.reduce(...) ... is not known to be
  part of a Monoid", as for binary.minus.
- A.reduce_rowwise(ibo(1)): AttributeError -> NotImplementedException, as
  for binary.minus.