Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
088620e
Name the op and types when a UDT UDF reads a field the record lacks
eriknw Nov 3, 2026
0668ef5
Refuse an op that takes no UDTs before converting a literal for it
eriknw Nov 3, 2026
0f13324
Compare an int beyond 64 bits in Scalar.isequal as == compares it
eriknw Nov 3, 2026
3e235fe
Name the int and the type when an int theta does not fit an IndexBina…
eriknw Nov 3, 2026
2814db6
Refuse a tuple of types an op does not take with a TypeError
eriknw Nov 3, 2026
58350ef
Refuse an empty Scalar as the identity of a Monoid
eriknw Nov 3, 2026
6b9e680
Refuse records in an array with a KeyError for a lifted op
eriknw Nov 3, 2026
a29d1be
Compare a str that bytes cannot hold as unequal to a bytes UDT
eriknw Nov 3, 2026
54d4491
Refuse an array literal that does not broadcast with an array UDT by …
eriknw Nov 3, 2026
9519efa
Chain the reason lookup_dtype could not register a dtype
eriknw Nov 3, 2026
b290c9e
Refuse raw bytes (a void dtype) with a clear error where Numba cannot…
eriknw Nov 3, 2026
5dab160
Type unary.abs of a complex UDT leaf as real, as the built-in abs does
eriknw Nov 3, 2026
3abbc28
Convert a list laid out like a record as a tuple, as udt.rst promises
eriknw Nov 3, 2026
332c969
Select a UDT's elements equal or unequal to a value, as == compares them
eriknw Nov 3, 2026
1e73309
Store identity, first or second of a UDT into another UDT as it computes
eriknw Nov 3, 2026
cd7924f
Say that built-in ops on UDTs need Numba where isequal fails without it
eriknw Nov 3, 2026
ce8d04b
Document that a stored literal converts as numpy assigns it
eriknw Nov 3, 2026
03599eb
List every case _get_udt_c_type_info gives None for
eriknw Nov 3, 2026
433e8a1
Type a UDT's select by equality at the call, as every select is
eriknw Nov 3, 2026
b92df87
Compare an int beyond 64 bits in Scalar.isclose as isequal does
eriknw Nov 3, 2026
8163d97
Compare an np.str_ that bytes cannot hold as unequal to a bytes UDT too
eriknw Nov 3, 2026
bf606e2
Give numpy's error for an empty array literal beside an array UDT
eriknw Nov 3, 2026
2fa4737
Say "a tuple of 1" where an op is indexed by a 1-tuple of types
eriknw Nov 3, 2026
ea2c068
Name the ops on UDTs that work without Numba in udt.rst
eriknw Nov 3, 2026
38f0380
Let graphblas.core.ss.binary and unary be a process's first import
eriknw Nov 3, 2026
3dc7f97
Leave the JIT control alone in test_ssjit's setup where SuiteSparse u…
eriknw Nov 3, 2026
a01bd97
Keep an array-UDT element whole in column-major export and import
eriknw Nov 4, 2026
7af312e
Import an array UDT's raw column-major values as they are
eriknw Nov 4, 2026
f488885
Export an array-UDT matrix given spare values as full or bitmap
eriknw Nov 4, 2026
a8f2f2a
Check what test_subarray_dtypes builds
eriknw Nov 4, 2026
89930f9
Build a monoid over an is_udt op on a built-in type with its typed co…
eriknw Nov 4, 2026
9cdaa6d
Convert array-UDT values to another array dtype number by number
eriknw Nov 4, 2026
70cecbc
Deserialize a type from C source by its C name, or say to pass dtype=
eriknw Nov 4, 2026
f06542d
Name a missing record field on Numba before 0.60 too
eriknw Nov 8, 2026
d592b84
Say to pass dtype= on SuiteSparse 8 too when deserialize cannot tell …
eriknw Nov 8, 2026
18c474d
Give Numba 0.60's sentence for x["name"] as it is, not as a field name
eriknw Nov 8, 2026
5928fec
Use isclose's own rule for an int beyond 64 bits in Scalar.isclose
eriknw Nov 12, 2026
77ec424
Test a UDT's select by a transposed comparison
eriknw Nov 12, 2026
d00f5f6
Export an iso UDT matrix as coo
eriknw Nov 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions docs/user_guide/udt.rst
Original file line number Diff line number Diff line change
Expand Up @@ -128,13 +128,20 @@ The following operators auto-lift to any UDT on first use:
``second``, ``any``, and ``pair`` work too, since they don't touch field
values.
- UnaryOps: ``ainv``, ``abs``.
- ``select`` with ``valueeq`` or ``valuene`` (``v.select("==", x)``), which
keeps the elements that ``==`` or ``!=`` finds.
- Monoids: ``plus``, ``times``, ``min``, ``max``, ``any``.
- Semirings combining a UDT-lifting monoid with a UDT-lifting BinaryOp
(e.g., ``plus_times``, ``min_plus``, ``max_times``, ``any_first``).
- Aggregators built on those monoids: ``sum``, ``prod``, ``min``, ``max``,
``any_value``, ``count``. The positional ``agg.ss.first`` and
``agg.ss.last`` work on any dtype, including UDTs.

Except ``pair``, ``count``, ``agg.ss.first`` and ``agg.ss.last``, they need
Numba. Without it, the others raise ``KeyError`` on a UDT, and so do ``==`` and
``isequal`` on a Vector or Matrix of a UDT; a ``Scalar``'s ``==`` and ``isequal``
compare in numpy.

For example:

.. code-block:: python
Expand Down Expand Up @@ -256,8 +263,9 @@ data.
* - ...from an existing object: ``w << v``, ``v.dup(dtype=...)``, with a
mask, an accumulator or ``replace``
- One pass through a cast op.
* - ...from an element-wise lifted op: ``apply``, ``ewise_mult``,
``ewise_union``, and infix operators that use them
* - ...from an element-wise lifted op, or ``identity``, ``first`` or
``second``: ``apply``, ``ewise_mult``, ``ewise_union``, and infix
operators that use them
- The op writes the object's type itself, in the same call:
``int_udt_vec += 0.5`` truncates, as it does for an ``INT8`` vector.
* - ...from anything else: ``mxm``, ``reduce`` into a Vector, ``ewise_add``,
Expand All @@ -272,6 +280,13 @@ data.
cast before it accumulates, as GraphBLAS converts a value of a built-in
dtype, and so is a literal: an ``INT8[3]`` Scalar's ``s += 0.5`` adds 0,
as an ``INT8`` Scalar's does.
* - A literal: ``v[0] = (1.5, 1)``, ``Scalar.from_value``,
``Vector.from_scalar``
- Converted as numpy assigns it to an element, so, unlike a literal that
an op makes an element, it need not fit: ``1.5`` in an integer field
truncates, text too long for its field is cut, and a datetime in a date
field loses its time of day. An int out of an integer field's range
raises ``OverflowError``, where an ``INT8`` Vector wraps it.
* - Between a UDT and a built-in dtype, or layouts that do not correspond
- ``DomainMismatch``.

Expand Down
41 changes: 34 additions & 7 deletions graphblas/core/dtypes.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,14 @@ def _get_udt_c_type_info(datatype):
``_gbudt_NNN`` name when the user didn't supply a name, or supplied one
that isn't a valid C identifier or is too long for SuiteSparse
(``udt_codegen._pick_c_type_name``), so the UDT can still take the JIT
path. Returns ``None`` only for the cases that remain unrepresentable in
C: an unsupported field type (object, datetime, string), or a field name
that collides with a C reserved word or isn't an identifier. Supports
record UDTs (including nested records) and array UDTs (e.g., ``FP64[3]``).
path. Returns ``None`` when the UDT has no C form here: a record laid out
other than as a C compiler lays out its struct (packed, as
``np.dtype([...])`` is without ``align=True``), a field name that is not
a valid C identifier (a C reserved word, say), a field that is an array
(only an array UDT is flattened, to ``v[n]``), a field or element type
with no C type (object, datetime, text), or a UDT that is neither a
record nor an array. Supports record UDTs (including nested records) and
array UDTs (e.g., ``FP64[3]``).
"""
from .operator.udt_codegen import _udt_c_typedef

Expand Down Expand Up @@ -212,6 +216,28 @@ def _numba_type(dtype):
f"Numba, which python-graphblas uses for UDTs, cannot type an array of {element}, "
f"as in {dtype}; use an array of {instead} instead"
) from exc
except numba.core.errors.NumbaError as exc:
# Numba's own error is "NumbaNotImplementedError: |V4", with terminal colors
raw = _raw_bytes_element(dtype)
if raw is None: # pragma: no cover (safety)
raise
raise ValueError(
f"Numba, which python-graphblas uses for UDTs, cannot type raw bytes ({raw}), "
f"as in {dtype}; use an array of uint8 instead, np.dtype((np.uint8, "
f"{raw.itemsize}))"
) from exc


def _raw_bytes_element(dtype):
"""The first void dtype with no fields (raw bytes) in dtype, or None."""
if dtype.subdtype is not None:
return _raw_bytes_element(dtype.subdtype[0])
if dtype.names is None:
return dtype if dtype.kind == "V" else None
for field_dtype, *_ in dtype.fields.values():
if (raw := _raw_bytes_element(field_dtype)) is not None:
return raw
return None


def register_new(name, dtype=None):
Expand Down Expand Up @@ -605,13 +631,14 @@ def lookup_dtype(key, value=None):
try:
# Auto-register!
return register_anonymous(key)
except Exception:
pass
except Exception as exc:
# Why it cannot be registered, such as an array of bool, which Numba cannot type.
reason = exc
try:
return lookup_dtype(key.literal_type) # For numba dtype inference
except Exception:
pass
raise ValueError(f"Unknown dtype: {key} of type {type(key)}")
raise ValueError(f"Unknown dtype: {key} of type {type(key)}") from reason


def _raise_dtype_or_arraylike(cls_name, dtype, exc):
Expand Down
61 changes: 36 additions & 25 deletions graphblas/core/literals.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@
from .operator.monoid import Monoid
from .udt import (
BUILTIN_UDT_BINARY_OPS,
_as_record_parts,
_casts_safely_into,
_dict_to_record,
_leaf_arrays,
_promoted_record,
_strip_leading_ones,
Expand Down Expand Up @@ -102,7 +102,10 @@ def _literal_operand(dtype, value, op, *, left=False):
return (math.inf if value > 0 else -math.inf), FP64
return value, literal_dtype
if getattr(op, "_udt_types", True) is None:
# An op that does not take UDTs at all (``lt``) reports that itself.
# An op that does not take UDTs at all (``lt``) reports that itself, here,
# before the literal becomes an element, which can fail first (300 in an
# int8 field).
op[dtype]
return value, dtype
if isinstance(op, Monoid) and not op._anonymous:
# Element-wise, a Monoid is its BinaryOp (``get_typed_op`` types it so).
Expand All @@ -113,11 +116,14 @@ def _literal_operand(dtype, value, op, *, left=False):
# truediv gives float leaves whatever the int, so, as a comparison, it
# types an int out of an integer leaf's range exactly instead of raising.
exact = is_comparison or op.name == "truediv"
if is_comparison and not isinstance(value, (np.generic, np.ndarray)):
np_type = dtype.np_type
parts = value
if isinstance(value, dict) and np_type.names is not None:
parts = _dict_to_record(np_type, value)
np_type = dtype.np_type
if is_comparison and np_type.kind == "S" and isinstance(value, str):
# A str, np.str_ included, that bytes cannot hold (numpy encodes ASCII)
# equals no element, as in numpy; one byte longer than any, it compares so.
if not value.isascii():
value = b"\xff" * (np_type.itemsize + 1)
elif is_comparison and not isinstance(value, (np.generic, np.ndarray)):
parts = _as_record_parts(np_type, value)
# First: making an int beyond its leaf infinity reshapes the part.
_refuse_misfit_fields(dtype, np_type, parts)
if _int_beyond_leaf(np_type, parts) is not None:
Expand Down Expand Up @@ -311,13 +317,12 @@ def _literal_type(dtype, value, *, exact=False):
"""
value = _as_python_number(value)
np_type = dtype.np_type
if isinstance(value, dict) and np_type.names is not None:
value = _dict_to_record(np_type, value)
value = _as_record_parts(np_type, value)
_refuse_misfit_fields(dtype, np_type, value)
if np_type.names is None and np_type.kind in "SU" and isinstance(value, (str, bytes)):
if _keeps_value(value, np_type):
return dtype
return lookup_dtype(np.dtype(f"{np_type.kind}{max(len(value), 1)}"))
return lookup_dtype(_own_text_type(np_type.kind, value))
if np_type.kind in "mM" and (date := _as_numpy_date(value, np_type)) is not None:
if date.dtype.kind == np_type.kind and _keeps_value(date, np_type):
return dtype
Expand All @@ -339,6 +344,17 @@ def _literal_type(dtype, value, *, exact=False):
return None


def _own_text_type(kind, text):
"""Return the type of ``text`` that does not fit a text leaf of ``kind``: its own length.

It keeps the leaf's kind, except that a str that bytes cannot hold (numpy
encodes ASCII) is a str, so beside bytes it equals nothing, as in numpy.
"""
if kind == "S" and isinstance(text, str) and not text.isascii():
kind = "U"
return np.dtype(f"{kind}{max(len(text), 1)}")


def _check_literal_fits(dtype, value):
"""Raise unless literal ``value`` may become an element of UDT ``dtype``, by its type.

Expand Down Expand Up @@ -373,8 +389,7 @@ def _check_literal_fits(dtype, value):
np_type = dtype.np_type
if np_type.names is not None:
target = np.zeros(1, dtype=np_type)
if isinstance(value, dict):
value = _dict_to_record(np_type, value)
value = _as_record_parts(np_type, value)
else:
# An array's elements, or the one value of a UDT that is neither (bytes).
base, shape = np_type.subdtype or (np_type, ())
Expand Down Expand Up @@ -445,8 +460,7 @@ def _converts_exactly(dtype, value):
np_type = dtype.np_type
try:
if np_type.names is not None:
if isinstance(value, dict):
value = _dict_to_record(np_type, value)
value = _as_record_parts(np_type, value)
given = np.zeros(1, dtype=_object_layout(np_type))
converted = np.zeros(1, dtype=np_type)
else:
Expand Down Expand Up @@ -671,7 +685,7 @@ def leaf_dtype(leaf, part, *, in_record=False):
# Text in a duration field, or text numpy cannot read as a date,
# takes the field's type, and converting it reports what is wrong.
if isinstance(part, (str, bytes)) and base.kind in "SU":
own = np.dtype(f"{base.kind}{max(len(part), 1)}")
own = _own_text_type(base.kind, part)
elif (date := _as_numpy_date(part, base)) is not None:
part = date
own = date.dtype
Expand Down Expand Up @@ -874,18 +888,19 @@ def _array_literal_udt(dtype, value):
``+ np.array([1, 2, 3])`` int64 ones. Its type is the array layout of its
dtype: in the UDT's shape when it broadcasts to that (``np.array([1])``
beside ``INT8[3]``), which a literal that must become an element needs,
else in its own shape when the two broadcast together, so that an op's
result takes the shape numpy's would (``INT8[3]`` and a 2-by-3 array give
``[2, 3]``). A 0-d structured array or ``np.void`` beside a record is its
record dtype. Anything else, such as a numeric array beside a record, is
converted into the UDT.
else in its own shape, so that an op's result takes the shape numpy's
would (``INT8[3]`` and a 2-by-3 array give ``[2, 3]``), and an op refuses
shapes that do not broadcast together, as for two UDTs. A 0-d structured
array or ``np.void`` beside a record is its record dtype. Anything else,
such as a numeric array beside a record, is converted into the UDT.
"""
np_type = dtype.np_type
if value.dtype.names is not None:
return lookup_dtype(value.dtype) if value.ndim == 0 else None
# A bool array converts as it is (a BOOL array UDT cannot be registered),
# and bool casts safely into every numeric element.
if np_type.names is not None or value.dtype.kind not in "iufc":
if np_type.names is not None or value.dtype.kind not in "iufc" or value.size == 0:
# An empty array's layout cannot be a UDT, so numpy says why it does not fit.
return None
shape = np_type.subdtype[1]
if _strip_leading_ones(value.shape) == _strip_leading_ones(shape):
Expand All @@ -894,9 +909,5 @@ def _array_literal_udt(dtype, value):
try:
np.broadcast_to(value, shape)
except ValueError:
try:
np.broadcast_shapes(value.shape, shape)
except ValueError:
return None
shape = value.shape
return lookup_dtype(np.dtype((value.dtype, shape)))
24 changes: 22 additions & 2 deletions graphblas/core/matrix.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,15 @@
values_to_numpy_buffer,
wrapdoc,
)
from .vector import Vector, VectorExpression, VectorIndexExpr, _isclose_recipe, _select_mask
from .vector import (
Vector,
VectorExpression,
VectorIndexExpr,
_isclose_recipe,
_select_by_comparison,
_select_mask,
_udt_select_comparison,
)

if backend == "suitesparse":
from .ss.matrix import ss
Expand Down Expand Up @@ -2809,9 +2817,10 @@ def select(self, op, thunk=None):
if thunk is None:
thunk = False # most basic form of 0 when unifying dtypes
given = thunk
comparison = _udt_select_comparison(op, self.dtype)
if type(thunk) is not Scalar:
# A thunk beside a UDT becomes an element of it, so it must fit, as in apply.
thunk, dtype = _literal_operand(self.dtype, thunk, op)
thunk, dtype = _literal_operand(self.dtype, thunk, comparison or op)
try:
thunk = Scalar.from_value(thunk, dtype, is_cscalar=None, name="")
except TypeError:
Expand All @@ -2823,6 +2832,17 @@ def select(self, op, thunk=None):
extra_message="Literal scalars also accepted.",
op=op,
)
if comparison is not None:
mask_expr = self.apply(comparison, right=thunk) # typed now, so errors raise now
return MatrixExpression(
method_name,
None,
[self, thunk, _select_by_comparison, (self, mask_expr)],
expr_repr=f"{{0.name}}.select({op}, thunk={{1._expr_name}})",
nrows=self._nrows,
ncols=self._ncols,
dtype=self.dtype,
)
op = get_typed_op(op, self.dtype, thunk.dtype, is_right_scalar=True, kind="select")
op, thunk = _value_select_in_own_type(op, self.dtype, thunk, given)
self._expect_op(op, ("SelectOp", "IndexUnaryOp"), within=method_name, argname="op")
Expand Down
4 changes: 4 additions & 0 deletions graphblas/core/operator/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -429,6 +429,10 @@ def __getitem__(self, type_):
if type(type_) is tuple:
from .utils import get_typed_op

if len(type_) != 2:
raise TypeError(
f"{self!r}[...] takes a dtype or a pair of dtypes, not a tuple of {len(type_)}"
)
dtype1, dtype2 = type_
dtype1 = lookup_dtype(dtype1)
dtype2 = lookup_dtype(dtype2)
Expand Down
4 changes: 3 additions & 1 deletion graphblas/core/operator/binary.py
Original file line number Diff line number Diff line change
Expand Up @@ -846,7 +846,9 @@ def _compile_udt(self, dtype, dtype2):
self.name, py_op, dtype, dtype2
)
elif self._numba_func is None:
raise KeyError(f"{self.name} does not work with {dtypes} types")
# Only without Numba does an op that lifts to UDTs get here.
needs = "; built-in ops work on UDTs only with Numba installed" if is_lifted else ""
raise KeyError(f"{self.name} does not work with {dtypes} types{needs}")
else:
numba_func = self._numba_func
sig = (dtype.numba_type, dtype2.numba_type)
Expand Down
16 changes: 16 additions & 0 deletions graphblas/core/operator/indexbinary.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,17 @@ def _theta_to_scalar(theta, dtype):
"(op[dtype](theta), or op(theta, dtype=...))."
) from exc
raise
except OverflowError as exc:
# cffi's message, "int too big to convert", names neither the int nor the type.
if type(theta) is not int or dtype is not None and dtype._is_udt:
raise
if dtype is None:
raise OverflowError(
f"Python integer {theta} out of bounds for int64, the type of a Python int "
"theta; give theta as a Scalar of a type that holds it, or name the op's "
"type (op[dtype](theta), or op(theta, dtype=...))."
) from exc
raise OverflowError(f"Python integer {theta} out of bounds for {dtype.np_type}") from exc


def _refuse_empty_theta(theta):
Expand Down Expand Up @@ -578,6 +589,11 @@ def _get_typed(self, dtype, dtype2, theta_type):
def __getitem__(self, type_):
if type(type_) is tuple and len(type_) == 3:
return self._get_typed(*(lookup_dtype(dtype) for dtype in type_))
if type(type_) is tuple and len(type_) != 2:
raise TypeError(
f"{self!r}[...] takes a dtype, or a tuple of two (x and y) or three (x, y and "
f"theta), not a tuple of {len(type_)}"
)
return super().__getitem__(type_)

def _add(self, op, *, is_jit=False):
Expand Down
16 changes: 14 additions & 2 deletions graphblas/core/operator/monoid.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
UINT64,
lookup_dtype,
)
from ...exceptions import check_status_carg
from ...exceptions import EmptyObject, check_status_carg
from .. import _has_numba, ffi, lib
from ..utils import libget
from .base import OpBase, ParameterizedUdf, TypedOpBase, _BinaryopJitDelegate, _hasop
Expand Down Expand Up @@ -304,6 +304,12 @@ def _build(cls, name, binaryop, identity, *, is_idempotent=False, anonymous=Fals
"a new BinaryOp (BinaryOp.register_new or BinaryOp.register_anonymous) "
"and pass that."
)
from ..scalar import Scalar

identities = identity.values() if isinstance(identity, Mapping) else [identity]
if any(isinstance(ident, Scalar) and ident._is_empty for ident in identities):
# A UDT monoid would get an identity of zero bytes, a wrong one for ``times``.
raise EmptyObject("identity is an empty Scalar; give a Scalar that holds a value")
if name is None:
name = binaryop.name
new_type_obj = cls(
Expand Down Expand Up @@ -378,7 +384,13 @@ def _compile_udt(self, dtype, dtype2):
else:
identity = Scalar.from_value(identity_val, dtype=ret_type, is_cscalar=True)
new_monoid = ffi_new("GrB_Monoid*")
status = lib.GrB_Monoid_new_UDT(new_monoid, binaryop.gb_obj, identity.gb_obj)
if ret_type._is_udt:
status = lib.GrB_Monoid_new_UDT(new_monoid, binaryop.gb_obj, identity.gb_obj)
else:
# An is_udt op typed for a built-in type, as a monoid over a UDF built by
# ``_build`` is. GrB_Monoid_new_UDT refused it with an empty DomainMismatch.
func = libget(f"GrB_Monoid_new_{ret_type.name}")
status = func(new_monoid, binaryop.gb_obj, identity.gb_obj[0])
check_status_carg(status, "Monoid", new_monoid[0])
op = TypedUserMonoid(
self,
Expand Down
Loading
Loading