Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
f75adab
Let UDF registration name its return dtype with ret_dtype
eriknw Aug 3, 2026
1449545
Use unique array shapes for the ret_dtype test UDTs
eriknw Aug 4, 2026
05ea255
Drop the stale separate-DataTypes note from _array_udt_operands
eriknw Sep 23, 2026
a884f18
Check a declared ret_dtype against what the UDF returns
eriknw Sep 25, 2026
6b70607
Ignore the UDT repr size warning in the ret_dtype mismatch test
eriknw Sep 27, 2026
f4cd16c
Cover a declared record ret_dtype returned as is, and a return that c…
eriknw Oct 20, 2026
71e8d00
Test that unpickling a named UDF re-registers it with its ret_dtype
eriknw Oct 20, 2026
5c0ceaf
Refuse a UDT UDF that raises for its operand types when the op is typed
eriknw Nov 3, 2026
adc32fa
Type a number as itself beside a user's op that cannot take it as an …
eriknw Nov 3, 2026
ba2a00a
Let ret_dtype name the operand's own date, duration or text UDT
eriknw Nov 3, 2026
a294ebd
Say that a complex return cannot be written as a real ret_dtype
eriknw Nov 3, 2026
dceefdf
Say when a wrong ret_dtype raises, and what it and inference cover
eriknw Nov 3, 2026
c04bd83
Test ret_dtype with lazy and parameterized registration for every op …
eriknw Nov 3, 2026
30020cb
Type a UDF that raises on stand-ins as before; only the literal rule …
eriknw Nov 3, 2026
038f803
Write 0 (False) into a UDF's built-in output before calling it
eriknw Nov 3, 2026
1bcc6bd
Probe a select UDF that returns an array, so a literal thunk can type…
eriknw Nov 3, 2026
61a3899
Let a complex return be declared as ret_dtype=BOOL again
eriknw Nov 3, 2026
dd30f6c
Say in udt.rst what a UDF that raises gets, at typing and when it runs
eriknw Nov 3, 2026
860cd1c
Keep a UDF's stand-in failure as text, so the op it records can be freed
eriknw Nov 3, 2026
450d00e
Test that each built-in-type UDF wrapper writes 0 before calling the UDF
eriknw Nov 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 48 additions & 2 deletions docs/user_guide/udt.rst
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,10 @@ dtype, so it never narrows. Below, ``v`` is a Vector of an ``INT8[3]`` UDT.
but converts exactly, such as ``0`` for a bool field or ``2.0`` for an
int field, still converts with a ``DeprecationWarning`` and will raise
in a future version; any other raises ``ValueError``, and a float too
large for a float field raises ``OverflowError``.
large for a float field raises ``OverflowError``. A Python or numpy
number beside a user-defined op that cannot be typed with an element
(``x[:2] * t`` with a 6-element ``t``) is typed as itself instead, as a
``Scalar`` of it would be.
- ``v.ewise_union(w, binary.plus, 0.5, 0)`` raises

A Monoid used element-wise (``apply``, ``ewise_mult``, ``ewise_union``) uses
Expand Down Expand Up @@ -365,12 +368,55 @@ array returned into an array-typed field of a record UDT. A tuple or list also
fills such a field, but only a one-dimensional one and only at its exact
length; unlike an array, it does not broadcast. Numba does not know the shape
of an array the UDF builds, or the length of a list, so the check runs the UDF
once on sample values; a UDF that raises on those values is not checked.
on sample values; a UDF that raises on those values is not checked, since it
may raise only for some values. An exception the UDF raises when the op runs
is printed and goes no further (Numba cannot pass it to GraphBLAS), so that
element is not written; a built-in output, such as a select's bool, is 0 or
``False`` there.

If your UDF references a field that doesn't exist, or returns the wrong arity,
you'll get a ``UdfParseError`` with the actionable diagnostic line surfaced
from Numba's typing pass instead of a 200-line traceback.

Naming the output type
~~~~~~~~~~~~~~~~~~~~~~

By default the return type is worked out from what the UDF returns, matched
against the input dtypes. That can only name a type the operator already has
in hand, so an output UDT that is not one of the operands is out of reach.
The ``x[:2]`` rejection above is a case of this: the output really is a
2-element UDT, but nothing says so.

Pass ``ret_dtype`` to say it outright. It takes anything ``lookup_dtype``
accepts and requires ``is_udt=True``:

.. code-block:: python

nine = gb.dtypes.register_anonymous(np.dtype((np.float64, (9,))), "Nine")
three = gb.dtypes.register_anonymous(np.dtype((np.float64, (3,))), "Three")

head = gb.unary.register_anonymous(lambda x: x[:3], "head", is_udt=True, ret_dtype=three)
head[nine].return_type # Three

``ret_dtype`` is available on ``register_anonymous`` and ``register_new`` for
``UnaryOp``, ``BinaryOp``, ``IndexUnaryOp``, and ``IndexBinaryOp``. It is a
property of the operator, not of a particular input dtype: the same output
type applies to every dtype the operator is typed for. An operator whose
output type should vary with its inputs still needs one registration per
output type.

Declaring the type does not switch off the shape check described above; it
points the check at the declared type instead. The result must also be
something the declared type can hold: one value per field for a record, an
array or a scalar for an array UDT, and a scalar for a scalar type. A UDF that
breaks either rule is rejected when the op is typed (``op[dtype]``, or its
first use), so a wrong ``ret_dtype`` is an error rather than a silently
mis-typed result.

It does not apply to ``SelectOp``, whose return type GraphBLAS fixes at
``BOOL``, nor to an op registered without ``is_udt=True``, whose return type
comes from compiling the function against each input type in turn.

.. _udt_jit_introspection:

JIT and introspection
Expand Down
42 changes: 42 additions & 0 deletions graphblas/core/literals.py
Original file line number Diff line number Diff line change
Expand Up @@ -127,10 +127,52 @@ def _literal_operand(dtype, value, op, *, left=False):
literal_type = _literal_type(dtype, value, exact=exact)
if literal_type is not None:
return value, literal_type
if (own := _own_scalar_type_for_user_op(dtype, value, op, left)) is not None:
return value, own
_check_literal_fits(dtype, value)
return value, dtype


def _own_scalar_type_for_user_op(dtype, value, op, left):
"""Return a number's own type, for a user's op that cannot take it as an element, or None.

Beside a user's ``is_udt`` op (an untyped BinaryOp, IndexUnaryOp or
SelectOp), a literal becomes an element of the UDT. A Python or numpy
number for an op whose function cannot be typed with that element (it
fails to compile, or raises on every set of sample values: ``x[:2] * t``
with a 6-element ``t``) is typed as itself instead, as a ``Scalar`` of it
would be, when the op can be typed so. Otherwise it stays an element,
and the op reports why it cannot take it.
"""
from ..exceptions import UdfParseError

if (
not getattr(op, "_is_udt", False)
or isinstance(op, (TypedOpBase, Monoid))
or not (
type(value) in {bool, int, float, complex} or isinstance(value, (np.number, np.bool_))
)
or (type(value) is int and not -(2**63) <= value < 2**63) # no INT64 holds it
or (own := lookup_dtype(type(value), value))._is_udt
):
return None

from .operator.udf import _RAISES_ON_SAMPLES

def types(dtype2):
try:
return op[(dtype2, dtype) if left else (dtype, dtype2)]
except UdfParseError:
return None

# A UDF that raised on every stand-in element types, but cannot take an element.
# The number's own type need only type: its stand-in is a fixed sample, not the number.
element = types(dtype)
if element is not None and element not in _RAISES_ON_SAMPLES or types(own) is None:
return None
return own


# Built-in ops that compare their operands, and so keep no result type.
_COMPARISONS = {"eq", "ne", "lt", "le", "gt", "ge"}
_COMPARISONS.update([f"value{name}" for name in list(_COMPARISONS)])
Expand Down
57 changes: 50 additions & 7 deletions graphblas/core/operator/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,39 @@ def _hasop(module, name):
)


def _validate_ret_dtype(ret_dtype, opclass, *, is_udt, parameterized):
"""Normalize a user-supplied ``ret_dtype`` to a DataType, or raise.

``ret_dtype`` names the operator's output type outright instead of letting
it be inferred from what the UDF returns. Inference can only name a type it
can see, which is why this is limited to the UDT path: the builtin path
derives its output from Numba's typing of each sample input, and forcing a
single type across all of them would silently recast results.
"""
if ret_dtype is None:
return None
if not is_udt:
raise ValueError(
f"{opclass}: ret_dtype requires is_udt=True. The return type for builtin "
f"dtypes comes from compiling the function for each input type, so a single "
f"fixed type cannot describe it."
)
if parameterized:
raise ValueError(
f"{opclass}: ret_dtype does not work with parameterized=True. "
f"A parameterized operator builds and registers its function when called, "
f"and that inner registration does not accept a return dtype; register the "
f"built function without parameterized=True to declare one."
)
try:
return lookup_dtype(ret_dtype)
except (ValueError, TypeError) as exc:
raise ValueError(
f"{opclass}: ret_dtype={ret_dtype!r} is not a recognized dtype. "
f"Pass a DataType, a numpy dtype, or a name such as 'FP64'."
) from exc


class OpPath:
def __init__(self, parent, name):
self._parent = parent
Expand Down Expand Up @@ -576,21 +609,25 @@ def _deserialize(cls, name, *args):
return cls.register_new(name, *args)

@classmethod
def _deserialize_udf(cls, name, orig_func, is_udt):
def _deserialize_udf(cls, name, orig_func, is_udt, ret_dtype=None):
"""Re-register a named UDF on unpickle, or reuse if already present.

Shared by the five UDF-capable subclasses (UnaryOp, BinaryOp,
IndexUnaryOp, SelectOp, IndexBinaryOp), all of which use the
default ``__reduce__`` below.
default ``__reduce__`` below. ``ret_dtype`` is passed on only when set:
SelectOp shares this path and takes no ret_dtype, and an op without one
pickles as the 3-tuple it always did.
"""
if (rv := cls._find(name)) is not None:
return rv
return cls.register_new(name, orig_func, is_udt=is_udt)
kwargs = {} if ret_dtype is None else {"ret_dtype": ret_dtype}
return cls.register_new(name, orig_func, is_udt=is_udt, **kwargs)

@classmethod
def _deserialize_anon_udf(cls, func, name, is_udt):
def _deserialize_anon_udf(cls, func, name, is_udt, ret_dtype=None):
"""Re-register an anonymous UDF on unpickle."""
return cls.register_anonymous(func, name, is_udt=is_udt)
kwargs = {} if ret_dtype is None else {"ret_dtype": ret_dtype}
return cls.register_anonymous(func, name, is_udt=is_udt, **kwargs)

def __reduce__(self):
"""Default ``__reduce__`` for UDF-capable subclasses.
Expand All @@ -600,13 +637,19 @@ def __reduce__(self):
``Aggregator`` define their own ``__reduce__`` because their pickle
shape differs (they hold a binary op + identity, etc.).
"""
# ``ret_dtype`` rides along only when set, so an op without one pickles
# exactly as before and older versions can still read it.
ret = () if (ret_dtype := getattr(self, "_ret_dtype", None)) is None else (ret_dtype,)
if self._anonymous:
if hasattr(self.orig_func, "_parameterized_info"):
return (_deserialize_parameterized, self.orig_func._parameterized_info)
return (type(self)._deserialize_anon_udf, (self.orig_func, self.name, self._is_udt))
return (
type(self)._deserialize_anon_udf,
(self.orig_func, self.name, self._is_udt, *ret),
)
if (name := f"{self._modname}.{self.name}") in _STANDARD_OPERATOR_NAMES:
return name
return (type(self)._deserialize_udf, (self.name, self.orig_func, self._is_udt))
return (type(self)._deserialize_udf, (self.name, self.orig_func, self._is_udt, *ret))

@classmethod
def _check_supports_udf(cls, method_name):
Expand Down
Loading
Loading