Skip to content

Refactor pyconvert rule storage and API to scoped rules with explicit ordering and update conversions #802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cjdoris wants to merge 4 commits into
base: v1
Choose a base branch
from
Open

Refactor pyconvert rule storage and API to scoped rules with explicit ordering and update conversions #802

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8c4682d
Revise pyconvert rule insertion ordering
cjdoris Jul 19, 2026
b1d103a
Document scoped pyconvert rules
cjdoris Jul 19, 2026
39d26d6
Resolve pyconvert source types from loaded modules
cjdoris Jul 19, 2026
e8af3f2
Store source type name in conversion rules
cjdoris Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@
* Python errors no longer automatically set `sys.last_traceback` etc. when displayed from Julia.
* Added [`fix_qt_plugin_path` preference](@ref pythoncall-config), replacing `CONFIG.auto_fix_qt_plugin_path`.
* Removed `PythonCall.CONFIG`.
* `pyconvert_add_rule(tname, T, func, priority)` has been replaced by
`pyconvert_add_rule(tname, T, S, func)`, where `S` is the rule's Julia scope and
must be a supertype of `T`. Matching ordinary rules are tried newest-first; rules
with scope `Any` provide the default conversion for their Python type.
* Changes to `PythonCall.GC` (now more like `Base.GC`):
* `enable(true)` replaces `enable()`.
* `enable(false)` replaces `disable()`.
Expand Down
51 changes: 40 additions & 11 deletions docs/src/conversion-to-julia.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,24 @@

## [Conversion Rules](@id py2jl-conversion)

The following table specifies the conversion rules used whenever converting a Python object to a Julia object. If the initial Python type matches the "From" column and the desired type `T` intersects with the "To" column, then that conversion is attempted. Conversions are tried in priority order, then in specificity order.
The following table specifies the conversion rules used whenever converting a Python object to a Julia object. If the initial Python type matches the "From" column, the desired type `T` intersects with the "To" column, and `T` is within the rule's scope, then that conversion is attempted. For a union target, it is sufficient for one union component to be within the scope. Matching rules are tried in reverse insertion order, so the most recently added rule is tried first.

From Julia, one can convert Python objects to a desired type using `pyconvert(T, x)` for example.

From Python, the arguments to a Julia function will be converted according to these rules with `T=Any`.

Rules with scope `Any` are canonical: they may be used by `pyconvert(Any, x)`, including
when JuliaCall passes an ordinary Python argument to Julia. Other rules are deliberately
narrower and are only considered when the caller requests a type within their scope.
PythonCall's internal wrapped-Julia and no-copy array rules are always tried before
ordinary rules.

| From | To |
| :----------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------- |
| **Top priority (wrapped values).** | |
| `juliacall.Jl` | `Any` |
| **Very high priority (arrays).** | |
| **Always-first internal conversions (scope `Any`).** | |
| `juliacall.Jl` | `Any` |
| Objects satisfying the buffer or array interface (inc. `bytes`, `bytearray`, `array.array`, `numpy.ndarray`) | `PyArray` |
| **High priority (canonical conversions).** | |
| **Canonical conversions (scope `Any`).** | |
| `None` | `Nothing` |
| `bool` | `Bool` |
| `numbers.Integral` (inc. `int`) | `Integer` (prefers `Int`, or `BigInt` on overflow) |
Expand All @@ -33,7 +38,7 @@ From Python, the arguments to a Julia function will be converted according to th
| `numpy.intXX`/`numpy.uintXX`/`numpy.floatXX` | `IntXX`/`UIntXX`/`FloatXX` |
| `numpy.datetime64` | `NumpyDates.DateTime64` |
| `numpy.timedelta64` | `NumpyDates.TimeDelta64` |
| **Standard priority (other reasonable conversions).** | |
| **Explicitly requested conversions (scope is normally the "To" type).** | |
| `None` | `Missing` |
| `bytes` | `Vector{UInt8}`, `Vector{Int8}`, `String` |
| `str` | `String`, `Symbol`, `Char`, `Vector{UInt8}`, `Vector{Int8}` |
Expand All @@ -52,12 +57,9 @@ From Python, the arguments to a Julia function will be converted according to th
| `numpy.bool_`/`numpy.intXX`/`numpy.uintXX`/`numpy.floatXX` | `Bool`, `Integer`, `Rational`, `Real`, `Number` |
| `numpy.datetime64` | `NumpyDates.InlineDateTime64`, `Dates.DateTime` |
| `numpy.timedelta64` | `NumpyDates.InlineTimeDelta64`, `Dates.Period` |
| Objects satisfying the buffer or array interface | `Array`, `AbstractArray` |
| **Low priority (fallback to `Py`).** | |
| Objects satisfying the buffer or array interface | `Array`, `AbstractArray` (scope `AbstractArray`) |
| **Fallback (scope `Any`).** | |
| Anything | `Py` |
| **Bottom priority (must be explicitly specified by excluding `Py`).** | |
| Objects satisfying the buffer interface | `PyBuffer` |
| Anything | `PyRef` |

See [here](@ref python-wrappers) for an explanation of the `Py*` wrapper types (`PyList`, `PyIO`, etc).

Expand All @@ -66,6 +68,33 @@ See [here](@ref python-wrappers) for an explanation of the `Py*` wrapper types (
To add a custom conversion rule, you must define a function to do the conversion and call
`pyconvert_add_rule` to register it.

The arguments are the Python type name, the Julia target type, the Julia scope, and the
conversion function. The target type must be a subtype of the scope. You may only define
a rule when you own either the Python type or the Julia scope. In particular, use scope
`Any` only for Python types that you own. For a rule converting a foreign Python type to
your own `MyType`, use `MyType` as both target and scope:

```julia
function pyconvert_rule_mytype(::Type{T}, x::Py) where {T}
# Construct a T, or return pyconvert_unconverted() when conversion is not possible.
return pyconvert_return(T(x.some_attribute))
end

function __init__()
PythonCall.pyconvert_add_rule(
"some_package:SomePythonType",
MyType,
MyType,
pyconvert_rule_mytype,
)
end
```

This rule is considered for `pyconvert(MyType, x)`, but not for `pyconvert(Any, x)`.
Consequently, loading a package that defines such a rule does not change JuliaCall's
default conversion of unrelated Python values. When multiple matching ordinary rules
exist, the rule added most recently is tried first.

You must not do this while precompiling, so these calls will normally be in the `__init__`
function of your module.

Expand Down
39 changes: 39 additions & 0 deletions docs/src/v1-migration-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,45 @@ The `PythonCall.CONFIG.auto_fix_qt_plugin_path` config has been replaced with th
`pkg> preference add PythonCall fix_qt_plugin_path=false` or the env var
`JULIA_PYTHONCALL_FIX_QT_PLUGIN_PATH=0`.

## Python-to-Julia conversion rules

`pyconvert_add_rule` now takes a Julia scope instead of a priority:

```julia
# v0.9
pyconvert_add_rule(
"some_package:SomePythonType",
MyType,
pyconvert_rule_mytype,
PYCONVERT_PRIORITY_NORMAL,
)

# v1
pyconvert_add_rule(
"some_package:SomePythonType",
MyType,
MyType,
pyconvert_rule_mytype,
)
```

The target type must be a subtype of the scope. A rule is considered only when the
requested Julia target is within that scope; for a union target, at least one component
must be within it. Ordinary matching rules are tried in reverse insertion order, so the
most recently registered rule is tried first.

You may only add a rule when you own either its Python source type or its Julia scope.
Use scope `Any` only when you own the Python type. This replaces canonical (or higher)
priority: such a rule can affect `pyconvert(Any, x)` and therefore JuliaCall's default
argument conversion. When converting a foreign Python type to a Julia type that you own,
normally use that Julia type as the scope. The rule will then be used only when callers
explicitly request your type and cannot change the default conversion merely because your
package was loaded.

The constants `PYCONVERT_PRIORITY_WRAP`, `PYCONVERT_PRIORITY_ARRAY`,
`PYCONVERT_PRIORITY_CANONICAL`, `PYCONVERT_PRIORITY_NORMAL`, and
`PYCONVERT_PRIORITY_FALLBACK` have been removed.

## `PythonCall.GC`

This submodule has been changed to closer mimic the `Base.GC` API.
Expand Down
5 changes: 0 additions & 5 deletions src/API/exports.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,11 +104,6 @@ export unsafe_pydel
export @pyconvert
export pyconvert
export pyconvert_add_rule
export PYCONVERT_PRIORITY_ARRAY
export PYCONVERT_PRIORITY_CANONICAL
export PYCONVERT_PRIORITY_FALLBACK
export PYCONVERT_PRIORITY_NORMAL
export PYCONVERT_PRIORITY_WRAP
export pyconvert_return
export pyconvert_unconverted

Expand Down
10 changes: 0 additions & 10 deletions src/API/types.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,3 @@
# Convert

@enum PyConvertPriority begin
PYCONVERT_PRIORITY_WRAP = 400
PYCONVERT_PRIORITY_ARRAY = 300
PYCONVERT_PRIORITY_CANONICAL = 200
PYCONVERT_PRIORITY_NORMAL = 0
PYCONVERT_PRIORITY_FALLBACK = -100
end

# Core

"""
Expand Down
4 changes: 2 additions & 2 deletions src/Convert/Convert.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ import ..PythonCall:
pyconvert_add_rule,
pyconvert_return,
pyconvert_unconverted,
pyconvert,
PyConvertPriority
pyconvert

export pyconvert_isunconverted,
pyconvert_add_rule_high_priority,
pyconvert_result,
pyconvert_result,
pyconvert_tryconvert,
Expand Down
22 changes: 11 additions & 11 deletions src/Convert/ctypes.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,18 +47,18 @@ function init_ctypes()
rule = pyconvert_rule_ctypessimplevalue{T,false}()
saferule = pyconvert_rule_ctypessimplevalue{T,true}()

t == "char_p" && pyconvert_add_rule(name, Cstring, saferule)
t == "wchar_p" && pyconvert_add_rule(name, Cwstring, saferule)
pyconvert_add_rule(name, T, saferule)
isuint && pyconvert_add_rule(name, UInt, sizeof(T) ≤ sizeof(UInt) ? saferule : rule)
isuint && pyconvert_add_rule(name, Int, sizeof(T) < sizeof(Int) ? saferule : rule)
t == "char_p" && pyconvert_add_rule(name, Cstring, Cstring, saferule)
t == "wchar_p" && pyconvert_add_rule(name, Cwstring, Cwstring, saferule)
pyconvert_add_rule(name, T, T, saferule)
isuint && pyconvert_add_rule(name, UInt, UInt, sizeof(T) ≤ sizeof(UInt) ? saferule : rule)
isuint && pyconvert_add_rule(name, Int, Int, sizeof(T) < sizeof(Int) ? saferule : rule)
isint &&
!isuint &&
pyconvert_add_rule(name, Int, sizeof(T) ≤ sizeof(Int) ? saferule : rule)
isint && pyconvert_add_rule(name, Integer, rule)
isfloat && pyconvert_add_rule(name, Float64, saferule)
isreal && pyconvert_add_rule(name, Real, rule)
isnumber && pyconvert_add_rule(name, Number, rule)
isptr && pyconvert_add_rule(name, Ptr, saferule)
pyconvert_add_rule(name, Int, Int, sizeof(T) ≤ sizeof(Int) ? saferule : rule)
isint && pyconvert_add_rule(name, Integer, Integer, rule)
isfloat && pyconvert_add_rule(name, Float64, Float64, saferule)
isreal && pyconvert_add_rule(name, Real, Real, rule)
isnumber && pyconvert_add_rule(name, Number, Number, rule)
isptr && pyconvert_add_rule(name, Ptr, Ptr, saferule)
end
end
72 changes: 40 additions & 32 deletions src/Convert/numpy.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,49 +112,57 @@ function init_numpy()
rule = pyconvert_rule_numpysimplevalue{T,false}()
saferule = pyconvert_rule_numpysimplevalue{T,true}()

pyconvert_add_rule(name, T, saferule, PYCONVERT_PRIORITY_ARRAY)
isuint && pyconvert_add_rule(name, UInt, sizeof(T) ≤ sizeof(UInt) ? saferule : rule)
isuint && pyconvert_add_rule(name, Int, sizeof(T) < sizeof(Int) ? saferule : rule)
isuint && pyconvert_add_rule(name, UInt, UInt, sizeof(T) ≤ sizeof(UInt) ? saferule : rule)
isuint && pyconvert_add_rule(name, Int, Int, sizeof(T) < sizeof(Int) ? saferule : rule)
isint &&
!isuint &&
pyconvert_add_rule(name, Int, sizeof(T) ≤ sizeof(Int) ? saferule : rule)
isint && pyconvert_add_rule(name, Integer, rule)
isfloat && pyconvert_add_rule(name, Float64, saferule)
isreal && pyconvert_add_rule(name, Real, rule)
iscomplex && pyconvert_add_rule(name, ComplexF64, saferule)
iscomplex && pyconvert_add_rule(name, Complex, rule)
isnumber && pyconvert_add_rule(name, Number, rule)
pyconvert_add_rule(name, Int, Int, sizeof(T) ≤ sizeof(Int) ? saferule : rule)
isint && pyconvert_add_rule(name, Integer, Integer, rule)
isfloat && pyconvert_add_rule(name, Float64, Float64, saferule)
isreal && pyconvert_add_rule(name, Real, Real, rule)
iscomplex && pyconvert_add_rule(name, ComplexF64, ComplexF64, saferule)
iscomplex && pyconvert_add_rule(name, Complex, Complex, rule)
isnumber && pyconvert_add_rule(name, Number, Number, rule)
end

# datetime64
pyconvert_add_rule(
pyconvert_add_rule("numpy:datetime64", InlineDateTime64, InlineDateTime64, pyconvert_rule_datetime64)
pyconvert_add_rule("numpy:datetime64",
NumpyDates.DatesInstant,
NumpyDates.DatesInstant,
pyconvert_rule_datetime64)
pyconvert_add_rule("numpy:datetime64", Missing, Missing, pyconvert_rule_datetime64)
pyconvert_add_rule("numpy:datetime64", Nothing, Nothing, pyconvert_rule_datetime64)

# timedelta64
pyconvert_add_rule("numpy:timedelta64", InlineTimeDelta64, InlineTimeDelta64, pyconvert_rule_timedelta64)
pyconvert_add_rule("numpy:timedelta64",
NumpyDates.DatesPeriod,
NumpyDates.DatesPeriod,
pyconvert_rule_timedelta64)
pyconvert_add_rule("numpy:timedelta64", Missing, Missing, pyconvert_rule_timedelta64)
pyconvert_add_rule("numpy:timedelta64", Nothing, Nothing, pyconvert_rule_timedelta64)
end

function init_numpy_high_priority()
for (t, T) in NUMPY_SIMPLE_TYPES
pyconvert_add_rule_high_priority(
"numpy:$t",
T,
Any,
pyconvert_rule_numpysimplevalue{T,true}(),
)
end
pyconvert_add_rule_high_priority(
"numpy:datetime64",
DateTime64,
Any,
pyconvert_rule_datetime64,
PYCONVERT_PRIORITY_ARRAY,
)
pyconvert_add_rule("numpy:datetime64", InlineDateTime64, pyconvert_rule_datetime64)
pyconvert_add_rule(
"numpy:datetime64",
NumpyDates.DatesInstant,
pyconvert_rule_datetime64,
)
pyconvert_add_rule("numpy:datetime64", Missing, pyconvert_rule_datetime64)
pyconvert_add_rule("numpy:datetime64", Nothing, pyconvert_rule_datetime64)

# timedelta64
pyconvert_add_rule(
pyconvert_add_rule_high_priority(
"numpy:timedelta64",
TimeDelta64,
pyconvert_rule_timedelta64,
PYCONVERT_PRIORITY_ARRAY,
)
pyconvert_add_rule("numpy:timedelta64", InlineTimeDelta64, pyconvert_rule_timedelta64)
pyconvert_add_rule(
"numpy:timedelta64",
NumpyDates.DatesPeriod,
Any,
pyconvert_rule_timedelta64,
)
pyconvert_add_rule("numpy:timedelta64", Missing, pyconvert_rule_timedelta64)
pyconvert_add_rule("numpy:timedelta64", Nothing, pyconvert_rule_timedelta64)
end
10 changes: 4 additions & 6 deletions src/Convert/pandas.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,9 @@ pyconvert_rule_pandas_na(::Type{Nothing}, x::Py) = pyconvert_return(nothing)
pyconvert_rule_pandas_na(::Type{Missing}, x::Py) = pyconvert_return(missing)

function init_pandas()
pyconvert_add_rule(
"pandas._libs.missing:NAType",
pyconvert_add_rule("pandas.api.typing:NAType",
Missing,
pyconvert_rule_pandas_na,
PYCONVERT_PRIORITY_CANONICAL,
)
pyconvert_add_rule("pandas._libs.missing:NAType", Nothing, pyconvert_rule_pandas_na)
Any,
pyconvert_rule_pandas_na)
pyconvert_add_rule("pandas.api.typing:NAType", Nothing, Nothing, pyconvert_rule_pandas_na)
end
Loading
Loading