gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__ (alternative2)

[dolfinus]

• Issue: _abc._abc_subclasscheck has very poor performance and (I think) a memory leak #92810

Alternative implementation of #131914 which does not check __subclasses__ recursively.
To handle cases like:

class Number(ABC): ...

class Real(Number): ...

class Integral(Real): ...

Integral.register(int)

assert issubclass(int, Number) is True

which previously were implemented via recursive __subclasses__ check, method cls.register(subclass) is calling super(cls).register(subclass) recursively ("bubble-up" registration to all the parents).

For benchmark from #131914:

sudo ./python -m pyperf system tune
taskset -c 0 ./python benchmark.py --metaclass abc.ABCMeta --rounds 3 --classes 5000
taskset -c 0 ./python benchmark.py --metaclass _py_abc.ABCMeta --rounds 3 --classes 5000

Impl	Max memory before, MB	Max memory after, MB
_abc	6331	50
_py_abc	4422	61

Impl	Total time before	Total time after
_abc	6m 16s	1m 23s
_py_abc	8m 48s	3m 53s

Check	Impl	before	after	Impl	before	after
isinstance(child, Parent)	_abc	0.115us 4MiB...15MiB	0.121us 2MiB...15MiB	_py_abc	0.212us 11MiB...24MiB	0.218us 11MiB...24MiB
issubclass(Child, Parent)	_abc	0.105us 0MiB...1MiB	0.111us 0MiB...1MiB	_py_abc	0.203us 6MiB...8MiB	0.208us 5MiB...8MiB
isinstance(child, Grandparent)	_abc	0.112us 0MiB...2MiB	0.119us 1MiB...1MiB	_py_abc	0.210us 4MiB...7MiB	0.216us 4MiB...7MiB
issubclass(Child, Grandparent)	_abc	0.103us 0MiB	0.108us 0MiB	_py_abc	0.201us 0MiB...1MiB	0.207us 0MiB...1MiB
not isinstance(child, Sibling)	_abc	0.113us 4MiB...14MiB	0.121us 3MiB...14MiB	_py_abc	0.348us 13MiB...23MiB	0.354us 13MiB...22MiB
not issubclass(Child, Sibling)	_abc	0.105us 1MiB	0.110us 1MiB...2MiB	_py_abc	0.328us 8MiB...10MiB	0.332us 9MiB...11MiB
not isinstance(child, Cousin)	_abc	0.115us 1MiB...2MiB	0.121us 1MiB...2MiB	_py_abc	0.350us 7MiB...9MiB	0.354us 7MiB...10MiB
not issubclass(Child, Cousin)	_abc	0.104us 0MiB	0.110us 0MiB...1MiB	_py_abc	0.329us 4MiB	0.333us 4MiB...5MiB
not isinstance(child, Uncle)	_abc	7.268us 6174MiB...6333MiB	0.125us 0MiB...1MiB	_py_abc	9.957us 4382MiB...4422MiB	0.360us 6MiB
not issubclass(Child, Uncle)	_abc	7.099us 6171MiB	0.114us 0MiB	_py_abc	9.936us 4380MiB	0.336us 4MiB...5MiB

Memory increment is measured during isinstance() / issubclass() calls, not during preparation, like class creation or registration where actual registry allocation is performed. So memory usage in tables below is almost always 0.

Timing drop in _py_abc implementation for 2 first rows is due to if subclass in cls._abc_registry: check added to match _abc.c implementation.

Check	Impl	before	after	Impl	before	after
isinstance(child, Parent.register)	_abc	0.273us 0MiB	0.272us 0MiB	_py_abc	0.440us 0MiB	0.275us 0MiB
issubclass(Child, Parent.register)	_abc	0.154us 0MiB	0.157us 0MiB	_py_abc	0.427us 0MiB	0.261us 0MiB
isinstance(child, Grandparent.register)	_abc	0.114us 0MiB	0.120us 0MiB	_py_abc	0.253us 0MiB	0.262us 0MiB
issubclass(Child, Grandparent.register)	_abc	0.103us 0MiB	0.109us 0MiB	_py_abc	0.240us 0MiB	0.247us 0MiB
not isinstance(child, Sibling.register)	_abc	0.027us 0MiB	0.030us 1MiB	_py_abc	0.028us 0MiB	0.029us 2MiB
not issubclass(Child, Sibling.register)	_abc	0.018us 0MiB	0.018us 1MiB	_py_abc	0.018us 0MiB	0.018us 2MiB
not isinstance(child, Cousin.register)	_abc	0.028us 0MiB	0.028us 2MiB	_py_abc	0.028us 0MiB	0.029us 3MiB
not issubclass(Child, Cousin.register)	_abc	0.018us 0MiB	0.018us 2MiB	_py_abc	0.019us 0MiB	0.018us 3MiB
not isinstance(child, Uncle.register)	_abc	0.249us 0MiB	0.233us 2MiB	_py_abc	0.843us 0MiB	0.866us 3MiB
not issubclass(Child, Uncle.register)	_abc	0.238us 0MiB	0.227us 2MiB	_py_abc	0.815us 0MiB	0.839us 3MiB

Just to check that nothing is broken:

Check	Impl	before	after	Impl	before	after
not isinstance(child, Unrelated)	_abc	0.028us 0MiB	0.028us 0MiB	_py_abc	0.028us 0MiB	0.029us 0MiB
not issubclass(Child, Unrelated)	_abc	0.018us 0MiB	0.018us 0MiB	_py_abc	0.018us 0MiB	0.018us 0MiB
not isinstance(child, UnrelatedABC)	_abc	0.118us 0MiB	0.121us 0MiB	_py_abc	0.469us 0MiB	0.466us 0MiB
not issubclass(Child, UnrelatedABC)	_abc	0.110us 0MiB	0.112us 0MiB	_py_abc	0.442us 0MiB	0.442us 0MiB

Flamegraphs for _py_abc impl and test issubclass_uncle (the most time and memory consuming case on main):
main_vs_pr150540.tar.gz

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #32877634 | 📁 Comparing 49e8b49 against main (9242700)

  🔍 Preview build  

2 files changed

± whatsnew/3.16.html
± whatsnew/changelog.html