Skip to content

Fix compatibility with Python 3.13 and Windows, enable --json flag su… - #343

Closed
OlaszPL wants to merge 2 commits into
WikiExtractor:masterfrom
OlaszPL:master
Closed

Fix compatibility with Python 3.13 and Windows, enable --json flag su…#343
OlaszPL wants to merge 2 commits into
WikiExtractor:masterfrom
OlaszPL:master

Conversation

@OlaszPL

@OlaszPL OlaszPL commented May 1, 2025

Copy link
Copy Markdown

…pport and preserve bullet points in text

@patrickxia

Copy link
Copy Markdown

This breaks it for (at least) Linux.

RuntimeError: A SemLock created in a fork context is being shared with a process in a spawn context. This is not supported. Please use the same context to create multiprocessing objects and Process.

@AngledLuffa

Copy link
Copy Markdown
Collaborator

The biggest limitation to using spawn - which I have not seen anyone address before, including this PR - is that everything needs to be passed to the spawns. With fork the children processes simply get a memory reference to the existing templates etc. With spawn those are initilized to empty.

https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Simply passing the entire templates hash would not work, either, as for large dumps that would be extremely slow and use a ton of memory. I am working on this, but not quite finished.

The other changes, such as to_json, regex updates, etc, have been merged in with other changes.

Thank you for the contribution, though. If you find more errors (starting from master), we'll be here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants