feat: filesystem exorcist worker by ethanholz · Pull Request #1833 · OpenFreeEnergy/openfe

ethanholz · 2026-02-07T22:26:24Z

Adds support for a Filesystem-backed exorcist worker that can run ProtocolUnits from an AlchemicalNetwork. Adding @atravitz and @IAlibay for reference.

Note that this is an implementation that does not require the changes to contexts in OpenFreeEnergy/gufe#753, to keep these features independent, but the expectation is that whenever we merge gufe #753, we should then update the relevant code here to take advantage of that introduced flexibility.

Also note that this PR took some commits from #1824, but circumvents changes to execute_DAG.

resolves #2072 and #1672

Checklist

All new code is appropriately documented (user-facing code must have complete docstrings). see exorcist user-facing docs #2076 for documentation PR
Added a news entry, or the changes are not user-facing: news item will get added in a follow-up when it's clearer what the user-facing bits will be.
Ran pre-commit: you can run pre-commit locally or comment on this PR with pre-commit.ci autofix.

Manual Tests: these are slow so don't need to be run every commit, only before merging and when relevant changes are made (generally at reviewer-discretion).

GPU integration tests
example notebook testing
packaging tests: run this for any large feature PRs or PRs that add test data.

Developers certificate of origin

I certify that this contribution is covered by the MIT License here and the Developer Certificate of Origin at https://developercertificate.org/.

…orage (OpenFreeEnergy#1632) * feat: remove result_server references in favor of gufe external_storage Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * fix: remove extra external_store * test: change from result_server to external_storage * test: remove external_store reference * test: change extension to item per class changes * refactor: remove result server import * chore: remove metadata store from resultclient Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * refactor: remove result server * refactor: remove metadata store --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com>

* refactor(warehouse): rename to warehouse * refactor: rename _ResultContainerTest to _ContainerTest * refactor: rename _Container instances to _DataContainer

…ehouse

* refactor: remove extra implementations We may end up adding these back later but for the time being, we are going to remove them to keep life easy. * refactor: remove load_* and store_* functions Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * refactor: keys are being stored really strangely * refactor(warehouse): huge refactor to simplify working with Warehouse * refactor(warehouse): remove dead code * fix: delete failed due to incorrect object * test: add a test for the FileSystemWarehouse * refactor: remove dead code * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * docs: update warehouse docstrings * feat(warehouse): leverage keyed_chain for object dedup * Update openfe/storage/warehouse.py * import Literal * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * docs(warehouse): change from return, to raises * refactor(warehouse): move _key_exists to exists * chore(warehouse): remove extra todo * chore(warehouse): remove _load_stream * fix(warehouse): deduplicate objects on the filesystem * chore(warehouse): add typehint to exists --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io>

* build with pooch@main to see if hotfix works * add link

…ehouse

* feat(warehouse): add result tokenizable store * docs(warehouse): add docs on the result store * chore: add property for result store * fix: use the correct function signature for handling setup --------- Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io>

…ehouse

codecov · 2026-02-07T23:33:58Z

Codecov Report

❌ Patch coverage is 96.09079% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.45%. Comparing base (755cfd1) to head (e8a0300).
⚠️ Report is 2 commits behind head on epic/execution_improvements.

Files with missing lines	Patch %	Lines
src/openfe/orchestration/__init__.py	88.18%	15 Missing ⚠️
src/openfecli/commands/worker.py	93.75%	4 Missing ⚠️
src/openfe/storage/warehouse.py	86.36%	3 Missing ⚠️
src/openfe/tests/orchestration/test_worker.py	98.43%	3 Missing ⚠️
src/openfecli/plan_alchemical_networks_utils.py	87.50%	3 Missing ⚠️
src/openfe/tests/orchestration/conftest.py	97.22%	1 Missing ⚠️
.../openfe/tests/orchestration/test_exorcist_utils.py	99.29%	1 Missing ⚠️
src/openfecli/commands/plan_rbfe_network.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                       @@
##           epic/execution_improvements    #1833      +/-   ##
===============================================================
+ Coverage                        90.24%   90.45%   +0.20%     
===============================================================
  Files                              209      217       +8     
  Lines                            20475    21239     +764     
===============================================================
+ Hits                             18477    19211     +734     
- Misses                            1998     2028      +30

Flag	Coverage Δ
fast-tests	`90.45% <96.09%> (+0.20%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ethanholz · 2026-02-11T21:15:47Z

Seems I still have some unit testing to finish up, but this should be enough to highlight the use case for interested partners.

…ehouse

ethanholz · 2026-03-06T01:18:26Z

A few notes on how to run this code:

We need to first generate a warehouse that contains everything to run. Currently, only RBFE planning works. When using openfe plan-rbfe-network, you just need to add the --warehouse flag. This will generate a FilesystemWarehouse.

To run a single task you use something like this:

openfe worker warehouse

This will only run a task to completion and then stop. To run this in a loop, you can do something like this: https://gist.github.com/ethanholz/775027b469d6038f3dc958110c635c4c

IAlibay · 2026-03-06T08:37:22Z

@atravitz I think we need to discuss this at some point soon. My understanding was that warehouse & exorcist was going to be decoupled from planning?

ethanholz · 2026-03-06T14:55:27Z

@atravitz I think we need to discuss this at some point soon. My understanding was that warehouse & exorcist was going to be decoupled from planning?

The exorcist implementation is built to accept units as discussed. To demonstrate capabilities, I have added a flag to warehouse so that you can test and validate some of the functionality today.

…ehouse

…o feat/exorcist

…ehouse

…o feat/exorcist

…gy/openfe into feat/exorcist

atravitz

I am approving this for the purpose of getting it merged so that my downstream changes to this can be merged directly into execution_improvements.

* refactor: remove result_server references in favor of gufe ExternalStorage (#1632) * feat: remove result_server references in favor of gufe external_storage Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * fix: remove extra external_store * test: change from result_server to external_storage * test: remove external_store reference * test: change extension to item per class changes * refactor: remove result server import * chore: remove metadata store from resultclient Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * refactor: remove result server * refactor: remove metadata store --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * refactor: rename to WarehouseBaseClass (#1660) * refactor(warehouse): rename to warehouse * refactor: rename _ResultContainerTest to _ContainerTest * refactor: rename _Container instances to _DataContainer * feat/refactor: add setup store (#1671) * refactor: remove extra implementations We may end up adding these back later but for the time being, we are going to remove them to keep life easy. * refactor: remove load_* and store_* functions Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * refactor: keys are being stored really strangely * refactor(warehouse): huge refactor to simplify working with Warehouse * refactor(warehouse): remove dead code * fix: delete failed due to incorrect object * test: add a test for the FileSystemWarehouse * refactor: remove dead code * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * docs: update warehouse docstrings * feat(warehouse): leverage keyed_chain for object dedup * Update openfe/storage/warehouse.py * import Literal * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * docs(warehouse): change from return, to raises * refactor(warehouse): move _key_exists to exists * chore(warehouse): remove extra todo * chore(warehouse): remove _load_stream * fix(warehouse): deduplicate objects on the filesystem * chore(warehouse): add typehint to exists --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io> * Temporarily build pooch from main w/ hotfix (#1806) * build with pooch@main to see if hotfix works * add link * Add support for result tokenizables to warehouse (#1763) * feat(warehouse): add result tokenizable store * docs(warehouse): add docs on the result store * chore: add property for result store * fix: use the correct function signature for handling setup --------- Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io> * test: cleanup warehouse test to be more modular (#1809) * feat: add warehouse primitives for handling protocol units * feat: inital worker for exorcist * test: add tests for warehouse * fix: can now return protocol unit * refactor: make things more consistent * test: initial test setup for orchestration subpackage * test: initial exorcist utility testing * refactor: provide a root path to the exorcist DB * test: inital worker testing * feat: add shared_store * feat: add better handling for CLI application Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * test: add new worker tests * feat: add exorcist worker to CLI * test: add for worker CLI command * docs: add numpy docstrings * feat: add support for planning an RBFE to a Warehouse * fix: correct edge direction for task graph * refactor: remove extra debugging from warehouse * refactor: cleanup handling of tasks for worker * refactor: fix issues in the CLI for running the worker * Update src/openfe/orchestration/__init__.py * fix dict type * add news item * ignore a couple type check violations for now * fix type checking * add exorcist to docs build --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io>

ethanholz and others added 10 commits November 3, 2025 07:56

refactor: rename to WarehouseBaseClass (OpenFreeEnergy#1660)

c27bfa5

* refactor(warehouse): rename to warehouse * refactor: rename _ResultContainerTest to _ContainerTest * refactor: rename _Container instances to _DataContainer

Merge branch 'main' of github.com:OpenFreeEnergy/openfe into feat/war…

8d9995c

…ehouse

Temporarily build pooch from main w/ hotfix (OpenFreeEnergy#1806)

f6b8499

* build with pooch@main to see if hotfix works * add link

Merge branch 'main' of github.com:OpenFreeEnergy/openfe into feat/war…

0e186a0

…ehouse

Merge branch 'main' of github.com:OpenFreeEnergy/openfe into feat/war…

eddba56

…ehouse

Merge branch 'main' of github.com:OpenFreeEnergy/openfe into feat/war…

c86aaea

…ehouse

test: cleanup warehouse test to be more modular (OpenFreeEnergy#1809)

3acdd18

ethanholz changed the title ~~[WIP] feat: exorcist worker~~ feat: exorcist worker Feb 11, 2026

ethanholz marked this pull request as ready for review February 11, 2026 21:13

Merge branch 'main' of github.com:OpenFreeEnergy/openfe into feat/war…

fd24777

…ehouse

ethanholz force-pushed the feat/exorcist branch from 946a9be to 8d4a139 Compare February 18, 2026 17:57

jameseastwood assigned atravitz Feb 23, 2026

Merge branch 'main' into feat/warehouse

70c62d0

atravitz and others added 9 commits June 4, 2026 10:19

Merge branch 'main' of github.com:OpenFreeEnergy/openfe into feat/war…

7ec596b

…ehouse

feat: add warehouse primitives for handling protocol units

d9b0ff7

feat: inital worker for exorcist

0d11be5

test: add tests for warehouse

5135713

fix: can now return protocol unit

f113a3a

refactor: make things more consistent

cb50733

test: initial test setup for orchestration subpackage

682d77c

test: initial exorcist utility testing

8b70c76

refactor: provide a root path to the exorcist DB

25678e8

atravitz added 3 commits July 14, 2026 10:38

Merge branch 'feat/warehouse' of github.com:OpenFreeEnergy/openfe int…

6c7b450

…o feat/exorcist

fix dict type

2fa2e76

Merge branch 'main' of github.com:OpenFreeEnergy/openfe into feat/war…

54e876b

…ehouse

atravitz changed the base branch from feat/warehouse to feat/filesystem-exorcist-worker July 23, 2026 17:25

atravitz changed the base branch from feat/filesystem-exorcist-worker to feat/warehouse July 23, 2026 17:26

atravitz linked an issue Jul 27, 2026 that may be closed by this pull request

filesystem-based exorcist implementation #2072

Closed

atravitz mentioned this pull request Jul 27, 2026

feat: add experimental support to output plan-*-network CLI commands to a FileSystemWarehouse #1672

Closed

atravitz and others added 5 commits July 29, 2026 08:23

Merge branch 'main' into feat/warehouse

86caac0

Merge branch 'main' of github.com:OpenFreeEnergy/openfe into feat/war…

e2edac1

…ehouse

add news item

ee2ccd1

Merge branch 'main' of github.com:OpenFreeEnergy/openfe into feat/war…

801cafe

…ehouse

ignore a couple type check violations for now

755cfd1

atravitz deleted the branch OpenFreeEnergy:epic/execution_improvements July 30, 2026 21:58

atravitz closed this Jul 30, 2026

atravitz reopened this Jul 30, 2026

atravitz changed the base branch from feat/warehouse to epic/execution_improvements July 30, 2026 22:01

atravitz added 4 commits July 30, 2026 15:02

Merge branch 'feat/warehouse' of github.com:OpenFreeEnergy/openfe int…

ee3fd58

…o feat/exorcist

Merge branch 'epic/execution_improvements' of github.com:OpenFreeEner…

12d623f

…gy/openfe into feat/exorcist

fix type checking

93b500e

add exorcist to docs build

e8a0300

atravitz force-pushed the feat/exorcist branch from 61347e6 to e8a0300 Compare July 30, 2026 23:07

atravitz self-requested a review July 30, 2026 23:08

atravitz approved these changes Jul 30, 2026

View reviewed changes

atravitz merged commit 2f467d3 into OpenFreeEnergy:epic/execution_improvements Jul 30, 2026
8 checks passed

atravitz mentioned this pull request Aug 3, 2026

filesystem-based exorcist implementation #2072

Closed

Navigation Menu

Uh oh!

feat: filesystem exorcist worker - #1833

feat: filesystem exorcist worker #1833
atravitz merged 48 commits into
OpenFreeEnergy:epic/execution_improvementsfrom
ethanholz:feat/exorcist

ethanholz commented Feb 7, 2026 •

edited by atravitz

Loading

Uh oh!

codecov Bot commented Feb 7, 2026 •

edited

Loading

Uh oh!

ethanholz commented Feb 11, 2026

Uh oh!

ethanholz commented Mar 6, 2026

Uh oh!

IAlibay commented Mar 6, 2026

Uh oh!

ethanholz commented Mar 6, 2026

Uh oh!

atravitz left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ethanholz commented Feb 7, 2026 • edited by atravitz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Developers certificate of origin

Uh oh!

codecov Bot commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ethanholz commented Feb 11, 2026

Uh oh!

ethanholz commented Mar 6, 2026

Uh oh!

IAlibay commented Mar 6, 2026

Uh oh!

ethanholz commented Mar 6, 2026

Uh oh!

atravitz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ethanholz commented Feb 7, 2026 •

edited by atravitz

Loading

codecov Bot commented Feb 7, 2026 •

edited

Loading