(Draft) New Data Model by SKernchen · Pull Request #480 · softwarepub/hermes

SKernchen · 2026-04-29T09:21:10Z

This pull request is not ready yet. It is only for showing the process

cont.active_ctx is very special with regards to the implementation of a dependency. This is ugly and tests will be added to test_pyld_utils.

The conversion tests (`_to_python` and `_to_expanded_json`) are pretty long now...

This reverts commit 4fe4ffb.

This reverts commit df5b0ef.

This reverts commit d825217.

This reverts commit 8b4f9a3.

Pull in relevant changes from `develop`

Disable tests that fail due to data model

Resolve #383: Test `ld_context`

…or/381-test-ld_container # Conflicts: # poetry.lock

Fix #454: Add end-to-end tests for the plugin-related `SoftwareMetadata` API

…c-api Refactor/423 implement public api

zyzzyxdonta

Some first very high-level comments. Will continue the review another day.

zyzzyxdonta · 2026-05-26T14:33:09Z

Should a top-level conftest exist? I think this belongs into test/ or test/hermes_test/

zyzzyxdonta · 2026-05-26T14:36:48Z

+    "pynacl>=1.5.0, <2.0.0",
+    "rdflib (>=7.1.4,<8.0.0)",
+    "schemaorg (>=0.1.1,<0.2.0)",
+    "tomlkit (>=0.14.0,<0.15.0)",


This is our second toml dependency (we already had toml. And there is also tomli as a transitive dependency. It might make sense to consolidate this into one implementation at some point, if possible.

zyzzyxdonta · 2026-05-26T14:50:44Z

+codemeta = "hermes.commands.process.standard_merge:CodemetaProcessPlugin"
+
+[project.entry-points."hermes.curate"]
+pass_curate = "hermes.commands.curate.pass_curate:DoNothingCuratePlugin"


"do nothing" and "pass" are two different names for the same thing. We should make up our minds and choose one of the two.

zyzzyxdonta · 2026-06-03T12:22:29Z

(Without having read any of the docs,) I find the jargon strange in this module. The module is called context_manager. It contains HermesCache which is a context manager, and HermesContext which is not a context manager but returns context managers. Maybe this could be cleared up by renaming the module. Maybe nothing with context in it as this term is already used for the context of the JSON-LD object.

zyzzyxdonta · 2026-06-03T12:27:17Z

+from .error import HermesContextError
+
+
+class HermesCache:


It seems this class is only ever instantiated in HermesContext. So it might be a good idea to add a docstring to the module that explains how to get such a cache object (i.e. via HermesContext).

zyzzyxdonta · 2026-06-03T12:29:44Z

+
+class HermesValidationError(Exception):
+    """
+    This exception should be thrown when input validation (e.g., during harvest) occurs.


Suggested change

This exception should be thrown when input validation (e.g., during harvest) occurs.

This exception should be raised when an error occurs during input validation (e.g., during harvest).

zyzzyxdonta · 2026-06-03T12:33:30Z

+
+class HermesContextError(Exception):
+    """
+    This exception should be thrown when interacting with the model context.


Suggested change

This exception should be thrown when interacting with the model context.

This exception should be raised when interacting with the model context.

zyzzyxdonta · 2026-06-03T12:34:17Z

+    ) -> None:
+        """
+        Create a new ld_merge_list.
+        For further information on this function and the errors it throws see :meth:`ld_list.__init__`.


Suggested change

For further information on this function and the errors it throws see :meth:`ld_list.__init__`.

For further information on this function and the exceptions it raises see :meth:`ld_list.__init__`.

zyzzyxdonta · 2026-06-03T12:45:45Z

+    """
+
+    @classmethod
+    def vocabulary(cls, base_url: str = "http://spam.eggs/") -> dict:


Only example.org and example.com are safe test domains

zyzzyxdonta · 2026-06-03T13:01:09Z

+    dict[str, Union["JSON_LD_VALUE", BASIC_TYPE, TIME_TYPE, "ld_dict", "ld_list"]],
+]
+""" Type description of valid JSON_LD objects that are partially represented by ld_containers """
+PYTHONIZED_LD_CONTAINER: TypeAlias = Union[


What does pythonized mean? 😅 The Python representation of the thing?

zyzzyxdonta · 2026-06-03T14:28:39Z

+        for mapping in active_ctx["mappings"].values():
+            if "@container" in mapping and long_iri:
+                value = {x: "none" for x in mapping["@container"]}


I don't understand what this is for.

The and long_iri I guess saves work if no IRI is given. But this could be expressed more clearly by doing if long_iri is None: return long_iri at the beginning of the function, just like _compact_iri() does it.

value overwritten in each new iteration in each iteration because it @container can only exist once? In that case, this would be clearer by breaking out of the loop.

But what does "none" do?

I think this whole module goes deep into pyld internals and a couple of comments would be helpful to understand what is going on.

zyzzyxdonta · 2026-06-04T08:55:17Z


-You should not need to interact with this data directly.
-Instead, use {class}`hermes.model.context.HermesContext` and respective subclasses to access the data in a consistent way.
+Output of the different `hermes` commands consequently is valid JSON-LD, serialized as JSON, that is cached in 


valid JSON-LD, serialized as JSON

Obviously 😁

Suggested change

Output of the different `hermes` commands consequently is valid JSON-LD, serialized as JSON, that is cached in

Output of the different `hermes` subcommands consequently are valid JSON-LD files that are cached in

zyzzyxdonta · 2026-06-04T08:57:03Z

-You should not need to interact with this data directly.
-Instead, use {class}`hermes.model.context.HermesContext` and respective subclasses to access the data in a consistent way.
+Output of the different `hermes` commands consequently is valid JSON-LD, serialized as JSON, that is cached in 
+subdirectories of the `.hermes/` directory that is created in the root of the project directory.


Maybe add a little directory tree like this (but updated with current file names):

.hermes ├── harvest │ ├── cff_contexts.json │ ├── cff.json │ ├── file_exists_contexts.json │ └── file_exists.json └── process ├── hermes.json └── tags.json

(tree .hermes in Linux or tree /f .hermes in Powershell on Windows)

zyzzyxdonta · 2026-06-04T09:01:30Z

+Output of the different `hermes` commands consequently is valid JSON-LD, serialized as JSON, that is cached in 
+subdirectories of the `.hermes/` directory that is created in the root of the project directory.

+The cache is purely for internal purposes, its data should not be interacted with.


Maybe the other way around:

Suggested change

The cache is purely for internal purposes, its data should not be interacted with.

The cache should only be interacted with via the `hermes` libraries.

zyzzyxdonta · 2026-06-04T09:08:19Z

+
+The following sections show how this class works. 
+
+##### Creating a data model instance


h5 is too many levels down. It's absolutely fine to have multiple paragraphs with multiple sentences each in a subdivision.

zyzzyxdonta · 2026-06-04T09:36:16Z

With the given poetry.lock, the docs don't compile. A lot of our packages are old and incompatible with the ones that aren't pinned. I got it to work like this:

diff --git i/pyproject.toml w/pyproject.toml index 17ea508..932427b 100644 --- i/pyproject.toml +++ w/pyproject.toml @@ -92,13 +92,13 @@ pytest-httpserver = "^1.1.5" optional = true [tool.poetry.group.docs.dependencies] -Sphinx = "^6.2.1" +Sphinx = "^8.0.0" # Sphinx - Additional modules -myst-parser = "^2.0.0" +myst-parser = "^4.0.0" sphinx-book-theme = "^1.0.1" sphinx-favicon = "^0.2" sphinxcontrib-contentui = "^0.2.5" -sphinxcontrib-images = "^0.9.4" +sphinxcontrib-images = "^1.0.1" sphinx-icon = "^0.1.2" sphinx-autobuild = "^2021.3.14" sphinx-autoapi = "^3.0.0"

zyzzyxdonta · 2026-06-09T07:15:07Z

+:caption: Injecting additional schemas
+from hermes.model import SoftwareMetadata
+
+# Contents served at https://bar.net/schema.jsonld:


Please only use reserved example domains! https://en.wikipedia.org/wiki/Example.com

zyzzyxdonta · 2026-06-09T07:17:18Z

+it will always be returned in a **list**-like object!
+```
+
+The reason for providing data in list-like objects is that JSON-LD treats all property values as arrays.


Link to the related section in the JSON-LD spec would be nice

zyzzyxdonta · 2026-06-09T07:25:20Z

+As mentioned in the [introduction to the data model](#data-model), 
+`hermes` uses a JSON-LD-like internal data model. 
+The API class {class}`hermes.model.SoftwareMetadata` hides many
+of the more complex aspects of JSON-LD and makes it easy to work
+with the data model.


This feels like a misplaced summary. I would just remove it.

zyzzyxdonta · 2026-06-09T07:26:43Z

+Python data:
+
+```{code-block} python
+:caption: Naive containment assertion that raises


It doesn't though. The mock output is from the path where the assertion holds.

zyzzyxdonta · 2026-06-09T07:28:14Z

+
+## See Also
+
+- API reference: {class}`hermes.model.SoftwareMetadata`


This link never works throughout the whole docs. For some reason, the class is only exposed in the docs as hermes.model.api.SoftwareMetadata

zyzzyxdonta · 2026-06-09T09:38:26Z

-The full code and structure is available at  [hermes-plugin-git](https://github.com/softwarepub/hermes-plugin-git).
+This tutorial will present the basic steps for writing additional plugins.
+
+The full code and structure of a harvest plugin is available at [hermes-plugin-git](https://github.com/softwarepub/hermes-plugin-git).


Emphasize that this is an example of a "standalone" (as in out-of-hermes-source) harvest plugin.

zyzzyxdonta · 2026-06-09T09:38:52Z

+
+The full code and structure of a harvest plugin is available at [hermes-plugin-git](https://github.com/softwarepub/hermes-plugin-git).
 This plugin extracts information from the local git history.
 The hermes-plugin-git will help to gather contributing and branch metadata.


Suggested change

The hermes-plugin-git will help to gather contributing and branch metadata.

The hermes-plugin-git will help to gather contribution and branch metadata.

zyzzyxdonta · 2026-06-09T09:39:34Z

 The hermes-plugin-git will help to gather contributing and branch metadata.
+
 ```{note}
 For this tutorial you should be familiar with HERMES. 


Suggested change

For this tutorial you should be familiar with HERMES.

To follow this tutorial you should be familiar with HERMES.

zyzzyxdonta · 2026-06-09T09:40:16Z

-If you never used HERMES before, you might want to check the tutorial: [Automated Publication with HERMES](https://docs.software-metadata.pub/en/latest/tutorials/automated-publication-with-ci.html).
+If you never used HERMES before, you might want to check the tutorial: [Automated Publication with HERMES](./automated-publication-with-ci).
+
+Also all metadata directly handled by HERMES is [JSON-LD](https://json-ld.org/) so you should be familiar with that when writing a plugin.


Link to the "JSON-LD for plugin developers" guide instead.

zyzzyxdonta · 2026-06-09T09:40:50Z

+If you never used HERMES before, you might want to check the tutorial: [Automated Publication with HERMES](./automated-publication-with-ci).
+
+Also all metadata directly handled by HERMES is [JSON-LD](https://json-ld.org/) so you should be familiar with that when writing a plugin.
+And uses the [schmea.org](https://schema.org/) (with prefix "schema") and the [CodeMeta](https://codemeta.github.io/) (without prefix) context.


Suggested change

And uses the [schmea.org](https://schema.org/) (with prefix "schema") and the [CodeMeta](https://codemeta.github.io/) (without prefix) context.

And uses the [schema.org](https://schema.org/) (with prefix "schema") and the [CodeMeta](https://codemeta.github.io/) (without prefix) context.

zyzzyxdonta · 2026-06-09T11:20:14Z

+}
+```
+
+HERMES would use the {py:class}`~hermes.model.merge.action.Reject` strategy for merging values of the key `full_property_iri` in objects of type `full_type_iri`. (A key in strategies being `None` instead of a string indicates to HERMES that its value is to be used as a default [i.e. if no more specific entry exists].)


This feels hacky. Have you considered collections.defaultdict?

zyzzyxdonta · 2026-06-09T11:22:32Z

This whole file feels more like a reference than a tutorial. A tutorial should be "learning by doing". An example could be having the reader re-build the git-plugin from scratch.

See also: https://diataxis.fr/tutorials/

zyzzyxdonta · 2026-06-09T11:30:40Z

+
+## Configure HERMES to use your plugin
+
+To integrate your plugin, you have to register it as a plugin in the `pyproject.toml`.


I would prefer calling the thing in the pyproject.toml "entry point". That's also how it's called in the PEP: https://peps.python.org/pep-0621/#entry-points

Suggested change

To integrate your plugin, you have to register it as a plugin in the `pyproject.toml`.

To integrate your plugin, you have to register it as an entry point in the `pyproject.toml`.

As https://packaging.python.org/en/latest/guides/writing-pyproject-toml/ also uses the word plugin, I think it would be best to also mention it here, but then use the term "entry point" for consistency and to differentiate the HERMES plugin and the line in the pyproject.toml.

zyzzyxdonta · 2026-06-09T11:33:43Z

+## Configure HERMES to use your plugin
+
+To integrate your plugin, you have to register it as a plugin in the `pyproject.toml`.
 To learn more about the `pyproject.toml` check https://python-poetry.org/docs/pyproject/ or refer to [PEP621](https://peps.python.org/pep-0621/).


I would remove the Poetry mention. Instead we can link to https://packaging.python.org/en/latest/guides/writing-pyproject-toml/ which gives an agnostic overview.

zyzzyxdonta · 2026-06-09T11:36:51Z

@@ -113,59 +313,76 @@ This variant is used to contribute to the HERMES community or adapt the HERMES w
 If you want to contribute, see the [Contribution Guidelines](https://docs.software-metadata.pub/en/latest/dev/contribute.html).
 After cloning the HERMES workflow repository you can adapt the pyproject.toml.
 In the code below you see the parts with the important lines.
-```{code-block} toml
+```{code-block}
 ...
 [tool.poetry.dependencies]
 ...
 pydantic-settings = "^2.1.0"
-hermes-plugin-git = { git = "https://github.com/softwarepub/hermes-plugin-git.git", branch = "main" }
+{plugin_package} = { {plugin_name} = "{link_to_your_repo}", branch = "main" }
 ...
 ...
-[tool.poetry.plugins."hermes.harvest"]
-cff = "hermes.commands.harvest.cff:CffHarvestPlugin"
-codemeta = "hermes.commands.harvest.codemeta:CodeMetaHarvestPlugin"
-git = "hermes_plugin_git.harvest:GitHarvestPlugin"
+[tool.poetry.plugins."hermes.{plugin_step}"]
+{plugin_name} = "{plugin_package}.{plugin_module}:{plugin_class}"
 ...
 ```
 In the dependencies you have to install your plugin. If your Plugin is pip installable than you can just give the name and the version.
 If your plugin is in a buildable git repository, you can install it with the given expression. 
 Note that this differs with the accessibility and your wishes, check [Explicit Package Sources](https://python-poetry.org/docs/repositories/#explicit-package-sources).

 The second thing to adapt is to declare the access point for the plugin.
-You can do that with `git = "hermes_plugin_git.harvest:GitHarvestPlugin"`.
-This expression makes the `GitHarvestPlugin` from the `hermes_plugin_git` package, a `hermes.harvest` plugin named `git`.
+You can do that with `{plugin_name} = "{plugin_package}.{plugin_module}:{plugin_class}"`.
+This expression makes the `plugin_class` from the `plugin_package` package, a `hermes.{plugin_step}` plugin named `plugin_name`.
 So you need to configure this line with your plugin properties.

 Now you just need to add the plugin to the `hermes.toml` and reinstall the adapted poetry package.


Same here. Poetry mentions should be removed. Instead we can use the generic version [project.entry-points."hermes..."] now. All of the Poetry stuff was written here because pyproject.toml support was poor in other tools years ago, when this was written.

led02 and others added 30 commits August 1, 2025 21:11

Add test for helpers.

a527fb0

Using mock_document for setup.

2fbe1e0

Remove test for pyld internals.

e362b15

cont.active_ctx is very special with regards to the implementation of a dependency. This is ugly and tests will be added to test_pyld_utils.

Switch test to use mock_context from helpers.

8b2e59f

Added more tests.

4b8c3b7

The conversion tests (`_to_python` and `_to_expanded_json`) are pretty long now...

Restructured tests for ld_container (type conversions).

3bceb49

Update poetry lock to get CI running.

ef2eb70

Fix linting errors.

af21ace

Delete tests for old data model.

8b4f9a3

Disable tests that fail due to missing old data model.

d825217

Fix test for mock document.

d9761ab

Fix mock document.

9113c7b

Silence flake8 on inactive tests.

df5b0ef

Silence flake8 on inactive tests more agressively.

4fe4ffb

Fix handling of identifier in compaction and expansion.

797ded3

Disable tests that fail due to missing old data model.

026b09f

Silence flake8 on inactive tests.

b347dc2

Silence flake8 on inactive tests more agressively.

7a2b0d6

Revert "Silence flake8 on inactive tests more agressively."

d577aab

This reverts commit 4fe4ffb.

Revert "Silence flake8 on inactive tests."

b432ccf

This reverts commit df5b0ef.

Revert "Disable tests that fail due to missing old data model."

6a10129

This reverts commit d825217.

Delete tests for old data model.

82b45e5

Revert "Delete tests for old data model."

de09fdb

This reverts commit 8b4f9a3.

Merge branch 'feature/347-update-build_backend' into refactor/data-model

158c8d3

Merge pull request #417 from softwarepub/refresh-based-on-develop

73df3ea

Pull in relevant changes from `develop`

Show skipped tests when running

6bcdbe9

Merge pull request #415 from softwarepub/refactor/411-disable-tests

80cade3

Disable tests that fail due to data model

Merge pull request #403 from softwarepub/refactor/383-test-ld_context

7b10cdc

Resolve #383: Test `ld_context`

Merge remote-tracking branch 'origin/refactor/data-model' into refact…

0edc582

…or/381-test-ld_container # Conflicts: # poetry.lock

Update poetry.lock for added test dependencies.

68aec7f

notactuallyfinn and others added 25 commits March 20, 2026 13:25

applied simple suggestions

7de8c7d

fixed misspelled class name

ad63e5a

implemented suggestions and fixed bug

7a8e8ae

added test_case and generate strategies automatically

9d3dc7f

finished implementation of CodemetaProcessPlugin

c699ae2

implement more comments

d514c9f

add codemeta_doi postprocess plugin

ba8b549

reworked zenodo_sandbox_auth for cli testing

9b1c48a

fix post process and add license header to conftest.py

55e086d

improve error handling of commands

9d770a1

make process more verbose for errors while merging

180cc10

potentially fixed error where multiple record ids are halucinated.

0251cd1

flake8

73467f5

adjusted logging a bit

34877b9

fix tests that are affected by error handling update

a0c0005

added another process test and fixed small bug

279e672

flake8

65399be

Imporve first cli harvest tests

89191ec

add reject strategies for special properties

a42c315

Add good case tests for cli

9294d0a

Merge pull request #456 from softwarepub/feature/454-e2e-test-plugin-api

e40965d

Fix #454: Add end-to-end tests for the plugin-related `SoftwareMetadata` API

Ignore flake8 C901 (too complex)

5108696

Merge pull request #479 from softwarepub/refactor/423-implement-publi…

224d5b7

…c-api Refactor/423 implement public api

update dependencies

3e7c83d

add/ update comments and doc strings

23a11b3

zyzzyxdonta reviewed May 26, 2026

View reviewed changes

zyzzyxdonta reviewed Jun 3, 2026

View reviewed changes

zyzzyxdonta reviewed Jun 4, 2026

View reviewed changes

zyzzyxdonta reviewed Jun 9, 2026

View reviewed changes

	This exception should be thrown when input validation (e.g., during harvest) occurs.
	This exception should be raised when an error occurs during input validation (e.g., during harvest).

	This exception should be thrown when interacting with the model context.
	This exception should be raised when interacting with the model context.

	For further information on this function and the errors it throws see :meth:`ld_list.__init__`.
	For further information on this function and the exceptions it raises see :meth:`ld_list.__init__`.

	Output of the different `hermes` commands consequently is valid JSON-LD, serialized as JSON, that is cached in
	Output of the different `hermes` subcommands consequently are valid JSON-LD files that are cached in

	The cache is purely for internal purposes, its data should not be interacted with.
	The cache should only be interacted with via the `hermes` libraries.


		The following sections show how this class works.

		##### Creating a data model instance


		## See Also

		- API reference: {class}`hermes.model.SoftwareMetadata`

	The hermes-plugin-git will help to gather contributing and branch metadata.
	The hermes-plugin-git will help to gather contribution and branch metadata.

	For this tutorial you should be familiar with HERMES.
	To follow this tutorial you should be familiar with HERMES.

	And uses the [schmea.org](https://schema.org/) (with prefix "schema") and the [CodeMeta](https://codemeta.github.io/) (without prefix) context.
	And uses the [schema.org](https://schema.org/) (with prefix "schema") and the [CodeMeta](https://codemeta.github.io/) (without prefix) context.


		## Configure HERMES to use your plugin

		To integrate your plugin, you have to register it as a plugin in the `pyproject.toml`.

Conversation

SKernchen commented Apr 29, 2026

Uh oh!

zyzzyxdonta left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zyzzyxdonta Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zyzzyxdonta Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

zyzzyxdonta Jun 3, 2026 •

edited

Loading

zyzzyxdonta Jun 4, 2026 •

edited

Loading