Allow indexer bolts to use a metadata value as the document ID#2012
Conversation
jnioche
left a comment
jnioche
left a comment
There was a problem hiding this comment.
Looks good, forgot to mention that we should add the new indexer.md.docid to core/src/main/resources/crawler-default.yaml
with a default value of "".
rzo1
left a comment
rzo1
left a comment
There was a problem hiding this comment.
Changes look good to me, but we should update the docs alongside this PR too, so people get an updated doc version ;-)
There was a problem hiding this comment.
Implementation looks correct and backward compatible. One design question for the maintainers: with content-based dedup, N URLs share one document. When a single URL goes away, the DeletionBolt deletes the document even if the content is still live under the other URLs, until one gets re-fetched. Acceptable as a documented limitation, or should deletion be skipped when the ID comes from metadata?
Inline comments cover a metadata.persist prerequisite and backend coverage of the new setting.
|
|
||
| | indexer.canonical.name | canonical | Metadata key for the canonical URL. Used to replace the URL with its canonical form before indexing. | ||
| | indexer.ignore.empty.fields | false | If true, skip fields with empty values when indexing. | ||
| | indexer.md.docid | - | Metadata key whose value, if present, is used as the document ID instead of the default SHA-256 digest of the URL. Useful e.g. for content-based deduplication when a `ParseFilter` stores a content hash in the metadata. |
There was a problem hiding this comment.
Two precisions for this row: the value is digested, not used raw (SHA-256 here, SHA-512 via CloudSearchUtils.getID for CloudSearch); and the SOLR/SQL indexers never call getDocumentID(), so the setting is silently ignored there. Worth stating the supported backends.
| * at. Falls back to the default (a SHA-256 digest of the URL) if not set or if the metadata | ||
| * does not contain a value for the configured key. | ||
| */ | ||
| public static final String DOC_ID_METADATA_PARAM_NAME = "indexer.md.docid"; |
There was a problem hiding this comment.
Worth documenting: on the deletion path (gone pages) there's no parsing, so the DeletionBolt only sees this key if it's in metadata.persist and round-trips through the status index. Otherwise it falls back to sha256(url) and the document indexed under the metadata ID is silently orphaned.
4 participants
The change should cover the discussion in the #671 issue.