Skip to content

fix(dfm-search): support Chinese-numeral years in semantic time parsing - #373

Merged
deepin-bot[bot] merged 1 commit into
linuxdeepin:masterfrom
Johnson-zs:agent/bugfix/35e3d606
Jul 31, 2026
Merged

fix(dfm-search): support Chinese-numeral years in semantic time parsing#373
deepin-bot[bot] merged 1 commit into
linuxdeepin:masterfrom
Johnson-zs:agent/bugfix/35e3d606

Conversation

@Johnson-zs

@Johnson-zs Johnson-zs commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

修复内容

修复文管智能语义搜索在解析"中文数字年份 + 月份 + 文件类型"叠加查询时结果为空的问题。

根因

dfm-search 语义搜索时间规则的两处相互依赖缺陷:

  1. time_rules.json 中三处年份捕获组 (?<year>\d{2,4}) 只匹配阿拉伯数字,不匹配"二五年"/"二零二五年"等中文数字年份。
  2. timeextractor.cpp::localeAwareToInt 仅处理阿拉伯数字、单个中文数字、含"十"的中文数字,无法解析"二五"(25)、"二零二五"(2025) 等逐位中文数字串。

导致"二五年十二月"中年份无法被时间规则消费,"二五年"泄露为搜索关键词,月份被当作当前年(2026)12月,最终查询条件错配 → 结果为空。

触发机理

输入"二五年十二月创建的表格"时:

  1. time_exact_year_month/time_exact_year 年份正则 \d{2,4} 拒绝"二五" → 不命中
  2. 退而命中 time_exact_month_current_year(优先级 120),把"十二月"当作当前年 12 月
  3. "二五年"未被任何时间规则消费 → 泄露为搜索关键词 keywords:["二五年"]
  4. 最终查询 = (文件名含"二五年") AND (创建时间在 2026-12) AND (表格后缀) → 测试文件名不含"二五年"、创建于 2025-12 → 结果为空

改动

  • time_rules.jsontime_exact_year/time_exact_year_month/time_exact_full_date 年份组扩展为 \d{2,4}|[零〇一二三四五六七八九]{2,4},digit_map 补 "〇": 0。字符集刻意不含"十"(避免"十二月"被误当年份)、不含"两"。
  • timeextractor.cpplocaleAwareToInt 新增逐位中文数字分支(在"十"相关分支之后),将全由 0-9 中文数字组成的串按位拼接。不影响"十一"/"二十"/"十二"等既有解析。
  • tst_chinese_nlp.cpp:新增 timeCustom_yearMonth_chineseYeartimeCustom_yearMonth_chineseYear_noResidualKeyword 两个回归测试。

验证(脚本复现,与开发人员 dfm-searcher -j 实测一致)

  • 25年十二月创建的表格 → 2025-12 ✅(原已正确)
  • 二五年十二月创建的表格 → 2025-12,无残留 keyword ✅(修复后)
  • 二五年12月创建的表格 → 2025-12 ✅(修复后)
  • 二零二五年十二月创建的表格 → 2025-12 ✅(修复后)
  • 二〇二五年十二月创建的表格 → 2025-12 ✅(修复后)
  • 十二月的文件 → 仍走 month_current_year,无回归 ✅

PMS: https://pms.uniontech.com/bug-view-372165.html

Summary by Sourcery

Support Chinese-numeral years in dfm-search semantic time parsing and ensure they are correctly consumed by year/year-month rules without leaking into keywords.

Bug Fixes:

  • Fix semantic time parsing so Chinese-numeral years (e.g. "二五年"/"二零二五年"/"二〇二五年") are interpreted as the intended year instead of falling back to current-year month and leaving residual keywords.

Enhancements:

  • Extend locale-aware numeric parsing to handle digit-by-digit Chinese numeral strings for years without regressing existing "十"-based date interpretations.

Tests:

  • Add regression tests covering Chinese-numeral year plus month queries, bare Chinese-numeral years, and full stacked queries to verify correct time constraints, filetype extraction, and absence of residual keywords.

The year capture group in time rules only matched Arabic numerals
(\d{2,4}), so queries like "二五年十二月创建的表格" failed to recognise
the year. The month fell back to the current-year rule and the
unconsumed "二五年" leaked into keywords, producing empty results.

- time_rules.json: extend year group to \d{2,4}|[零〇一二三四五六七八九]{2,4}
  for time_exact_year, time_exact_year_month, time_exact_full_date;
  add "〇":0 to their digit_map.
- timeextractor.cpp: add digit-by-digit Chinese numeral parsing in
  localeAwareToInt (e.g. "二五"→25, "二零二五"→2025).
- tst_chinese_nlp.cpp: add regression tests for Chinese-numeral year
  queries and verify no residual keyword leak.

Log: 修复中文数字年份叠加搜索结果为空的问题
PMS: BUG-372165
Influence: 语义搜索现在能正确识别"二五年"/"二零二五年"等中文数字年份,不再将其泄露为关键词或误绑到当前年份。

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @Johnson-zs, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@deepin-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Johnson-zs

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sourcery-ai

sourcery-ai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Reviewer's Guide

This PR fixes dfm-search semantic time parsing so that Chinese-numeral years (e.g. “二五年”, “二零二五年”, “二〇二五年”) are correctly recognised by year/year-month/full-date rules, parsed into proper year values, and fully consumed without leaking into residual keywords, while preserving existing month-only and “十”-style numeral behaviour.

Sequence diagram for Chinese-numeral year parsing in dfm-search semantic time extraction

sequenceDiagram
    actor User
    participant SemanticSearch
    participant TimeRules
    participant TimeExtractor
    participant KeywordExtractor

    User->>SemanticSearch: search("二五年十二月创建的表格")
    SemanticSearch->>TimeRules: apply time_exact_year_month
    TimeRules-->>SemanticSearch: match year="二五" month="十二"
    SemanticSearch->>TimeExtractor: localeAwareToInt("二五", digitMap)
    TimeExtractor-->>SemanticSearch: 2025
    SemanticSearch->>TimeExtractor: localeAwareToInt("十二", digitMap)
    TimeExtractor-->>SemanticSearch: 12
    SemanticSearch-->>KeywordExtractor: remaining_text("创建的表格")
    KeywordExtractor-->>SemanticSearch: keywords(["创建","表格"])
    SemanticSearch-->>User: results(filtered_by_2025_12)
Loading

File-Level Changes

Change Details Files
Add digit-by-digit Chinese numeral parsing in the time extractor so multi-character Chinese years map to integer years.
  • Introduce an allSingleDigits check in localeAwareToInt to detect inputs composed entirely of mapped 0–9 Chinese digits.
  • Guard the new branch to only apply to strings of length >= 2 and exclude characters that are unmapped or map to values > 9 (e.g. “十”, “两”).
  • Compute the numeric value by iterating through the Chinese digits and accumulating value = value * 10 + digitMap.value(ch), returning the constructed integer when the branch applies.
src/dfm-search/dfm-search-lib/semantic/extractors/timeextractor.cpp
Extend Chinese time rules to capture Chinese-numeral years and support the new digit-by-digit parsing.
  • Update year capture groups in year-related time rules (exact year, year-month, full date) from matching only \d{2,4} to also match 2–4 Chinese numeral characters for digits 0–9, excluding “十” and “两” to avoid month mis-parsing.
  • Ensure the rules recognise Chinese-written years such as “二五年”, “二零二五年”, and “二〇二五年” while not reinterpreting standalone month expressions like “十二月” as year-month.
  • Extend the digit_map to include the mapping for the Chinese zero character “〇” to 0 so it can be used in digit-by-digit year parsing.
src/dfm-search/dfm-search-lib/semantic/rules/zh_CN/time_rules.json
Add regression tests for Chinese-numeral year parsing and the original bug scenario to prevent leaks into keywords and ensure correct time constraints and file type extraction.
  • Declare two new test slots timeCustom_yearMonth_chineseYear and timeCustom_yearMonth_chineseYear_noResidualKeyword in the test class.
  • Add test cases that verify parsing of 2-digit and 4-digit Chinese-numeral years combined with Chinese and Arabic months, checking start/end year and month and coverage of the full year or month as appropriate.
  • Add a bug scenario test for the full query “二五年十二月创建的表格” asserting correct 2025-12 time constraint, spreadsheet file extensions in the result, absence of residual keywords, and that the consumed span comes from the time_exact_year_month rule.
  • Include a regression check that month-only phrases like “十二月的文件” still resolve to the current year’s month and are not mis-parsed as year-month after the rule changes.
autotests/dfm-search-tests/tst_chinese_nlp.cpp

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@deepin-ci-robot

Copy link
Copy Markdown

deepin pr auto review

★ 总体评分:100分

■ 【总体评价】

代码实现了中文数字年份的解析功能,逻辑严谨且测试覆盖全面
修复了中文数字年份无法识别的缺陷,各维度均无问题,故给予满分

■ 【详细分析】

  • 1.语法逻辑(基本正确)✓

timeextractor.cpplocaleAwareToInt 函数中新增了逐位中文数字解析逻辑,通过遍历字符串并检查 digitMap 中的映射值是否大于 9,正确排除了"十"和"两"等非逐位数字,确保"二五"转换为 25,"二零二五"转换为 2025。time_rules.json 中的正则表达式正确引入了 [零〇一二三四五六七八九]{2,4},并补充了"〇"的映射,与 C++ 逻辑配合完美。
潜在问题:无
建议:无

  • 2.代码质量(良好)✓

代码注释清晰说明了处理逐位中文数字的背景和排除特定字符的原因。测试用例覆盖了多种边界情况,包括 2 位和 4 位中文年份、包含"零"和"〇"的情况,以及回归测试确保"十二月"不会被误解析为年份。
潜在问题:无
建议:无

  • 3.代码性能(高效)✓

localeAwareToInt 函数中新增的逻辑仅包含一次 O(n) 的遍历和简单的数学运算,对于年份这种极短的字符串,性能开销可以忽略不计。
潜在问题:无
建议:无

  • 4.代码安全(存在0个安全漏洞)✓

漏洞对比统计:新增漏洞 0 个,减少漏洞 0 个,持平 0 个
本次修改仅涉及本地字符串解析和正则匹配,不涉及外部命令执行、网络请求或敏感数据操作,不存在安全风险。

  • 建议:无

■ 【改进建议代码示例】

// 代码已经非常完善,无需额外修复

@Johnson-zs

Copy link
Copy Markdown
Contributor Author

/forcemerge

@deepin-bot

deepin-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

This pr force merged! (status: blocked)

@deepin-bot
deepin-bot Bot merged commit e782e11 into linuxdeepin:master Jul 31, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants