Skip to content

Ignore bare numbers when detecting the language in search_dates - #1466

Open
AdrianAtZyte wants to merge 5 commits into
scrapinghub:masterfrom
AdrianAtZyte:search-bare-number-detection
Open

AdrianAtZyte wants to merge 5 commits into
scrapinghub:masterfrom
AdrianAtZyte:search-bare-number-detection

Conversation

@AdrianAtZyte

@AdrianAtZyte AdrianAtZyte commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Fixes #575, resolves #921

@codecov

codecov Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.19%. Comparing base (b1a87f9) to head (af3b818).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1466   +/-   ##
=======================================
  Coverage   99.19%   99.19%           
=======================================
  Files         239      239           
  Lines        4098     4102    +4     
=======================================
+ Hits         4065     4069    +4     
  Misses         33       33           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed

codspeed Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 10 untouched benchmarks


Comparing AdrianAtZyte:search-bare-number-detection (af3b818) with master (b1a87f9)

Open in CodSpeed

@AdrianAtZyte
AdrianAtZyte marked this pull request as ready for review September 28, 2026 11:28

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Besides #921, this fixes a big class of detection bugs: on master, languages=None treats "10", "11", "12"… as Vietnamese words, so things like "January 12, 2012 10:00 PM" or "Published: March 12, 2020 at 10:30 AM" come back as fragments, and with this PR they parse correctly.

One regression though: those digit tokens were also what made the usual Vietnamese format win detection. After the vi simplifications drop "ngày" and "năm", only "tháng" is left, so vi now ties with ja/zh (which match the "am" in "nam") and loses:

from datetime import datetime
from dateparser.search import search_dates
rb = {"RELATIVE_BASE": datetime(2020, 6, 15, 12)}
search_dates("ngày 11 tháng 12 năm 2014", settings=rb, add_detected_language=True)
# master: [('ngày 11 tháng 12 năm 2014', datetime(2014, 12, 11, 0, 0), 'vi')]
# PR:     [('11 t', datetime(2020, 11, 15, 0, 0), 'ja'), ('12', datetime(2020, 12, 15, 0, 0), 'ja'), ('2014', datetime(2014, 6, 15, 0, 0), 'ja')]
search_dates("21 tháng 09 năm 2025", settings=rb)
# master: [('21 tháng 09 năm 2025', datetime(2025, 9, 21, 0, 0))] — PR: fragments

Could the digit hits be kept as a last tiebreaker rather than dropped? E.g. return [dict_cnt, skip_cnt, digit_cnt] from _count_words_present_in_the_dictionary (counting bare numbers that the dictionary maps to something in digit_cnt) and use key=lambda p: (p[1][0], p[1][1], p[1][2]) in _best_language. In my tests that keeps both #921 examples and the English improvements, and brings the Vietnamese dates back. A test with "ngày 11 tháng 12 năm 2014" and languages=None would guard it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants