Repository navigation
Ignore bare numbers when detecting the language in search_dates - #1466
AdrianAtZyte wants to merge 5 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1466 +/- ##
=======================================
Coverage 99.19% 99.19%
=======================================
Files 239 239
Lines 4098 4102 +4
=======================================
+ Hits 4065 4069 +4
Misses 33 33 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
serhii73
left a comment
There was a problem hiding this comment.
Thanks! Besides #921, this fixes a big class of detection bugs: on master, languages=None treats "10", "11", "12"… as Vietnamese words, so things like "January 12, 2012 10:00 PM" or "Published: March 12, 2020 at 10:30 AM" come back as fragments, and with this PR they parse correctly.
One regression though: those digit tokens were also what made the usual Vietnamese format win detection. After the vi simplifications drop "ngày" and "năm", only "tháng" is left, so vi now ties with ja/zh (which match the "am" in "nam") and loses:
from datetime import datetime
from dateparser.search import search_dates
rb = {"RELATIVE_BASE": datetime(2020, 6, 15, 12)}
search_dates("ngày 11 tháng 12 năm 2014", settings=rb, add_detected_language=True)
# master: [('ngày 11 tháng 12 năm 2014', datetime(2014, 12, 11, 0, 0), 'vi')]
# PR: [('11 t', datetime(2020, 11, 15, 0, 0), 'ja'), ('12', datetime(2020, 12, 15, 0, 0), 'ja'), ('2014', datetime(2014, 6, 15, 0, 0), 'ja')]
search_dates("21 tháng 09 năm 2025", settings=rb)
# master: [('21 tháng 09 năm 2025', datetime(2025, 9, 21, 0, 0))] — PR: fragmentsCould the digit hits be kept as a last tiebreaker rather than dropped? E.g. return [dict_cnt, skip_cnt, digit_cnt] from _count_words_present_in_the_dictionary (counting bare numbers that the dictionary maps to something in digit_cnt) and use key=lambda p: (p[1][0], p[1][1], p[1][2]) in _best_language. In my tests that keeps both #921 examples and the English improvements, and brings the Vietnamese dates back. A test with "ngày 11 tháng 12 năm 2014" and languages=None would guard it.
Fixes #575, resolves #921