Repository navigation
fix(automl): enable limit_resource on LRL1Classifier (#1612) - #1634
Abhinav Maurya (Fire162) wants to merge 9 commits into
Conversation
LRL1Classifier omitted limit_resource = True, causing budgeted training in AutoML.fit to bypass resource and timeout limits. This adds limit_resource = True matching LRL2Classifier and unit tests verifying that resource limits are applied during fit.
There was a problem hiding this comment.
🟡 Changes recommended
The new invocation test fails on Windows and needs to skip platforms without resource limiting.
1 open issue
What changed in this PR
Enables existing resource limiting for L1 logistic regression, addressing budget overruns reported in #1612.
Changes:
- Sets
LRL1Classifier.limit_resource = True, matching L2. - Adds tests for the flag, budgeted fitting, and limiter invocation.
| File | Description |
|---|---|
test/automl/test_model.py |
Adds L1 resource-limiting regression tests. |
flaml/automl/model.py |
Enables resource limiting for LRL1Classifier. |
Review effort: Balanced
|
|
||
|
|
||
| def test_lrl1_resource_limit_called(monkeypatch): | ||
| import flaml.automl.model as model_mod |
Li Jiang (thinkall)
left a comment
Li Jiang (thinkall)
left a comment
There was a problem hiding this comment.
Overall review of the complete PR: changes requested for a deterministic Windows test failure.
The LRL1 resource-limit flag matches LRL2 and reuses the existing production platform guard. The added invocation assertion does not respect that guard. Running the two new tests on Windows produced 1 passed and 1 failed: test_lrl1_resource_limit_called asserts a call even though resource is unavailable and BaseEstimator.fit intentionally bypasses the limiter. Please make that test capability-aware; do not remove the production guard. The review-code harness and current Balanced Copilot review agree on this blocker.
Posted by thinkall-agent-auto-reviewer
| X, y = make_classification(50, 5, random_state=0) | ||
| lr = LRL1Classifier() | ||
| lr.fit(X, y, budget=1.0) | ||
| assert len(called) > 0, "limit_resource was not invoked for LRL1Classifier" |
There was a problem hiding this comment.
This assertion fails on supported Windows platforms. model_mod.resource is None there, so BaseEstimator.fit deliberately does not enter limit_resource, regardless of the new LRL1 flag. I reproduced the failure with python -m pytest test/automl/test_model.py::test_lrl1 test/automl/test_model.py::test_lrl1_resource_limit_called -q (1 passed, 1 failed). Skip the invocation-specific case when resource is unavailable, or assert the appropriate call/no-call behavior for each capability while retaining the cross-platform fit smoke test.
Posted by thinkall-agent-auto-reviewer
Li Jiang (thinkall)
left a comment
Li Jiang (thinkall)
left a comment
There was a problem hiding this comment.
Re-reviewed commits 4d9553cefbb14233782ccfd928264bfde5ca739c -> 6d5a3a774153cf7836d651b90b73ad48e33694bb. The only delta is main's website/yarn.lock update of follow-redirects 1.16.0 -> 1.16.1. One explicitly head-bound review-code Low execution (actual runtime gpt-6.1-sol) found no new issue in this identical inherited patch. Controller Git comparisons verify all non-lockfile tree entries are unchanged. This is a scoped merge-delta review, not a fresh clean signoff for unchanged feature code.
Version/URL/caret/SRI shape checks pass. No independent local registry/tarball verification, package smoke test or website rebuild is claimed. Previous feature findings and runtime validation limits remain preserved.
Changes remain required. The existing Windows test blocker remains: test_lrl1_resource_limit_called still requires a limiter call when resource is unavailable and production deliberately bypasses that limiter. The test and production guard are unchanged by this merge. The prior Windows result was one passed and one failed; this cycle does not claim to rerun that native test. The original changes-requested review remains applicable.
At the 2026-10-26T12:13:44.943946+00:00 gate snapshot, actual current-head workflows were: Build in_progress/no conclusion; Code formatting completed/success; docs completed/success. Copilot has Balanced review without an Approval recommended verdict; 2 review thread(s) remain unresolved. Gate approval is not successful CI completion, and no new PR approval is being submitted while required gates remain unmet.
Posted by thinkall-agent-auto-reviewer
| def test_lrl1(): | ||
| BaseEstimator.search_space(1, "") | ||
| assert LRL1Classifier.limit_resource is True | ||
| X, y = make_classification(100, 10, random_state=0) | ||
| lr = LRL1Classifier() | ||
| lr.predict(X) | ||
| lr.fit(X, y, budget=1e-5) |
Li Jiang (thinkall)
left a comment
Li Jiang (thinkall)
left a comment
There was a problem hiding this comment.
Reviewed new commits from 6d5a3a774153cf7836d651b90b73ad48e33694bb to c831ee525ff1c995a893577a47ced80c164d0e7d. The main-branch advance is inherited across this update; the driver proves the complete resulting tree is the previously reviewed feature tree plus the same upstream patch, with no changed contributor feature blobs or function ASTs outside that integration.
One shared review-code Medium package used six independent isolated grok-4.7/xhigh lenses. All six original outcomes are issues_found and all canonical responses are committed on exact owning PR 1642 source aaa3b672462534d3e28dfbd3e88ce2e1c64a21ef, with explicit tree/content bindings to the updated heads. This is not six new whole-feature clearances or a blanket endorsement of unchanged feature code.
P2 inherited-upstream follow-up: real seasonal-naive HPO with 60,005 minute rows, period=12001, sample=True, min_sample_size=10000, max_iter=3, time_budget=15 chooses season 12,001 for a 10,000-row first trial and aborts with ValueError on current main and the owning updated head. Identical real-search settings succeed on exact prior main on both native stacks. Bound the season against the minimum actual sampled training window or avoid undersized sampling for that configuration. This issue is already in current main and inherited, not a new contributor-specific feature edit; it is preserved as an upstream follow-up rather than duplicated as an unrelated inline request. The shared review therefore does not grant new full-source approval. max_iter=1 bypasses search and refits full data, so those initial successful diagnostics were not used as HPO clearance.
Selected integration validation passed 592 executions of 296 distinct cases on native Windows Python 3.10/pandas 2.3.3/NumPy 1.26.4 and Python 3.13/pandas 3.0.5/NumPy 2.3.3. That includes actual classifier and naive/seasonal-average fits plus controlled accessor/facade tests. Additional native controls reject unproven zero-lag/season-1/NaT-lookup/sparse-exog allegations; actual SARIMAX sparse prediction selects the requested rows. No native Spark/TFT training or complete optional/minimum-version matrix is claimed. Passing selected tests does not override the independently reproduced P2.
Changes remain required for the existing feature finding: the limiter-invocation assertion is still unconditional on Windows where production intentionally bypasses the limiter because resource is unavailable. Both model and test blobs are byte-identical to the prior reviewed head. No duplicate inline is being posted.
At 2026-10-31T20:28:04.854853+00:00, actual current-head pull_request workflows were: Build in_progress/None (attempt 2); Code formatting completed/success (attempt 2); docs completed/success (attempt 2). 3 review thread(s) remain unresolved. No explicit current-head Balanced Copilot approval recommendation is claimed. No new PR approval is submitted while source or external gates remain unmet.
Posted by thinkall-agent-auto-reviewer
Li Jiang (thinkall)
left a comment
Li Jiang (thinkall)
left a comment
There was a problem hiding this comment.
Review cycle 20261102-2221; exact source 4194ac5d433118503a4e0fafc0913fd1c9620082.
Reviewed the new inherited Joi-only merge delta with review-code (Low, actual gpt-6.1-sol; local evidence commit e938ad78f120a83bc8825d27b8c114205b089ab9). Complete-tree reconstruction confirms the contributor's feature blobs are unchanged. This clean lock-delta result is not a new whole-feature clean vote.
Changes still required: the limiter-invocation test remains unconditional on Windows, where production bypasses the limiter because resource is unavailable. The lock update leaves that verified test/production capability mismatch unchanged. Existing inline discussions are retained rather than duplicated.
The previously reproduced inherited-main seasonal-naive HPO P2 remains: 60,005 minute observations, period 12,001, sampling/minimum sample 10,000, max_iter=3 and time_budget=15 select an invalid season for the first sampled trial and abort AutoML.fit; prior-main controls pass on both native stacks. The Joi-only main advance changes none of that Python code. No unchanged Python/native/Spark/TFT suite was rerun for this lock-only update.
Fresh gates: 3 unresolved review thread(s); no explicit current-head Balanced approval recommendation; 11 non-success applicable check context(s). Actual workflow holds: Build:in_progress/None. Workflow permission approval is not validation success. No PR approval or foreign-thread resolution was performed.
Posted by thinkall-agent-auto-reviewer
There was a problem hiding this comment.
🟡 Changes recommended
The invocation test always fails in Windows CI when resource limiting is unavailable, and the budget tests do not verify budget forwarding, timeout enforcement, or a usable fallback model.
3 open issues
Review effort: Balanced
|
|
||
|
|
||
| def test_lrl1_resource_limit_called(monkeypatch): | ||
| import flaml.automl.model as model_mod |
Li Jiang (thinkall)
left a comment
Li Jiang (thinkall)
left a comment
There was a problem hiding this comment.
Review cycle 20261106-1600; exact source 34ced39dc2eb8024a123657adb9d36a8d19d9172.
The branch was updated with a fresh permission/conflict/head guard. Complete-tree reconstruction matches the exact previous feature tree merged with current main 46bce0ce989aae0271e68e5daac5b8a9b0522b87; no unrelated merge edits were found.
Independent review is incomplete. The shared merge delta is planned Low under review-code, but its authenticated CLI prerequisite remains unresolved. No reviewer model ran and no new clean skill artifact was produced. Earlier feature findings and scope qualifications are retained, not converted into a new whole-feature clean vote.
Bounded shared-main validation: the Spark-backend/R2 tests passed 15 cases on each native Windows Python 3.10 and 3.13 stack. The same external test files against the immediate previous main produced six expected failures and nine passing controls per stack. Spark tests use a controlled backend, not an actual Spark cluster; these checks do not cover the whole feature or optional workload matrix.
Existing inherited-main HPO hold remains. Real AutoML.fit search with 60,005 minute observations, period=12001, sample=True, min_sample_size=10000, max_iter=3 and time_budget=15 still selects season 12,001 for the first 10,000-row trial and aborts with SeasonalNaive needs at least season=12001 training observations, got 10000. This was reproduced on both native stacks for new main and its immediate predecessor; the newly merged changes did not repair the older issue. Historical pre-introduction passing controls remain separate. max_iter=1 is not search/sampling proof.
Changes still required: test\automl\test_model.py::test_lrl1_resource_limit_called is unconditional on Windows although production bypasses the unavailable resource limiter. The exact refreshed head was tested on both native Windows stacks: the test still fails with limit_resource was not invoked for LRL1Classifier. Both contributor feature blobs are identical to the previously inspected head. Existing inline threads are not duplicated or resolved.
Fresh gates: 4 unresolved thread(s); explicit current-head Balanced approval recommendation: no; 11 non-success applicable context(s). Actual current-head workflow holds: Build:in_progress/None. Workflow permission approval is not validation success. No PR approval, foreign-thread mutation or review dismissal was performed.
Posted by thinkall-agent-auto-reviewer
Why are these changes needed?
In
flaml/automl/model.py,SKLearnEstimator.fit()checksgetattr(self, "limit_resource", None)before wrapping execution in thelimit_resourcecontext manager. WhileLRL2Classifierdeclareslimit_resource = True,LRL1Classifieromitted this attribute.Because
lrl1is included in the default classification estimator list, trainingLRL1Classifierwith a specifiedtime_budgetbypassed resource and timeout limiting entirely, causing long-running fits to overrun budgets significantly on wide or many-class datasets.This change:
limit_resource = TruetoLRL1Classifierinflaml/automl/model.py, aligning it withLRL2Classifier.test/automl/test_model.pyverifying thatLRL1Classifier.limit_resourceisTrue, fit respects budget constraints, andlimit_resourceis invoked during execution.Related issue number
Closes #1612
Checks