Repository navigation
perf(query): paginate table sidebar and cap pages at 50k - #673
Merged
Merged
Conversation
- load one 50k row page per request instead of looping every information_schema page - fetch count and page in parallel; keep the old page visible while loading - search table names server-side (LIKE) and page through filtered results - store the tree in a shallowRef and replace it on change (~45% less heap at 50k) - use DESC for column metadata (~3ms vs ~0.7s via information_schema.columns) - get table names with SHOW TABLES instead of materialising the full tree - stop rebuilding CodeMirror completions on every table-tree change
The virtual list height used 100vh minus hard-coded header/footer variables, which did not match the real sidebar layout and produced an extra scrollbar. Measure the tree container with useElementSize and pass the actual height to Arco's virtual list.
The global .table-tree rule adds 8px bottom padding, so the virtual list height measured from the container overflowed the tree by 8px. Move the spacing to the container and zero the tree's bottom padding.
Tree nodes live in a shallowRef, so the computed SQL kept the empty timeIndexName captured before columns were loaded and emitted ORDER BY "". Build the SQL lazily at click time, load columns first when needed, and fall back to no ORDER BY when no time index is known.
Page-local keys are reused when the user pages. Guard column/detail loads with the expected table name, ignore out-of-order page/search responses, and verify the ALTER target still exists after awaiting column metadata.
Logs Query feeds every table name into the selector. With 100k tables Arco rendered 100k option nodes and froze the page when the dropdown opened. Enable the select's virtual list (36px rows) so only visible options render.
Logs/Traces builders loaded every table name into the select: +85 MB heap at 100k tables (~850 MB at 1M). Start with declared signal tables plus the first 200 tables, search the server by prefix (range scan, ~1s at 100k) with a substring LIKE fallback, and remember the selected table's database.
The trace-logs picker and association pool enumerated every table and then
qualified each one. Qualification requires trace_id, so start from the cached
tablesHavingColumn('trace_id') set (one GROUP BY query), declared log tables,
and the current table.
Add a shared TableSelect/useTableOptions (500-row window, remote prefix search, virtual list) and migrate the logs/traces drilldown pickers, logs settings, ingest, and flow. Trace-logs discovery now asks for traceOnly and logs auto-bind no longer enumerates every table.
The shared picker used the generic first-500 table window for every signal,
so Traces listed tables without trace_id. For traces, derive the allowed set
from listSignalTables('traces') and filter both the initial window and search
results; logs keep the unrestricted list.
Load and search at most 500 metric names and render the tree through Arco's virtual list, so large metric catalogs no longer mount thousands of nodes. Label/value expansion stays lazy and the count label still reports the cap.
The 500-row window made the sidebar label approximate. Count metric-engine tables with one catalog query (the Prometheus name universe) so the label is exact while the tree and search stay bounded.
Measure the tree container content box (excluding its 8px bottom padding), re-measure on keep-alive activation and after data changes, and pass the table sidebar's threshold/buffer virtual-list props so the list fills the remaining card height without overflowing.
Metric names now come from catalog pages (500 + offset) with an exact filtered count, a pager row (range, page input, total), and search-aware pagination. Search keeps the same 500-row pages instead of a hard cap.
checkTables only marked a status with no readers and preloaded the first 50k-row table page. The Table Query sidebar, ingest, flow, and drilldown own their lazy loading, so remove the call and the now-unused store action.
Removing the connection-level checkTables preload exposed that the sidebar only loaded when activeDatabase was empty. Load the first page whenever the active database has no tree yet, so the sidebar owns its initial fetch.
Fetch metric names with limit=0 so the catalog is no longer truncated at 40k, and keep large pools cheap: shallow storage for the name array, no-copy paths when no search/filter is active, and no per-use string array copy in the prefix tree. Cards remain batch-rendered.
Table Query paged at 50k and Metrics Query at 500. Use one shared TABLE_PAGE_SIZE of 10k: a tree rebuild stays around 100ms, memory stays bounded, and 100k rows/names fit in 10 pages.
Metric-engine logical tables share greptime_physical_table, which is an internal table rather than a user table. Exclude it from table pages, counts, prefix search, and name caches so Table Query matches Metrics Query.
Add client-side ordering only: exact table names move to the top of search results, and the last ten tables selected per database are remembered in localStorage and prepended on load. No new settings or UI are required.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.