Skip to content

perf(query): paginate table sidebar and cap pages at 50k - #673

Merged
sunchanglong merged 22 commits into
mainfrom
optimizequery
Nov 9, 2026
Merged

sunchanglong merged 22 commits into
mainfrom
optimizequery

Conversation

@sunchanglong

@sunchanglong sunchanglong commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator
  • load one 50k row page per request instead of looping every information_schema page
  • fetch count and page in parallel; keep the old page visible while loading
  • search table names server-side (LIKE) and page through filtered results
  • store the tree in a shallowRef and replace it on change (~45% less heap at 50k)
  • use DESC for column metadata (~3ms vs ~0.7s via information_schema.columns)
  • get table names with SHOW TABLES instead of materialising the full tree
  • stop rebuilding CodeMirror completions on every table-tree change
  • Added a shared TableSelect with a 500-row window, remote prefix search, and virtual rendering.
  • Migrated Logs/Traces Query table pickers off full-table loading.
  • Migrated Drilldown Logs/Traces overview, Logs Settings, Ingest, and Flow sink tables to the shared picker.
  • Restricted the Traces table picker to tables containing trace_id.
  • Prefiltered Trace Logs candidates with a cached tablesHavingColumn('trace_id') query.
  • Removed full-table enumeration from Logs auto-binding.
  • Virtualized the Metrics sidebar and capped each page at 500 metrics.
  • Replaced the approximate metric count with an exact metric-engine table count.
  • Added search-aware Metrics pagination with range display, prev/next, and page-number jump.
  • Added getTablesByPrefix, getMetricNamesPage, and exact getMetricNamesCount APIs.
  • Aligned Metrics and table-picker virtual-list height handling with the table sidebar.

- load one 50k row page per request instead of looping every information_schema page
- fetch count and page in parallel; keep the old page visible while loading
- search table names server-side (LIKE) and page through filtered results
- store the tree in a shallowRef and replace it on change (~45% less heap at 50k)
- use DESC for column metadata (~3ms vs ~0.7s via information_schema.columns)
- get table names with SHOW TABLES instead of materialising the full tree
- stop rebuilding CodeMirror completions on every table-tree change
The virtual list height used 100vh minus hard-coded header/footer
variables, which did not match the real sidebar layout and produced
an extra scrollbar. Measure the tree container with useElementSize
and pass the actual height to Arco's virtual list.
The global .table-tree rule adds 8px bottom padding, so the virtual list
height measured from the container overflowed the tree by 8px. Move the
spacing to the container and zero the tree's bottom padding.
Tree nodes live in a shallowRef, so the computed SQL kept the empty
timeIndexName captured before columns were loaded and emitted ORDER BY "".
Build the SQL lazily at click time, load columns first when needed, and
fall back to no ORDER BY when no time index is known.
Page-local keys are reused when the user pages. Guard column/detail loads
with the expected table name, ignore out-of-order page/search responses,
and verify the ALTER target still exists after awaiting column metadata.
Logs Query feeds every table name into the selector. With 100k tables Arco
rendered 100k option nodes and froze the page when the dropdown opened.
Enable the select's virtual list (36px rows) so only visible options render.
Logs/Traces builders loaded every table name into the select: +85 MB heap
at 100k tables (~850 MB at 1M). Start with declared signal tables plus the
first 200 tables, search the server by prefix (range scan, ~1s at 100k)
with a substring LIKE fallback, and remember the selected table's database.
The trace-logs picker and association pool enumerated every table and then
qualified each one. Qualification requires trace_id, so start from the cached
tablesHavingColumn('trace_id') set (one GROUP BY query), declared log tables,
and the current table.
Add a shared TableSelect/useTableOptions (500-row window, remote prefix
search, virtual list) and migrate the logs/traces drilldown pickers, logs
settings, ingest, and flow. Trace-logs discovery now asks for traceOnly and
logs auto-bind no longer enumerates every table.
The shared picker used the generic first-500 table window for every signal,
so Traces listed tables without trace_id. For traces, derive the allowed set
from listSignalTables('traces') and filter both the initial window and search
results; logs keep the unrestricted list.
Load and search at most 500 metric names and render the tree through Arco's
virtual list, so large metric catalogs no longer mount thousands of nodes.
Label/value expansion stays lazy and the count label still reports the cap.
The 500-row window made the sidebar label approximate. Count metric-engine
tables with one catalog query (the Prometheus name universe) so the label is
exact while the tree and search stay bounded.
Measure the tree container content box (excluding its 8px bottom padding),
re-measure on keep-alive activation and after data changes, and pass the table
sidebar's threshold/buffer virtual-list props so the list fills the remaining
card height without overflowing.
Metric names now come from catalog pages (500 + offset) with an exact
filtered count, a pager row (range, page input, total), and search-aware
pagination. Search keeps the same 500-row pages instead of a hard cap.
checkTables only marked a status with no readers and preloaded the first
50k-row table page. The Table Query sidebar, ingest, flow, and drilldown own
their lazy loading, so remove the call and the now-unused store action.
Removing the connection-level checkTables preload exposed that the sidebar
only loaded when activeDatabase was empty. Load the first page whenever the
active database has no tree yet, so the sidebar owns its initial fetch.
Fetch metric names with limit=0 so the catalog is no longer truncated at 40k,
and keep large pools cheap: shallow storage for the name array, no-copy paths
when no search/filter is active, and no per-use string array copy in the
prefix tree. Cards remain batch-rendered.
Table Query paged at 50k and Metrics Query at 500. Use one shared
TABLE_PAGE_SIZE of 10k: a tree rebuild stays around 100ms, memory stays
bounded, and 100k rows/names fit in 10 pages.
Metric-engine logical tables share greptime_physical_table, which is an
internal table rather than a user table. Exclude it from table pages,
counts, prefix search, and name caches so Table Query matches Metrics Query.
Add client-side ordering only: exact table names move to the top of search
results, and the last ten tables selected per database are remembered in
localStorage and prepended on load. No new settings or UI are required.
@sunchanglong
sunchanglong merged commit 3407a0c into main Nov 9, 2026
1 check passed
@sunchanglong
sunchanglong deleted the optimizequery branch November 9, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant