Add corpus synchronization time metric by brun0-matheus · Pull Request #5530 · google/clusterfuzz

brun0-matheus · 2026-10-06T15:58:45Z

Bug: b/568557979

Overview

Creates new metrics called FUZZER_SYNC_CORPUS_TIME and JOB_SYNC_CORPUS_TIME to keep track of the time spent doing corpus synchronization in engine fuzzing (since blackbox fuzzing does not use corpus). The metric has 4 labels, and this is the rationale I used to choose them:

fuzzer: (only used in FUZZER_SYNC_CORPUS_TIME) the fuzzer name, like in FUZZER_TOTAL_FUZZ_TIME.
job: (only used in JOB_SYNC_CORPUS_TIME) the job name, like in JOB_TOTAL_FUZZ_TIME.
platform: I use it because many other similar metrics use it as well (like TOTAL_FUZZ_TIME).
runtime: Same reason as above.
stage: The stage of corpus synchronization. Can be either 'preprocess', 'upload' or 'download'.
success: Self-explanatory.

Changes

New metrics FUZZER_SYNC_CORPUS_TIME and JOB_SYNC_CORPUS_TIME created, which are being tracked in fuzz_task.py.
Created TimeMetricTracker in metrics.monitor, which is a context manager to automatically track time. Works for any metric except Gauge.
Add the insert method in the Metric class. This method receives a float value and calls the correct method to add this value to the metric (increment_by for Counter, add for Distribution and set for Gauge). It also converts float to int if the metric only supports int.
Add SplitMetricHelper in monitor.py. This helper allows to emit FUZZER_* and JOB_* with a single function call. Currently

Tests

Unit tests TrackSyncCorpusTimeTest created in fuzz_task_test.py, which tests if the metric is being measured correctly for success and failure cases.

Unit tests TimeMetricTrackerTest created in monitor_test.py.

I deployed in the dev environment and it's working as intended. Evidence in the bug comments.

Add a metric do keep track of corpus synchronization. The metric also distinguishes between corpus download/upload and whether it was successful or not.

Add unittest to check if SYNC_CORPUS_TIME is being emmited correctly. It tests all 4 possible combinations of (`success`, `is_upload`).

decoNR · 2026-10-07T01:52:55Z

Hi, I think it would be nice to make it clear why the metric is only for engine fuzzing.

decoNR

Just to confirm, were you able to verify that these were the only two functions associated with corpus sync?

brun0-matheus · 2026-10-07T12:46:07Z

Hi, I think it would be nice to make it clear why the metric is only for engine fuzzing.

It's because blackbox fuzzing does not use corpus.

brun0-matheus · 2026-10-07T13:38:40Z

Just to confirm, were you able to verify that these were the only two functions associated with corpus sync?

There is one function missing: corpus_manager.get_fuzz_target_corpus. At first glance I ignored it because it seemed to only fill a protobuf message, but it lists some files in the network and signs some URLs. I'll track this one as well.

Creates a context manager `TimeMetricTracker` to automatically track time. It supports Counter and CumulativeDistribution metrics. `_MockMetric` and `stub_unavailable` were modified to include the metric type that is being mocked, in order for the tracker to correctly determine the metric type during testing. Also, the `is_upload` label was removed from the SYNC_CORPUS_TIME metric.

During preprocessing, there are some possibly time intensive operations regarding listing corpus files and/or signing URLs. Added code to track time spent in this region. Also added unittests to check this.

decoNR · 2026-10-07T22:38:57Z

since blackbox fuzzing does not use corpus

nit: I think it does use but it does not sync corpus.

decoNR

Please update the description to match your recent refactors.

I like to add a section of "Key changes" with the reason for each one (e.g https://depot.code.corp.goog/gosst/casper/pull/408), feel free to do the same if you think it is good.

decoNR · 2026-10-07T22:51:44Z

-            max_upload_urls=_get_max_corpus_uploads_per_task(),
-            max_download_urls=25000,
-            use_backup=True).serialize())
+    with _track_sync_corpus_metric(fuzz_target.fully_qualified_name()):


I think we only want to track in main. Can you investigate how this copy differs from the corpus sync?

This is a setup for future corpus sync operations. It lists files on the corpus and signs URLs for download/upload, which can be time consuming. So I believe it's a good idea to track this, since it's related to corpus sync.

Nice, agreed.

Before closing the associated bug, document all of these decisions (the final state) there so we can easily revisit it in the future.

Maybe should use another metric for this. IMO the corpus sync metric is meant to determine how much time we spent syncing corpus before running the fuzz session in main. Adding this here will cause confusion afterwards once we start trying to analyze each fuzz step duration.

We could later also try to have metrics for preprocess (like how much time we spend signing URLs) and postprocess.

I believe that adding a 'stage' label addresses this issue, and also allows us to have more granularity (since in the main stage there are two places with corpus sync: on startup to download the corpus and once it finishes, to send new corpus files).

If in the future we decide to use another metric for this, we can just change the metric and it will be done.

brun0-matheus · 2026-10-08T12:38:22Z

since blackbox fuzzing does not use corpus

nit: I think it does use but it does not sync corpus.

I talked to @PauloVLB, and he confirmed that it doesn't use corpus. It uses something different called data bundle.

PauloVLB · 2026-10-08T12:54:22Z

Hey @decoNR and @brun0-matheus. Just trying to clarify:

nit: I think it does use but it does not sync corpus.

Yeah, there's some confusion here because of naming, but mostly yes: blackbox fuzzing doesn't use a corpus the way engine fuzzing does.

Blackbox fuzzers use a data bundle instead. It's a GCS bucket synced one way to the bot (update_data_bundle) and passed to the fuzzer script as --input_dir (here). Nothing gets uploaded back to it.
The confusing part is the naming: the /corpora page is actually the data bundle UI (corpora.py), and the proto is called DataBundleCorpus (get_proto_data_bundle_corpus).
The actual corpus (GCS sync, uploading new files, pruning, backups) is engine only. Preprocess only builds it when a fuzz target is picked (here), and that only happens for engine jobs (_pick_fuzz_target).

So keeping this metric engine only makes sense to me. Data bundle sync is a separate path in setup, so it's out of scope here.

decoNR · 2026-10-09T12:42:24Z

Hey @decoNR and @brun0-matheus. Just trying to clarify:

nit: I think it does use but it does not sync corpus.

Yeah, there's some confusion here because of naming, but mostly yes: blackbox fuzzing doesn't use a corpus the way engine fuzzing does.

Blackbox fuzzers use a data bundle instead. It's a GCS bucket synced one way to the bot (update_data_bundle) and passed to the fuzzer script as --input_dir (here). Nothing gets uploaded back to it.

The confusing part is the naming: the /corpora page is actually the data bundle UI (corpora.py), and the proto is called DataBundleCorpus (get_proto_data_bundle_corpus).

The actual corpus (GCS sync, uploading new files, pruning, backups) is engine only. Preprocess only builds it when a fuzz target is picked (here), and that only happens for engine jobs (_pick_fuzz_target).

So keeping this metric engine only makes sense to me. Data bundle sync is a separate path in setup, so it's out of scope here.

Nice! Thanks for the details.

What I meant is that, in a way, it still uses a corpus; the only difference is how it is stored and synced. Stating that "blackbox does not use a corpus" makes it sound as it relies on a completely different mechanism.

decoNR

LGTM

Add to the description that the setup time is also measured in the metric.

Create a generic method `insert` in Metric, which automatically handles which method to call and does integer conversition if needed. Also change TimeMetricTracker and the correspondings tests.

ViniciustCosta

fuzzer: [...] My conclusion is that the fully_qualified_name (which is the fuzz_task fully_qualified_name for engine fuzzing) fully describes the corpus, so there is no need to keep a job label.

The fact that the fully qualified name describes the corpus is unrelated to the metric label itself. We will use the metric label to understand if certain jobs or fuzzers take more time than other, for example. Multiple jobs can have the same fully qualified fuzzer name.

One thing to notice is that, for engine fuzzers, fuzzer_name is the fuzz engine, while for blackbox fuzzers, it is the specific fuzzer (since there is no fuzz target).

AFAICT there are two approaches for current labels in cf:

The FUZZING_SESSION_DURATION approach, which uses a CumulativeDistribution metric type with fuzzer_name (not the fully quilified name that includes the fuzz target to avoid high cardinality) and job.
The FUZZER_TOTAL_FUZZ_TIME/JOB_TOTAL_FUZZ_TIME approach, which uses a Counter metric with separate metrics for per job and per fuzz target visibility.

It comes down to answering "is the goal of the metric system/job-level observability or to compare with TOTAL_FUZZ_TIME and pinpoint slow individual fuzz_targets? We can discuss offline to decide that

ViniciustCosta · 2026-10-15T17:16:43Z

+               metric: Metric,
+               labels=None,
+               has_success_label: bool = False):
+    # If |has_success_label| is true, a "success" label will be added upon exit,


Add this to the docstring (go/pystyle#class-docs)

The stage label can have 3 values: preprocess, download and upload.

Create a SplitMetricHelper to aid emission of this type of metric.

brun0-matheus · 2026-10-16T14:34:40Z

fuzzer: [...] My conclusion is that the fully_qualified_name (which is the fuzz_task fully_qualified_name for engine fuzzing) fully describes the corpus, so there is no need to keep a job label.

The fact that the fully qualified name describes the corpus is unrelated to the metric label itself. We will use the metric label to understand if certain jobs or fuzzers take more time than other, for example. Multiple jobs can have the same fully qualified fuzzer name.

One thing to notice is that, for engine fuzzers, fuzzer_name is the fuzz engine, while for blackbox fuzzers, it is the specific fuzzer (since there is no fuzz target).

AFAICT there are two approaches for current labels in cf:

The FUZZING_SESSION_DURATION approach, which uses a CumulativeDistribution metric type with fuzzer_name (not the fully quilified name that includes the fuzz target to avoid high cardinality) and job.

The FUZZER_TOTAL_FUZZ_TIME/JOB_TOTAL_FUZZ_TIME approach, which uses a Counter metric with separate metrics for per job and per fuzz target visibility.

It comes down to answering "is the goal of the metric system/job-level observability or to compare with TOTAL_FUZZ_TIME and pinpoint slow individual fuzz_targets? We can discuss offline to decide that

As discussed offline, I decided to do the FUZZER_TOTAL_FUZZ_TIME/JOB_TOTAL_FUZZ_TIME approach.

brun0-matheus added 2 commits October 6, 2026 14:18

Add SYNC_CORPUS_TIME metric

93be813

Add a metric do keep track of corpus synchronization. The metric also distinguishes between corpus download/upload and whether it was successful or not.

Add unittest to SYNC_CORPUS_TIME metric emission

164c89a

Add unittest to check if SYNC_CORPUS_TIME is being emmited correctly. It tests all 4 possible combinations of (`success`, `is_upload`).

brun0-matheus requested a review from decoNR October 6, 2026 15:58

brun0-matheus self-assigned this Oct 6, 2026

brun0-matheus requested a review from a team as a code owner October 6, 2026 15:58

decoNR reviewed Oct 7, 2026

View reviewed changes

Comment thread src/clusterfuzz/_internal/bot/tasks/utasks/fuzz_task.py Outdated

Comment thread src/clusterfuzz/_internal/bot/tasks/utasks/fuzz_task.py Outdated

brun0-matheus added 3 commits October 7, 2026 15:04

Add unit test for TimeMetricTracker

0fd4781

Track corpus sync time in preprocess

3de0d57

During preprocessing, there are some possibly time intensive operations regarding listing corpus files and/or signing URLs. Added code to track time spent in this region. Also added unittests to check this.

decoNR reviewed Oct 7, 2026

View reviewed changes

Fixed type annotations of gcs_corpus.upload_files

fb30eab

decoNR approved these changes Oct 9, 2026

View reviewed changes

brun0-matheus added 2 commits October 9, 2026 15:11

Update metric description and minor refactor

371f1e4

Add to the description that the setup time is also measured in the metric.

Add generic insert method in Metric

c892d9d

Create a generic method `insert` in Metric, which automatically handles which method to call and does integer conversition if needed. Also change TimeMetricTracker and the correspondings tests.

decoNR reviewed Oct 13, 2026

View reviewed changes

Comment thread src/clusterfuzz/_internal/metrics/monitor.py Outdated

ViniciustCosta self-requested a review October 14, 2026 14:07

ViniciustCosta requested changes Oct 15, 2026

View reviewed changes

ViniciustCosta reviewed Oct 15, 2026

View reviewed changes

brun0-matheus added 3 commits October 15, 2026 17:41

Change comment to docstring and minor refactor

bc7c3de

Add stage label to sync corpus metric

43baea0

The stage label can have 3 values: preprocess, download and upload.

Make FUZZER_* and JOB_* metrics for sync corpus

e7559a0

Create a SplitMetricHelper to aid emission of this type of metric.

Conversation

brun0-matheus commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Changes

Tests

Uh oh!

decoNR commented Oct 7, 2026

Uh oh!

decoNR left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

brun0-matheus commented Oct 7, 2026

Uh oh!

brun0-matheus commented Oct 7, 2026

Uh oh!

decoNR commented Oct 7, 2026

Uh oh!

decoNR left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

decoNR Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

brun0-matheus Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

decoNR Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

ViniciustCosta Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

brun0-matheus Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

brun0-matheus commented Oct 8, 2026

Uh oh!

PauloVLB commented Oct 8, 2026

Uh oh!

decoNR commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

decoNR left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ViniciustCosta left a comment

Choose a reason for hiding this comment

Uh oh!

ViniciustCosta Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

brun0-matheus Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

brun0-matheus commented Oct 16, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

brun0-matheus commented Oct 6, 2026 •

edited

Loading

decoNR commented Oct 9, 2026 •

edited

Loading