Skip to content

API Fuzz testing telemetry & usage ping information

Problem to solve

Understanding how our users use the product helps us make better, data-driven decisions about what to do in future iterations. This is difficult without usage ping information to show how users actually use the product.

Proposal

Add usage ping information to report on:

  1. Number of jobs with fuzz testing run

Add Snowplow information to report on:

  1. Number of jobs with fuzz testing run

Note: We need to be able to identify these usage pings separate from coverage-fuzzing usage pings and then be able to combine the two to still get the distinct user count. This is so we can report on each individual approach as well as fuzz testing as a whole group. As an example, we will combine both and count distinct users on this graph and then create another graph to count just the API fuzz testing users.

With each report, provide a way to identify which organization and user in it caused the event to occur. Note: We explicitly do NOT want to collect exact names of organizations or individual users. We are interested in being able to understand if there are a few or many unique users generating reports, but we do not need nor want names of individual users. Consider what anonymization techniques we use in other parts of usage ping and/or using a one-way hash function.

Links / references

Technical Implementation

  1. Review Ping Usage
  2. Create queries for each metric
    1. Optimize queries with #database-lab
  3. Add the metric definition
    1. Update Event Dictionary’s Usage Ping table
  4. Manual testing
  5. Write tests (assuming Ping can be tested, no reference in usage guide)
  6. Add a changelog file
  7. Ask for a Telemetry Review
  8. Create an issue to verify your metric post production

Later iterations of this issue

In later iterations we will want to collect:

  1. Number of fuzz testing jobs that found a faults
  2. Number of fuzz testing jobs that found no faults

Create a report to report (these two are both to help us address our North Star metric for fuzzing):

  1. The total number of fuzz results that have had any interaction
  2. The total number of fuzz results that have been reported
Edited by Michael Eddington