Skip to content

Add API Fuzzing usage data

Michael Eddington requested to merge 210345-api-fuzz-telemetry-by-faults into master

What does this MR do?

Adds a user_api_fuzzing_jobs and user_api_fuzzing_dnd_jobs event to the secure stage pings.

  • Event dictionary updated

Addresses #210345 (closed)

Related MR updating index for performance !47692 (merged)

Queries

Numerous queries are performed, the following are the ones related to the new events.

SELECT
   COUNT(DISTINCT "ci_builds"."user_id") 
FROM
   "ci_builds" 
WHERE
   "ci_builds"."type" = 'Ci::Build' 
   AND "ci_builds"."name" = 'apifuzzer_fuzz' 
   AND "ci_builds"."user_id" >= 1 
   AND "ci_builds"."user_id" < 10001

Cold Query:

Aggregate  (cost=766.17..766.18 rows=1 width=8) (actual time=6.128..6.129 rows=1 loops=1)
   Buffers: shared hit=186
   ->  Index Only Scan using index_secure_ci_builds_on_user_id_name_created_at on public.ci_builds  (cost=0.56..766.10 rows=27 width=4) (actual time=5.998..5.998 rows=0 loops=1)
         Index Cond: ((ci_builds.user_id >= 1) AND (ci_builds.user_id < 10001) AND (ci_builds.name = 'apifuzzer_fuzz'::text))
         Heap Fetches: 0
         Buffers: shared hit=177
SELECT
   COUNT(DISTINCT "ci_builds"."user_id") 
FROM
   "ci_builds" 
WHERE
   "ci_builds"."type" = 'Ci::Build' 
   AND "ci_builds"."name" = 'apifuzzer_fuzz' 
   AND "ci_builds"."created_at" BETWEEN '2020-10-16 02:28:58.590164' AND '2020-11-13 02:28:58.590362' 
   AND "ci_builds"."user_id" >= 1 
   AND "ci_builds"."user_id" < 10001
Aggregate  (cost=947.17..947.18 rows=1 width=8) (actual time=2.791..2.793 rows=1 loops=1)
   Buffers: shared hit=177
   ->  Index Only Scan using index_secure_ci_builds_on_user_id_name_created_at on public.ci_builds  (cost=0.56..947.16 rows=2 width=4) (actual time=2.770..2.771 rows=0 loops=1)
         Index Cond: ((ci_builds.user_id >= 1) AND (ci_builds.user_id < 10001) AND (ci_builds.name = 'apifuzzer_fuzz'::text) AND (ci_builds.created_at >= '2020-10-16 02:28:58.590164'::timestamp without time zone) AND (ci_builds.created_at <= '2020-11-13 02:28:58.590362'::timestamp without time zone))
         Heap Fetches: 0
         Buffers: shared hit=177
SELECT
   COUNT(DISTINCT "ci_builds"."user_id") 
FROM
   "ci_builds" 
WHERE
   "ci_builds"."type" = 'Ci::Build' 
   AND "ci_builds"."name" = 'apifuzzer_fuzz_dnd' 
   AND "ci_builds"."user_id" >= 1 
   AND "ci_builds"."user_id" < 10001
Aggregate  (cost=766.17..766.18 rows=1 width=8) (actual time=2.715..2.716 rows=1 loops=1)
   Buffers: shared hit=177
   ->  Index Only Scan using index_secure_ci_builds_on_user_id_name_created_at on public.ci_builds  (cost=0.56..766.10 rows=27 width=4) (actual time=2.698..2.698 rows=0 loops=1)
         Index Cond: ((ci_builds.user_id >= 1) AND (ci_builds.user_id < 10001) AND (ci_builds.name = 'apifuzzer_fuzz_dnd'::text))
         Heap Fetches: 0
         Buffers: shared hit=177
SELECT
   COUNT(DISTINCT "ci_builds"."user_id") 
FROM
   "ci_builds" 
WHERE
   "ci_builds"."type" = 'Ci::Build' 
   AND "ci_builds"."name" = 'apifuzzer_fuzz' 
   AND "ci_builds"."created_at" BETWEEN '2020-10-16 02:28:58.590164' AND '2020-11-13 02:28:58.590362' 
   AND "ci_builds"."user_id" >= 1 
   AND "ci_builds"."user_id" < 10001
Aggregate  (cost=947.17..947.18 rows=1 width=8) (actual time=2.961..2.963 rows=1 loops=1)
   Buffers: shared hit=177
   ->  Index Only Scan using index_secure_ci_builds_on_user_id_name_created_at on public.ci_builds  (cost=0.56..947.16 rows=2 width=4) (actual time=2.935..2.935 rows=0 loops=1)
         Index Cond: ((ci_builds.user_id >= 1) AND (ci_builds.user_id < 10001) AND (ci_builds.name = 'apifuzzer_fuzz_dnd'::text) AND (ci_builds.created_at >= '2020-10-16 02:28:58.590164'::timestamp without time zone) AND (ci_builds.created_at <= '2020-11-13 02:28:58.590362'::timestamp without time zone))
         Heap Fetches: 0
         Buffers: shared hit=177
SELECT
   MIN("ci_builds"."user_id") 
FROM
   "ci_builds" 
WHERE
   "ci_builds"."type" = 'Ci::Builds' 
   AND "ci_builds"."name" IN 
   (
      'container_scanning',
      'dast',
      'dependency_scanning',
      'license_management',
      'license_scanning',
      'sast',
      'secret_detection',
      'coverage_fuzzing',
      'apifuzzer_fuzz',
      'apifuzzer_fuzz_dnd'
   )
   AND "ci_builds"."created_at" BETWEEN '2020-10-16 02:28:58.590164' AND '2020-11-13 02:28:58.590362'
Aggregate  (cost=13669478.06..13669478.07 rows=1 width=4) (actual time=267263.917..267263.919 rows=1 loops=1)
   Buffers: shared read=5059270
   I/O Timings: read=224760.045
   ->  Index Scan using index_ci_builds_on_status_and_type_and_runner_id on public.ci_builds  (cost=0.70..13669478.05 rows=1 width=4) (actual time=267263.909..267263.909 rows=0 loops=1)
         Index Cond: ((ci_builds.type)::text = 'Ci::Builds'::text)
         Filter: ((ci_builds.created_at >= '2020-10-16 02:28:58.590164'::timestamp without time zone) AND (ci_builds.created_at <= '2020-11-13 02:28:58.590362'::timestamp without time zone) AND ((ci_builds.name)::text = ANY ('{container_scanning,dast,dependency_scanning,license_management,license_scanning,sast,secret_detection,coverage_fuzzing,apifuzzer_fuzz,apifuzzer_fuzz_dnd}'::text[])))
         Rows Removed by Filter: 0
         Buffers: shared read=5059270
         I/O Timings: read=224760.045

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #210345 (closed)

Edited by Michael Eddington

Merge request reports