Convert gl-apifuzzing-runner to csharp

Problem

API Fuzzing has a python script gl-apifuzzing-runner that consumes various static definition files converting them into HTTP requests and sending them to the API Fuzzing scanner.

This worked great at first, however the features of gl-apifuzzing-runner are expanding and will require code duplication if kept as a python script. Also, long term, generating HTTP requests from definition files will need to move into the scanner core. By rewriting the runner in csharp we will be ready for the integration.

The following commands will be migrated:

Command	Migrate
burp	No
cmd	No
folder	No
har	Yes
postman	Yes
swagger	Yes

Proposal

Create a new command line csharp tool that implements the existing runner functionality in csharp. The implementation will leverage existing work and classes from the Peach.Web project. The implementation will also make use of existing dependencies used by PeachWeb such as NSwag for OpenAPI/Swagger support, and the Newtonsoft json library.

Add a new command line csharp application Peach.Web.Runner
1. Executable name gl-apifuzzing-runner
Use the same command line arguments and commands
1. See https://github.com/commandlineparser/commandline

Internally the requests will end up as Peach.Web.Core.Models.WebApi.Operation objects. This will include creating EndPoint and Resources objects as well. These WebApi models what the scanning engine operates on. In the future this new runner code will be integrated and these models will be operated on directly.

The WebApi is modeled after REST API concepts and language used by the OpenAPI specifications.

To send the requests through the scanner proxy, convert Operations into the standard csharp web client. Multipart messages must be supported.

graph LR
  HAR --> Operation --> HttpClient

Runner Overview

The purpose of runner is to allow testing of static requests from supported file formats. This is done by sending the requests provided by a support file format repeatedly using the scan engine as a proxy.

High level internal logic:

Convert static file into an array of requests
1. Update with base url if provided via --base-url
2. Update headers if provided via --header (note- multiple --header options can be provided)
Start a new test session if we are running stand alone
For each request in requests
1. Tell PeachWeb a new request is being tested
2. Do
  1. Update request with overrides.
    1. Overrides can change over time if an --overrides_cmd is provided
  2. Send request via PeachWeb proxy port
3. While PeachWeb says continue

runner has two modes of operation: stand alone, or in conjunction with gl-apifuzzing-ci. When running standalone the runner will create a new scan job and then send the requests through the proxy. When used by gl-apifuzzing-ci the runner will not create a new job.

The method of use can be detected by looking for the following environment variables:

FUZZAPI_SESSIONID - This provides the job id needed to call the PeachWeb APIs
FUZZAPI_PROXY - This provides the proxy URL used when sending requests

CLI Options

Here are the existing CLI options. All of the long form parameters (--api, --header, etc.) can also be provided as environment variables with the FUZZAPI_ prefix. So for --api foo you could set FUZZAPI_API=foo. This is handled automatically by the python click module the options are defined using. If possible I would like to support environment variables in the csharp version, but it's not a must have.

Usage: gl-apifuzzing-runner [OPTIONS] COMMAND [ARGS]...

  Run one or more commands that generate traffic through API Fuzzing.

Options:
  --ci / --no-ci              Running from CI script, ensure no auto start.
  -a, --api TEXT              API Fuzzing API URL. Defaults to FUZZAPI_API
                              environ.
  -p, --project TEXT          API Fuzzing project to use. Ex. Default
  -r, --profile TEXT          API Fuzzing profile to use. Ex. Quick
  -u, --base-url TEXT         Base url for requests.  This overrides the 
                              recorded base url. Ex: '-u http://api.foo.com', 
                              '-u http://api.foo.com:7777'"
  -H, --header TEXT           Provide header. Multiple header arguments can be 
                              provided. Ex: '-H "Header: Value"'
  -o, --overrides TEXT        File containing overrides for headers and
                              cookies.
  --overrides_env TEXT        Provide overrides file contents via command line
  --overrides_cmd TEXT        Command that creates/updates overrides file. If
                              --overrides_interval provided, run at provided
                              interval.
  --overrides_interval FLOAT  Interval in minutes to run --overrides_cmd.
  --dryrun / --no-dryrun      Try operation with out API Fuzzing
  --timeout INTEGER           Timeout for a single test response. Defaults to
                              30 seconds.
  -v                          Increase verbosity of output. Supply more than
                              once to continue increasing.
  -F, --config TEXT           Use to specify full path to config file for this
                              job run.  Defaults to FUZZAPI_CONFIG environ
  --auth [httpbasic]          Select authentication type: httpbasic
  --username TEXT             Provide a username for authentication. Requires
                              --auth parameter.
  --password TEXT             Provide a password for authentication. Requires
                              --auth parameter.
  --help                      Show this message and exit.

Commands:
  har       Replay recorded requests from exported HTTP...
  postman   Replay recorded requests from exported...
  swagger   Generate requests using Swagger/OpenAPI...

Overrides

The overrides feature allows users to set or override headers, cookies, and parameters using a simple json document. The document is provided through a command line option --overrides_env={...}, a file on disk --overrides=filename, or a combination of file on disk, command, and run interval.

When the user provides a command to run, the command is expected to update the overrides json file when it runs. This feature is provided to handle cases where authentication tokens expire. When using a command the following options are required:

--overrides - Profile a filename
--overrides_cmd - Provides a command line to execute
--overrides_interval - Interval in minutes to run the command

The command is run before sending any requests to create the initial json file. Then in the background every N minutes.

One of the design goals for overrides is providing a simple, small format that can be provided easily through an environmental variable.

The current format used by the python runner will be extended for the csharp runner to include several optional properties: query, body-form, body-json, body-xml.

{
  "headers":    { "name": "value"  }, 
  "cookies":    { "name": "value"  },
  "query":      { "name": "value"  },
  "body-form":  { "name": "value"  },
  "body-json":  { "jsonpath": "value" },
  "body-xml" :  { "xpath":    "value" }
}

Changing a header

{ "header": {"Authorization": "Token b5638ae7-6e77-4585-b035-7d9de2e3f6b3"} }

Parameter in query string

{ "query": {"api_key": "b5638ae7-6e77-4585-b035-7d9de2e3f6b3"} }

Parameter in form encoded body

{ "body-form": {"api_key": "b5638ae7-6e77-4585-b035-7d9de2e3f6b3"} }

Parameter in json encoded body

The parameter key is a JSONPath expression.

{ "body-json": {"$.api_key": "b5638ae7-6e77-4585-b035-7d9de2e3f6b3"} }

Parameter in xml encoded body

The parameter key is an xpath expression.

Set value to all xml elements named api_key:
```
{ "body-xml": {"//api_key": "b5638ae7-6e77-4585-b035-7d9de2e3f6b3"} }
```
Set value to all attributes named api_key:
```
{ "body-xml": {"//@api_key": "b5638ae7-6e77-4585-b035-7d9de2e3f6b3"} }
```

Changing multiple parameters

When changing multiple parameters, additional properties are added.

{ 
  "body-json": {
    "$.api_key": "b5638ae7-6e77-4585-b035-7d9de2e3f6b3",
    "$.other": "some other value"
  } 
}

Changing a header and parameter

{ 
  "header": { "X-Special-Header": "XYZ" },
  "body-json": {
    "$.api_key": "b5638ae7-6e77-4585-b035-7d9de2e3f6b3",
    "$.other": "some other value"
  } 
}

HTTP Archive (HAR)

Only one version v1.2 is supported
Support loading as local file and URL

OpenAPI

Support both v2.0 and v3 OpenAPI specification formats
Support both local files and also URLs to load the specification
Use example/default data for operations if provided in the specification

Postman

Support both v2.0 and v2.1 Postman Collection types
Support nested item groups
Support loading by file name and URL

Postman variables overrides

In order to support postman variables. There are two possible sources for the values of postman variables. They can be specified thru environmental variables:

FUZZAPI_POSTMAN_COLLECTION Contains a file path to a postman file.
FUZZAPI_POSTMAN_COLLECTION_VARIABLES contains a file path to a json file. The JSON file is a collection of key-value pais, for example: { "var_name1" : "value_1", "var_name2" : "value_2" }

When the runner is converting Postman request into Operation and it finds a variable reference. It can look up for its value in the following order:

First, into the JSON defined by FUZZAPI_POSTMAN_COLLECTION_VARIABLES.
Second, into the postman file defined by FUZZAPI_POSTMAN_COLLECTION to look up the value of the variable.
Third, the input postman (input) file provided in the command line.

If the value is not found in any of the sources, then an error can be thrown indicating the missed value for the variable.

Tasks

Edited Feb 20, 2021 by Herber Madrigal