Advanced Scan Configuration Options

Introduction

Scan Configuration File Structure

Types

Enumerated Types

Bitmasks

Default Values

Terminology

Scan Configuration Schema Changes

ScanConfig Structure

ScanConfig's Scalar Values

ScanConfig's Objects

ScanConfig's Lists

CrawlConfig Structure

CrawlConfig's Scalar Values

CrawlConfig's Objects

CrawlConfig's Lists

AuthConfig Structure

AuthConfig 's Scalar Values

AuthConfig 's Objects

AuthConfig 's Lists

AnalyzerConfig Structure
 AnalyzerConfig 's Scalar Values

AnalyzerConfig 's Objects

AnalyzerConfig 's Lists

AttackerConfig Structure

AttackerConfig 's Scalar Values

AttackerConfig 's Objects

AttackerConfig 's Lists

Introduction
The Scan Configuration file stores all scan parameters: name of the scan, list of URLs to scan, what
attacks to run, and many more. The Scan Configuration file is created by the AppSpider User Interface
application and is loaded by the ScanEngine process every time a new scan is started based on that
Scan Configuration file.

The most commonly used Scan Configuration options have specialized editors in the Scan
Configuration Editor window. This document describes the advanced options that do not have
corresponding pages in the Scan Configuration Editor Window and can be accessed either by editing
the Scan Configuration file directly or modifying them on the Advanced Page in the Scan Configuration
Editor.

Scan Configuration File Structure

The Scan Configuration file is an XML file and has a hierarchical structure. The top-level data structure
in the file is ScanConfig. Everything else in the file resides somewhere below ScanConfig in the
hierarchy.

There are three types of elements in the Scan Configuration file:

● Scalar: a single value that can be of one of the type accepted in the Scan Configuration file
(see Types paragraph). For example, ScanName is a Scalar object
● Composite Object: an object that can contain other heterogeneous elements: Scalars,
Composite Objects and Lists. For example, the top-level object in the Scan Configuration file
(ScanConfig object) is a Composite Object.
● List - is an element that can contain homogeneous elements: Scalars or Composite Objects.
Note that Lists cannot contain other Lists directly.

Types
Scalar values in the Scan Configuration file can be of one of the following types:
 ▪ String: A String type.
▪ LargeString - A String type optimized for large strings
▪ UniqueString - A string type optimized for string that most likely be repeated multiple times.
▪ LargeUniqueString - A string type optimized for large strings that a likely to be repeated
multiple times.
▪ LargeCompressedString - A string type that store strings in compressed form.
▪ ByteNumber - A one-byte number
▪ WordNumber - A two-byte number
▪ Number - A four-byte number
▪ LargeNumber - An eight-byte number
▪ Enumeration - A data type consisted on named predefined values (see section 'Enumerated
Types')
▪ Float - a four-byte floating point number
▪ Double - an eight-byte floating point number
▪ Boolean - A boolean type. Note that Boolean is not an enumerated type; it is stored as a
Number with values either 0 or 1. 1 has a meaning of TRUE, 0 signifies FALSE.
▪ Date - Store Date data. It is stored in the following format: YYYY-MM-DD HH:MM:SS
▪ Duration - Store time duration data. It is stored in the following format: [DD Day[s]]
HH:MM::SS, where DD is the number of days, and square brackets show optional elements

Enumerated Types
Many Scalar elements in the Scan Configuration file have Enumeration types. Enumeration type is
stored as an integer but allows a user to reference integer values by their symbolic name. An
enumerated type value can be set using either its numeric or symbolic form. For example, the
following two lines from Scan Configuration file are equivalent because 'High' is enumerated value
defines as '4'

<Severity>High</Severity>

<Severity>4</Severity>

For every Scalar element of Enumerated Type, the document provides all possible values of that
element.

Bitmasks
Some Enumerated Types can be BitMasks, where the value of an element is defined by several bits.
Enumerated values of BitMasks are defined in such way that they do not have same bit set and can be
used in bitwise-OR operation to produce the final value. Multiple bits that are set in the bitmask are
separated with character '|' as depicted below:

<AttackPoints>Directory|File|Parameter</AttackPoints>
As for Enumerated Types, the values of the Bitmasks can be set using either numeric or symbolic
form. The following two lines below are equivalent because value 'Parameter' is defined as '16'

<AttackPoints>Directory|File|Parameter</AttackPoints>

<AttackPoints>Directory|File|16</AttackPoints>

Default Values
When a new Scan Configuration file is created all Scalar Elements are assigned their default values.
Default values are configured in such a way that they produce best results for majority of web sites.
For every element, this document provides the default value used by AppSpider. Note that even if an
element is not present in the Scan Configuration file (for example, it was removed from the file by the
user), AppSpider will use the default value of the element. For every Scalar value, this document
defines its default value used by AppSpider.

Terminology

This section describes commonly used terms that are used throughout the document and may cause
some confusion because the meaning of those terms in the context of AppSpider is sometimes
different than the common use of those terms

● Domain. Domain is a website located on a certain port. Domain is identified by the name or IP
address of website, its protocol and port. For instance, the following domains are different
○ ​ [Link] and
○ [Link] host

because they have different protocols: HTTP and HTTPS. Also, the following domains are
different

○ [Link]
; and
○ [Link]
are two different domains

because they have different ports: 8080 and 80.

If the same domain is referenced by IP address and name in the application, AppSpider
creates two domains. Consider the following example:

○ [Link]
; and
○ [Link]

Chances are those web sites represent the same site on the web server but AppSpider creates
two domains, because they website names are different: ‘localhost’ and [Link].

● Web Resource. Web resource is a request that was found by the crawler. Request can be a a
static URL referenced in an HTML page, a form action request (POST request), AMF request
 sent from a Flash application, XMLHTTPRequest sent from a javascript, in general, any HTTP
request that can be sent by the client application to the web server. Web Resource is
identified by its URL and the set of used parameters (Query and POST) and parameter values.
For instance, below is the example of two different Web Resources
○ [Link]
○ [Link] 2

because they have different parameters values: id=21 and id=22.

● Crawl Result. Crawl Result the Web Resource that the crawler retrieved from the server for
analysis. Note that not all Web Resources that were found by the crawler are crawled. There
are many reasons why web resources are not crawled, most common are: web resources are
on another website, the user blacklisted web resource is scan configuration, one of the
crawler’ crawl limits prevent crawling this web resource, for instance, the maximum number
of Web Resource that crawler is allowed to crawl.

Scan Configuration Schema Changes

AppSpider development is an ongoing process. Developers frequently come across new types of web
sites that require some special treatment and require new ways of configuring a scan execution. As a
result, the number of parameters in the Scan Configuration increases with every major release of
AppSpider. To avoid any problems with Scan Configuration files created by the previous versions of
AppSpider or by user's programs (via AppSpider API), only new elements are added to the scan
configuration files and all existing elements and their meaning are preserved.

ScanConfig Structure
ScanConfig is the top-level structure in the Scan Configuration File. ScanConfig's composite objects are
presented in the Advanced Tab of the Scan Configuration Dialog. The scalar values of ScanConfig can
only be modified by editing the ScanConfiguration file. Note, that after manual editing, the Scan
Configuration File, AppSpider application should be re-started.

ScanConfig's Scalar Values

Property Description
Name The name of the scan configuration.

Value:
Type: String
Default: None

Remarks:

This parameter is mandatory. It cannot have an empty string.

AppVersion The version of the application that was used to create the scan
configuration.

Value:
Type: String
Default: Current Major Version of Scan Engine

Remarks:

This parameter is not used by the scan engine.

Log Enables/Disables logging into the operation log.

Value:
Type: Boolean
● 0: logging is disabled
● 1: logging is enabled

Default: 1

Remarks:

DetailedLogging Enables/Disables detailed logging. Detailed logging

Value:
Type: Boolean
● 0: detailed logging is disabled
● 1: detailed logging is enabled

Default: 0

Remarks:

Detailed logging increases the amount of the information ScanEngine

logs during scan execution. That extra information appears in the
operation log. While it is useful to enable detailed logging for
debugging, it is recommended to disable it for normal scan execution.
Enabling detailed logging significantly increases the size of the log files
on disk and slows the scan.

IncludeTraffic Enables/Disables detailed logging of the network traffic

Value:
Type: Boolean
● 0: network traffic logging is disabled
● 1: network traffic logging is enabled
 Default: 0

Remarks:

WindowsErrors Deprecated
UseSystemDsn Deprecated
Recrawl Deprecated
PauseOnRecoverableError This flag controls the behavior of the scanner when it encounters a
recoverable error. A recoverable error is an error that often can be
corrected by the user. Following is a partial list of the errors that a user
may correct:
● Re-login problem
● Out of disk space
● Out of memory

Value:
Type: Boolean
● 0: scan will fail on a recoverable error
● 1: scan will pause on a recoverable error

Default: 1

Remarks:

A recoverable error is the error in scan execution that can be corrected

by the user. It is desirable to pause the scan (as opposed to stopping it)
in case of a recoverable error and give the user a chance to correct the
problem and continue the scan. For instance, a lack of disk space is
considered to be a recoverable error: the user can clean up the disk and
resume the scan. Other types of recoverable errors include Failure to
Re-login, lack of physical memory ofnthe machine and many more
ExecuteCommandLineURL Deprecated
NotifyScanDoneURL Deprecated
JavaScriptEngine This flag specifies which browser component should be used by the
scan.

Value:
Type: Enum
Enum Values:
● Internet Explorer (0): Internet Explorer Web Browser Control
will be used to execute javascript

Default: Internet Explorer

Remarks:
 MaxDatabaseSize The maximum size of the jet database scan data file after which the
scan is stopped.

Value:
Type: LargeNumber
Default: 1073741824 (1 GB)

Remarks:

MaxTrafficFiles The maximum number of traffic files the scanner will keep.

Value:
Type: Number
● '0' means unlimited traffic files will be kept

Default: 0

Remarks:
The scanner removes old traffic files (FIFO) after number of traffic file
reaches the number specified in this parameter

ScanConfig's Objects
Object Name Description
CrawlConfig Defines crawler parameters
AttackerConfig Defines attacker parameters
AttackPolicyConfig Defines attack policy: list of attack modules for the scan and
their parameters
AnalyzerConfig Defines analyzer parameters
AuthConfig Authentication configuration. This structure contains everything
related to authentication, login, re-login, logout detection.
ProxyConfig Proxy settings.
RemediationConfig Contains parameters for calculating remediation efforts
SSLCertConfig SSL client certificate settings
NetworkSettingsConfig Network parameters
PerformanceConfig Performance parameter
SystemRecommendationsConfig Contains parameters for computer hardware recommendations
HTTPHeadersConfig HTTP Headers
ManualCrawlingConfig List of traffic log files to import
AutoSequenceConfig Automatic sequence discovery settings
MacroConfig List of macros for the scan
SeleniumConfig List of selenium scripts and setting to run selenium scripts
WebServiceConfig Web service configuration
ReportConfig Report generation settings
 WAFConfig Deprecated
ScheduleConfig Deprecated
SiteTechnologyConfig Contains parameters that
OneTimeTokenConfig Contains parameters of One-Time Tokens (XSRF tokens)
CVSSConfig CVSS configuration
ParameterParserConfig Contains custom URL parameter parsers
ParameterValueConfig Contains description of parameters used to populate form
controls.

ScanConfig's Lists
List Name Description
N/A N/A

CrawlConfig Structure
CrawlConfig's Scalar Values
Property Description
MaxDomain Maximum number of domains that AppSpider will crawl.

Value:
Type: Number
Default: 100

Remarks:

Domain is a host name and protocol ([Link]

in the URL. For instance, [Link] and
[Link] are different domains. If AppSpider finds
a URL with an explicit IP address that matches a host name in
another URL (for instance [Link] and
[Link] AppSpider will consider those as
two different domains. Also, the protocol is a part of the
domain name, and, as a result, [Link] and
[Link] are two different domains.

MaxCrawlResults The maximum number of web resources that AppSpider is

allowed to retrieve from the server during the scan. A web
resource is identified by a unique combination of a URL and a
parameter (Query, POST). After that number is reached,
crawling is stopped.

Value:
Type: Number
Default: ​
5000
 Remarks:
MaxPerWebSiteCrawlResults Maximum number web resource crawler is allowed to crawl
per domain.

Value:
Type: Number
Default: -1 (Unlimited)

Remarks

MaxPerDirCrawlResults Maximum number of web resources in any directory the

crawler is allowed to retrieve.

Value:
Type: Number
Default: 500

Remarks

MaxPerLinkCrawlResults Maximum number of web resources for a given link the

crawler is allowed to retrieve. Link is a URL with all the
parameter.

Value:
Type: Number
Default: ​
50

Remarks
This option limits how many resources that have the same
URL but different variations of POST parameters can be
crawled
MaxPerNormalizedLinkCrawlResult Maximum number of resources the crawler is allowed to
request for a given normalized link. Normalized link is a URL
without parameter values.

Value:
Type: Number
Default: 100

Remarks

MaxPerDirChildNodes Maximum number of child nodes in the directory the crawler

is allowed to crawl. Child node is a directory or a file. This
parameter does not count grand children.

Value:
 Type: Number
Default: 300

Remarks

This parameter includes Crawl Results and sub-directories.

The crawler will stop crawling new resources in a directory if
one of the limits specified in MaxPerDirCrawlResult or
MaxPerDirChildNodes is reached for that directory. This
implies that the value of MaxPerDirCrawlResult should be
greater or equal to the value of MaxPerDirChildNodes.
MaxBlackListExtCrawlResults Number of resources that have blacklisted based on
extension the crawler is allowed to retrieve.

Value:
Type: Number
Default: 100

Remarks

Maximum number of Web Resources that are on

BlackListExtensionList ​ GrayListExtensionList ​
and ​ that
AppSpider is allowed to crawl. Even if the resource should be
blacklisted based on extension, AppSpider will still crawl a
small number of those resources the number of resources
specified in this parameter

This value is per domain.

MaxAttackFeedbackLinksCount Maximum number of new links discovered in attack traffic the

crawler will insert in the queue.

Value:
Type: Number
Default: ​
300

Remarks

Crawler monitors traffic of attack modules and tries to find

new links in the traffic. ​
This parameter specified maximum
number of new Web Resource found in the responses
received by attack modules. Some attack requests result is
responses that contain some of the attack payload or invalid
links.

Consider this example: It is very common for website to

return a reference to a URL that could help user to solve the
 problem. That reference often include the description of the
problem. For instance, if the attack set the following request:
GET /users/userinfo?userid=alert(‘111’), the response to the
attack could have the following statement (and a new link):

Invalid parameter to the request

/users/userinfo?userid=alert(‘111’)
<a href=”/mgmt/[Link]=Invalid parameter URL:
/users/userinfo?userid=alert(‘111’)”>Report error</a>

Clearly, every attack would result in new Web Resource

‘/mgmt/[Link]’ with different parameter values.
While it make sense to analyze that web resource once, it
does not make to analyze all of those web resources.
AppSpider tries automatically detect those invalid links and
avoid crawling them; this configuration parameter is a safety
net in case invalid link detection algorithm fails.

MaxPerFileNameCrawlResults Maximum number of Web Resources with the same file name
the crawler is allowed to analyzed

Value:
Type: Number
Default: 250

Remarks

During counting number of resources, only file name of the

URL is considered and path is ignored). For instance the
following two URLs are considered to have the same File
Name

[Link]
[Link]
78,

because the filename is the same for both URLs

RecursionDepth Maximum repetition that AppSpider will tolerate in URL.

Value:
Type: Number
Default: 2

Remarks:

This parameter defines how many times a part of a URL can

be repeated. For instance, in the example below, if the
 recursion depth is set to 2, URL #1 will be crawled, while the
URL #2 will be ignored because its recursion depth is 3.

#1: [Link]/dir1/dir2/dir1/dir2/[Link]
#2: [Link]/dir1/dir2/dir1/dir2/dir1/dir2/[Link]

MaxDirDepth Maximum number of directories AppSpider will look into.

URLs that have more directories in their path than the value
of this parameters will be ignored. For instance, URL
[Link]/dir1/dir2/dir3/[Link] will be ignored if
MaxDirDepth parameter is set to value smaller than 3.

Value:
Type: Number
Default: 10

Remarks

DiscoveryDepth Maximum discovery depth that AppSpider can go into the

site. Discovery depth of a URL is the number of steps (URL
navigations) it is required for the user to discover the link.

Value:
Type: Number
Default: -1 (Unlimited discovery depth)

Remarks

UrlRepetitionTolerance Maximum number of identical normalized URLs AppSpider is

allowed to crawl. Normalized URL is the URL without query
parameter values.

Value:
Type: Number
Default: 25

Remarks

SequenceRepetitionTolerance Maximum number of similar sequences that AppSpider will

try to follow.

Value:
Type: Number
Default: 5

Remarks
MaxReportedImages Maximum number of discovered Image links that AppSpider
should store in the database

Value:
Type: Number
Default: 500

Remarks

MaxReportedLinks This parameter defines maximum number of discovered Web

Resources that AppSpider should store in the database in
addition to Web Resources that will be crawled by the crawler

Value:
Type: Number
Default: 2500

Remarks

The crawler almost always sees more links that it will crawl.
Often it discovers many times more links that it will crawl.
Many Web Resources are ignored because one or more
crawler limitation. This parameter describes how many Web
Resources will be store in the database on top of the Web
Resources that were be crawled. While Web Resources take
significantly less space in the database then Crawl Results,
they still take some space, so it is recommended to keep this
number below the value specified in parameter
MaxCrawlResults

MaxReportedComments Maximum number of discovered HTML comments that

AppSpider should store in the database

Value:
Type: Number
Default: 500

Remarks

MaxReportedScripts Maximum number of discovered SCRIPTs that AppSpider

should store in the database

Value:
Type: Number
Default: 500

Remarks
MaxReportedEmails Maximum number of discovered Email addresses that
AppSpider should store in the database

Value:
Type: Number
Default: 500

Remarks

MaxReportedForms Maximum number of discovered forms that AppSpider should

store in the database

Value:
Type: Number
Default: 500

Remarks

MaxBrowserPageWaitTimeout Maximum time AppSpider should wait for the Browser

component to load the page and perform all operations.

Value:
Type: Number (time in milliseconds)
Default: 60000 (60 seconds)

Remarks

MaxBrowserWaitTillRequestTimeout Maximum time AppSpider should wait for the javascript on

the page to send an AJAX request to the server after firing an
event (for example, 'onclick' or 'onmouseover').

Value:
Type: Number (time in milliseconds)
Default: 4000 (4 seconds)

Remarks

MaxBrowserDOMDepth Maximum depth of DOMs that AppSpider should try to

analyze within an HTML page. DOM depth is minimum
number of user actions (events) that are required to reach
that DOM from the initial DOM of the page.

Value:
Type: Number
Default: 4
 Remarks

MaxBrowserEventsPerLink Maximum number of javascript events AppSpider should fire

per one link. A link is a URL without a query parameter and
the fragment.

Value:
Type: Number
Default: 200

Remarks

MaxBrowserEventsPerCrawlResult Maximum number of javascript events AppSpider should fire

per one web resource.

Value:
Type: Number
Default: 100

Remarks

MaxBrowserEventsPerDOM Maximum number of javascript events AppSpider should fire

per one DOM view.

Value:
Type: Number
Default: 100

Remarks

NotInsertedLinkCountThreshold Maximum number of ignored links that should be reported in

the User Log.

Value:
Type: Number
Default: 2

Remarks

AppSpider reports ignored links in the User Log so that the

user could easily notice that some URLs were unintentionally
ignored and correct the problem by modifying scan
configuration settings. In most cases if scan is configured
incorrectly, first several ignored URLs show the problem.

Note that only important messages should be reported in

User Log. AppSpider ignores many links during crawling (for
 instance, out-of-domain links), on an average site the number
of ignore links can be in thousands. To avoid cluttering User
Log, it is recommended to keep this number low.
CrawlPrioritization This parameter defines the algorithm that will be used to
crawl the site.

Value:
Type: Enum
Values:
▪ FIFO(numeric: 0)
▪ Smart(numeric: 1)
▪ DirBreadthFirst(numeric: 2)
▪ FoundBreadthFirst(numeric: 3)
▪ FoundDepthFirst(numeric: 4)
▪ Juicy(numeric: 5)
▪ LoginFormDiscovery(numeric: 6)
▪ Login(numeric: 7)

Default: Smart

Remarks

FileNotFoundRegex Regular Expression that is used by AppSpider to identify

custom 404 responses (File not found)

Value:
Type: String
Default: (page|resource) (you requested )?(was not|cannot
be) found|Page not found|404(.0)? - ((File (or directory )?not
found)|(Not Found))|HTTP Status 404|404 Not Found

Remarks

ServerErrorRegex Regular Expression that is used by AppSpider to identify error

responses from the web server

Value:
Type: String
Default: None

Remarks

InvalidURLRegexAttack Regular Expression that identifies URLs that comes from

attack traffic as Invalid so that AppSpider does not attack an
invalid URL.
 Value:
Type: String
Default:
['\"\\(\\)<>]|\\d([-+]|%2[bd])\\d|repeat\\(|alert\\(|/x\\w{7}\
\.txt

Remarks

InvalidURLRegexCrawl Regular Expression that identifies a URL that was discovered

during crawling as Invalid so that AppSpider does not crawl
and analyze an invalid URL.

Value:
Type: String
Default:
((\\s|%20)(OR|AND|MOD|ASC|DESC)(\\s|%20)|(<|%3c)(a|di
v|script|style|iframe|img)|[?&=]x[a-z0-9]{7}$|C=N;O=D|\\?C
=M)

Remarks

LockCookies Flags that tells AppSpider whether it should preserve the

value of the cookies supplied by the user in the Scan
Configuration even if the web server requested to change the
cookie.

Value:
Type: Boolean
▪ 1 - Lock cookie values
▪ 0: Do not look cookie values

Default: 1 - Lock cookie values

Remarks

CaseSensitivity This parameter tells AppSpider how to treat URLs of the web
site. The website can have either a case sensitive or a case
insensitive file system on the back end.

Value:
Type: Enum
Enum Values:
▪ AutoDetect (numeric: 0)
▪ CaseSensitive (numeric: 1)
▪ CaseInsensitive (numeric: 2)

Default: CaseSensitive
 Remarks

UniqueUrlsAcrossWebsites Deprecated
SaveReferences This parameter controls whether the crawler should store
cross-references in the database.

Value:
Type: Boolean
▪ 1: Save cross-references
▪ 0: Do not save cross-references

Default: 0 - Do not save cross-references

Remarks

Note that storing references significantly increases the size of

the database, and it is advised not to enable this feature.

UseBrowser Flag that tells the crawler to use browser to execute javascript
event handlers.

Value:
Type: Boolean
▪ 1: Use browser
▪ 0: Do not use browser

Default: 1 - Use browser

Remarks
This flag only affect using browser for crawling. It has no
affect on using browser for Macros, Sequences or attacks

ShowBrowser Flag that tells the crawler to show browser window during
traversing web site's pages

Value:
Type: Boolean
▪ 1: Show browser
▪ 0: Do not show browser

Default: 0 - Do not show browser

Remarks
 This flag was designed to be used to debug various crawling
problems. It is advised to disable this feature for regular
scans.

If this feature enabled, it is recommended to make the scan

Single-Threaded. This way, only one browser window will be
shown at any given moment.

StayOnPort Flag that tells the crawler to not deviate from the port of
original seed URLs. This implies that all seed URLs should be
on the same port if that option is enabled.

Value:
Type: Boolean
▪ 1: Crawler should stay on port
▪ 0: Crawler can request URLs from other ports

Default: 0: Crawler can request URLs from other ports

Remarks

RestrictToMacro This flag forces AppSpider to not crawl any links other than
the requests sent during macro execution

Value:
Type: Boolean
▪ 1: Crawler should try to discover new links
▪ 0: Crawler can discover new links

Default: 0 - Crawler can discover new links

Remarks

RestrictToManualCrawling This flag forces AppSpider to not crawl any links other than
the requests imported from proxy logs.

Value:
Type: Boolean
▪ 1: Crawler should try to discover new links
▪ 0: Crawler can discover new links

Default: 0 - Crawler can discover new links

Remarks

If the value is set to ‘1’, AppSpider will analyzed/attacked only

the requests that it imported from the proxy logs. If imported
 responses contain other links, those links will not be
crawler/analyzed/attacked. So, if the imported traffic has five
requests, only five requests will be analyzed/attacked.

RestrictToSeedList This flag forces AppSpider to not crawl any links other than
the seed links provided in the scan configuration.

Value:
Type: Boolean
▪ 1: Crawler should try to discover new links
▪ 0: Crawler can discover new links

Default: 0 - Crawler can discover new links

Remarks
RestrictToWebService This flag forces AppSpider to not crawl any links other than
the web service requests.

Value:
Type: Boolean
▪ 1: Crawler should try to discover new links
▪ 0: Crawler can discover new links

Default: 0 - Crawler can discover new links

Remarks
RestrictToSelenium This flag forces AppSpider to not crawl any links other than
requests performed during execution of Selenium scripts.

Value:
Type: Boolean
▪ 1: Crawler should try to discover new links
▪ 0: Crawler can discover new links

Default: 0 - Crawler can discover new links

Remarks
ImportCookiesFromTraffic This flag controls what AppSpider does with cookies that it
finds in the imported traffic.

Value:
Type: Boolean
▪ 1: Import cookies
▪ 0: Ignore cookies

Default: 0 - Ignore cookies

 Remarks
PageEqualThreshhold This parameter sets the minimum value of the similarity
coefficient above which two pages are considered to be
identical.

Value:
Type: Double
Default: 0.95

This parameter was introduced to deal with randomness in

the responses. Some pages always have advertisement
frames that website randomly inserts in the responses. This
parameter allows AppSpider to ignore that random content.

This parameter controls a proprietary algorithm that

determines similarity of the pages. The more similar the
pages are, the higher is their similarity coefficient. Two
responses with with similarity coefficient above the value of
the parameter are considered by AppSpider to be identical.
This information is used in several components in AppSpider:
when comparing responses with custom 404 response, to
determine whether that response is seen too often and can
be ignored, etc. It is recommended to not change the value of
this parameter.

PageSimilarThreshhold This parameter sets the minimum value of the similarity

coefficient above which two pages are considered to have
same structure.

Value:
Type: Double
Default: 0.80

This parameter was introduced to deal with pages that return

logically equivalent responses. For example, the shopping
cart’s checkout page with a sweater in the shopping cart will
look very similar to the page with a scarf in the shopping cart.
This parameter helps AppSpider to understand what pages
are similar. Several attacks and the analyzer use analysis for
page similarity.

This parameter controls a proprietary algorithm that

determines similarity of the pages. The more similar the
pages are, the higher is their similarity coefficient. Two
responses with with similarity coefficient above the value of
the parameter are considered by AppSpider to be similar. It is
recommended to not change the value of this parameter.
Flash This flags tells AppSpider whether it should analyze Flash files.

Value:
Type: Boolean
▪ 1: Should analyze Flash files
▪ 0: Should not analyze Flash files

Default: 1 - Should analyze Flash files

Remarks

EnableAdvancedParsers Internal parameter. The value provided in the scan

configuration file is overwritten
SearchForUrls This flags tells AppSpider whether it should try to find URLs in
places other than HTML structure: comments, javascript text,
etc.

Value:
Type: Boolean
▪ 1: Should look for URLs in non-standard locations
▪ 0: Should not look for URLs in non-standard locations

Default: 1 - Should look for URLs in non-standard locations

For example, consider the following HTML page with an HTML

comment block:

++++++
<a href=”/admin/show_users.php”>Show Users</a><br>
<!--
Do not forget that we need to remove
‘/admin/[Link]’ when we are done debugging
--->
<a href=”/admin/server_info.php”>Server Information</a>
++++++

If this flag is set to ‘1’, AppSpider will find URL

‘/admin/[Link]’. If the flag is set to ‘0’, AppSpider
won’t find/analyze/attack that URL.

MaxWebResourcesOverhead This flags tells AppSpider how many links it can add to the
crawl queue over the value specified in MaxCrawlResults
parameter. Those extra links provide the Crawler with ability
to pick more promising links to crawl. Without that
parameter, the crawler would stop looking for new links once
the queue is full.
 Value:
Type: Number
Default: 1000

Remarks

Note that even if those extra resources are added to the

queue, the crawler will stop crawling once it reached the
crawled number of links specified in the MaxCrawlResult
option

CrawlConfig's Objects
Property Description
N/A N/A

CrawlConfig's Lists
Property Description
SeedUrlList List of seed URLs from which AppSpider should start the
scan.
ScopeConstraintList This parameter contains rules that specify what URLs
AppSpider should crawl
BlackListExtensionList List of extensions that the crawler is not allowed to
crawl. See parameter ​ MaxBlackListExtCrawlResults​ for
the details.
GrayListExtensionList List of extensions that the crawler is not allowed to crawl
if Web Resource with the specified extensions do not
have query parameters. See parameter
MaxBlackListExtCrawlResults​ for the details
BinaryExtensionList List of file extensions that usually files with binary
content have
TextExtensionList List of file extensions that usually files with text content
have
BinaryContentTypeList List of content types that identify files with binary
content
HTMLContentTypeList List of content types that identify HTML content
TextContentTypeList List of content types that identify text content
XMLContentTypeList List of content types that identify XML content
BrowserDownloadWhitelistList List of URLs that browser should always download (for
example, javascript files)
 BrowserDoNotDownloadExtentionList List of file extensions that should not be downloaded
even if they were requested by the browser
BrowserDoNotDownloadContentTypeList List of content type of files that should not be
downloaded even if they were requested by the browser
LockedCookieList List of cookie names that should not change value for the
duration of the scan

AuthConfig Structure
AuthConfig data structure describes parameters required for authentication.

AuthConfig 's Scalar Values

Property Description
Type This parameter defines the type of authentication that will
be used by AppSpider.

Value:
Type: Enum
Enum Values:
▪ None (numeric: 0): No authentication
▪ Form (numeric: 1): Form-based automatic
authentication
▪ Macro (numeric: 2): Macro is used to authenticate
the user. The macro should be specified in
parameter 'MacroFile'
▪ SessionTakeover (numeric: 3): The user will provide
session cookies
▪ SSORedirect (numeric: 4)
▪ Bootstrap (numeric: 5)

Default: None

Remarks

HttpAuth Flag that tells that AppSpider should use HTTP username
and password from the config to login to site that use HTTP
authentication (Basic, NTLM, Kerberos, etc)

Value:
Type: Boolean
▪ 1: Should use HTTP authentication credentials
▪ 0: Should not use HTTP authentication credentials

Default: 0 - Should not use HTTP authentication credentials

ReloginAfterSessionLoss Flag that specifies whether AppSpider should re-login after
it detected session loss.

Value:
Type: Boolean
▪ 1: Should re-login
▪ 0: Should not re-login

Default: 1 - Should re-login

LogoutDetection Flag that specifies whether AppSpider should try to detect

whether it lost the session.

Value:
Type: Boolean
▪ 1: Should detect
▪ 0: Should not detect

Default: 1 - Should detect

UserAssistance Reserved for future use

AssumeSuccessfulLogin Flag that defines whether AppSpider should check if the
user was logged in using the regular expression in
parameter LoggedInRegex or it can just assume that the
user was logged in.

Value:
Type: Boolean
▪ 1: Assume that the user was logged in.
▪ 0: Use regular expression to detect whether the
user was logged in,

Default: 0

Remarks:
This parameter is often used in conjunction with macro
login when the user can see in the browser that AppSpider
logged in and does not want to craft a regular expression
that detects a logged in state.

VerifyNotLoggedin This flag defines whether AppSpider should verify that the
session is not logged in before trying to re-login. If the
session was logged in and that flag is set, AppSpider will
not try to re-login.

Value:
Type: Boolean
 ▪ 1: AppSpider will verify whether the session was
logged in.
▪ 0: AppSpider will verify whether the session was
not logged in

Default: 1

Remarks:

If during scanning a false positive logout was detected on

one of the responses , AppSpider will try to re-login into
session that is perfectly valid. This parameter configures
how AppSpider behaves in this situations. If the value of
the parameter is set to ‘1’, AppSpider first check whether
the user is already login, prior to starting login process. If it
is set to ‘0’, AppSpider will reset session

PostponeLoginAction Flag that tells AppSpider whether it should postpone

crawling the link if the is defined in the action attribute of
the login form

Value:
Type: Boolean
▪ 1: AppSpider will postpone crawling of the action
link.
▪ 0: AppSpider will crawl the action link.

Default: 1

Remarks:

CreateNonAuthenticatedSession Flag that determines whether AppSpider should create a

non-authenticated session along with the authenticated
session. This flag should only be set if the user provided
authentication information in the scan configuration: login
macro, username and password for form authentication,
etc.

Value:
Type: Boolean
▪ 1: Create non-authenticated session.
▪ 0: Do not create non-authenticated session

Default: 0

TreatFailedReloginAsError The flag that tells AppSpider what to do when it fails to

re-login the user. If that flag is set, then the scan will stop if
 relogin is failed. If the flag is not set then AppSpider
continues with the scan with the logged out session. Note
that the initial login is always treated as an error.

Value:
Type: Boolean
▪ 1: Consider re-login failure as an error.
▪ 0: Do not treat re-login failure as an error and
continue with the scan

Default: 1

BlacklistSinglePasswordForms This flag determines whether the crawler should send

requests from forms that have one password field.

Value:
Type: Boolean
▪ 1: Do not crawl forms with one password field
▪ 0: Allowed to crawl forms with one password field

Default: 0
BlacklistMultiPasswordForms This flag determines whether the crawler should
sentdrequests from forms that have two password fields.

Value:
Type: Boolean
▪ 1: Do not crawl forms with two password fields
▪ 0: Allowed to crawl forms with two password fields

Default: 1
ResetCookies This flag tells AppSpider whether it should reset all cookies
before every re-login.

Value:
Type: Boolean
▪ 1: Reset all cookies.
▪ 0: Do not reset cookies that were in the session
before re-login

Default: 1
AccountType Deprecated
UsernameForm The user name that will be used for form authentication

Value:
Type: String
Default: None
 Remarks:
This parameter is only used if parameter Type is set to
‘Form’
PasswordForm The user password that will be used for form
authentication

Value:
Type: String
Default: None

Remarks:
This parameter is only used if parameter Type is set to
‘Form’
UsernameHttp The user name that will be used for HTTP authentication
(Basic, NTLM or Kerberos)

Value:
Type: String
Default: None

Remarks:
Note that for NTLM authentication with domain, the
format of username should be <domain>/<username>
PasswordHttp The user password that will be used for HTTP
authentication (Basic, NTLM or Kerberos)

Value:
Type: String
Default: None

Remarks:
AutoLogonSecurity This parameter defines the scope for which AppSpider
should use Windows user identity for Integrated Windows
Authentication.

Value:
Type: Enum
Enum Values:
● AutoLogonSecurityLow (numeric: 0): ​ An
authenticated log on using the default credentials
is performed for all requests
● AutoLogonSecurityMedium (numeric: 1): ​ An
authenticated log on using the default credentials
is performed only for requests on the local Intranet
● AutoLogonSecurityHigh (numeric: 2): ​ Default
credentials are not used. Note that this flag takes
 effect only if you specify the server by the actual
machine name. It will not take effect, if you specify
the server by "localhost" or IP address.

Default: AutoLogonSecurityMedium

Remarks:
LoginLinkRegex Defines the regular expression that AppSpider uses to
determine whether a link is a login link (link used in login
process)

Value:
Type: String
Default: ((log|sign)[ -]?(in|on))|auth

Remarks:
LoggedInRegex Defines the regular expression that AppSpider uses to
determine whether the user was logged in as a result of
login macro execution or login form submission or any
other type of supported authentication

Value:
Type: String
Default: (sign|log)[ -]?(out|off)

Remarks:
SessionLossRegex Defines the regular expression that AppSpider uses to
determine whether the user was logged out. This regex is
only applied to HTTP response body

Value:
Type: String
Default: please (re)?login|have been logged out|session
has expired

Remarks:
AppSpider applies that regex to all responses (as opposed
to regular expression in SessionLossOnCanaryPageRegex)
SessionLossHeaderRegex Defines the regular expression that AppSpider uses to
determine whether the user was logged out. This regex is
only applied to HTTP headers

Value:
Type: String
Default: Location:
[^\\n]{0,100}((sign|log)(in|on|out)|unauthenticated)\\b
 Remarks:
LogoutLinkRegex Defines the regular expression that AppSpider uses to
determine whether a link is a logout link. This helps
AppSpider to stay logged in by not clicking on or requesting
logout links

Value:
Type: String
Default: (sign|log|time)[ -]?(in|on|out|off)|password

Remarks:
LogoutPostBodyRegex Defines the regular expression that AppSpider uses to
determine whether a request with POST data can cause
session logout. This helps AppSpider to stay logged in by
not clicking on or requesting logout links

Value:
Type: String
Default: (sign|log|time)[ -]?(in|on|out|off)

Remarks:
CanaryPage Defines the URL that AppSpider will periodically request to
determine whether the session was lost.

Value:
Type: String
Default: None

Remarks:

Note that this parameter should be used in conjunction

with parameter SessionLossOnCanaryPageRegex
SessionLossOnCanaryPageRegex Defines the regular expression that AppSpider uses to
determine whether a request with POST data can cause
session logout. This helps AppSpider to stay logged in by
not clicking on or requesting logout links

Value:
Type: String
Default: None

Remarks:
Note that this parameter should be used in conjunction
with parameter CanaryPage
FormSubmissionScript Reserved for future
SessionCookieRegex This parameter contains the regular expression that
AppSpider uses to determine whether a cookie is a session
cookie. The regular expression is applied to the cookie’s
name only.

Value:
Type: String
Default:
\\b(CFID|CFTOKEN|SESSION|JSESSIONID|ASPSESSIONID[A-
Z0-9]+|PHPSESSID|ASP[.]NET_SessionId)\\b

SessionCookieLifespan This parameter determines the maximum lifespan of the

cookie below which the cookie is considered a session
cookie.

Value:
Type: Number (Number of days)
Default: 32

LogoutDetectionFrequency Deprecated
DiscoveryMaxLinks This parameter defines maximum number links that the
login component can crawl in search for a login form.

Value:
Type: Number
Default: 200

LoginMaxLinks This parameter defines the maximum number of links that

the login component can crawl after submitting alogin
form while it is looking for the page that indicates that the
user session was logged in.

Value:
Type: Number
Default: 50
DiscoveryDepth This parameter determines how deep into the web site the
crawler should go in search of the login form. The depth of
a link is the minimum number of links (steps) that the user
should visit to discover this link.

Value:
Type: Number
Default: 10
LoginDepth This parameter determines how deep into the web site the
crawler should go after the submitting login form in search
of the page that can determine a logged in state. The depth
of a link is the minimum number of links (steps) that the
 user should visit to discover this links starting from the
page with the login form

Value:
Type: Number
Default: 10

MaxMacroReloginAttempts Maximum number of times AppSpider should try to

re-login.

Value:
Type: Number
Default: 3

Remarks:

Note that this parameter is not used for initial login, which
is performed only once.
DiscoveryPrioritization This parameter determines the algorithm the login form
discovery crawler should use.

Value:
Type: Enum
Values:
▪ FIFO(numeric: 0)
▪ Smart(numeric: 1)
▪ DirBreadthFirst(numeric: 2)
▪ FoundBreadthFirst(numeric: 3)
▪ FoundDepthFirst(numeric: 4)
▪ Juicy(numeric: 5)
▪ LoginFormDiscovery(numeric: 6)
▪ Login(numeric: 7)

Default: LoginFormDiscovery

Remarks:

It is not recommended to change the value from this

parameter from the one selected by AppSpider by default.

LoginPrioritization This parameter determines the algorithm the crawler

should use after the submission of login form to find the
page would indicate a logged in state. (e.g. “Welcome back
to [Link] Bob”)

Value:
 Type: Enum
Values:
▪ FIFO(numeric: 0)
▪ Smart(numeric: 1)
▪ DirBreadthFirst(numeric: 2)so thi
▪ FoundBreadthFirst(numeric: 3)
▪ FoundDepthFirst(numeric: 4)
▪ Juicy(numeric: 5)
▪ LoginFormDiscovery(numeric: 6)
▪ Login(numeric: 7)

Default: Login

Remarks:

It is not recommended to change the value from this

parameter from the one selected by AppSpider by default.

AuthConfig 's Objects

Property Description
MacroFile Macro file that will be used for authentication. Note that this parameter
is used only if Type value is set to 'Macro'

AuthConfig 's Lists

Property Description
ScopeConstraintList List of scope constraints for the login crawler that determine which part
of the site the login crawler is allowed to crawl. Note that this parameter
is only used when the Type value is set to 'Form'

AnalyzerConfig Structure
AnalyzerConfig 's Scalar Values
Property Description
Enabled Deprecated
 NotExistingFilePath This parameter defines the URL request that AppSpider sends to retrieve
the response that the web server returns for a non-existing file path. The
value of that parameter is appended to the URL of the directory.

Value:
Type: String
Default: /[Link]

NotExistingDirPath This parameter defines the URL request that AppSpider sends to retrieve
the response that the web server returns for a non-existing directory
path. The value of that parameter is appended to the URL of the
directory.

Value:
Type: String
Default: /aaaaaaaa/

AnalyzerConfig 's Objects

Property Description
N/A N/A

AnalyzerConfig 's Lists

Property Description
N/A N/A

AttackerConfig Structure
AttackerConfig 's Scalar Values
Property Description
ParametersToAttackBeforeLimitingAttacks
LinksToAttackBeforeLimitingAttacks
MaxSameNameParameterAttackPoints This parameter determines how many parameter
values that have the same name (query or POST)
AppSpider is going to attack.

Value:
Type: Number
Default: 50
 MaxSameCookieParameterAttackPoints This parameter determines on how many pages a
cookie can be attacked by AppSpider.

Value:
Type: Number
Default: 25
MaxSameNameParameterAttackPointsPer This parameter determines how many parameter
Link values that have the same name (query or POST)
AppSpider is going to attack on links that have the same
URL.

Value:
Type: Number
Default: 3
MaxParameterAttackPointsPerLink This parameter determines how many parameters
AppSpider is going to attack on URLs that have the
same URL.

Value:
Type: Number
Default: 50
MaxNormalizedSameNameParameterAttac This parameter determines how many parameters with
kPointsPerLink the same normalized name AppSpider is going to attack
on links that have the same URL. Normalized name is
the name of the parameter without array index or any
other indexing type.

Value:
Type: Number
Default: 10

Remarks:

For instance if a page has parameter params[1],

params[2], params[3], all those parameters will have
the same normalized name: params[]

AttackerConfig 's Objects

Property Description
N/A N/A
AttackerConfig 's Lists

Property Description
ScopeConstraintList List of scope constraints that determines which URLs
AppSpider can attack. Note that even if the that list is
empty ,AppSpider will not attack URLs that do not comply
with constraints specified for the crawler in
[Link]
DefaultDoNotAttackParamList List of parameter names that AppSpider should not attack.
This list should not be changed by the user. For
convenience, user-defined parameters that should not be
attacked are moved into a separate parameter:
UserDoNotAttackParamList
UserDoNotAttackParamList List of parameter that AppSpider should not attack.