Мечтают ли WAF’ы о статанализаторах

Virtual patching (VP) has been one of the most popular trends in application protection in recent years. Implemented at the level of a web application firewall, VP allows protecting web applications against exploitation of previously defined vulnerabilities. (For our purposes, a web application firewall, or WAF, will refer to a dedicated solution operating on a separate node between an external gateway and web server.)

In short, VP works by taking the results of static application security testing (SAST) and using them to create rules for filtering HTTP requests on the WAF. The problem, though, is that SAST and WAFs rely on different application presentation models and different decision-making methods. As a result, none of the currently available solutions do an adequate job of integrating SAST with WAFs. SAST is based on the white-box model, which applies formal approaches to detect vulnerabilities in code. Meanwhile, a WAF perceives an application as a black box, so it uses heuristics for attack detection. This state of affairs makes VP sub-optimal for preventing attacks when the exploitation conditions for a vulnerability go beyond the trivial http_parameter=plain_text_attack_vector.

But what if we could make SAST and a WAF “play nice” with each other? Perhaps we could obtain information about an application’s internal structure via SAST but then make this information available to the WAF. That way we could detect attacks on vulnerabilities in a provable way, instead of by mere guessing.

Splendors and miseries of traditional VP

The traditional approach to automated virtual patching for web applications involves providing the WAF with information about each vulnerability that has been detected with SAST. This information includes:

vulnerability class;
vulnerable entry point to the web application (full or partial URL);
values of additional HTTP request parameters necessary for the attack;
values of the vulnerable parameter constituting the attack vector;
set of characters or words (tokens) whose presence in a vulnerable parameter will lead to exploitation of the vulnerability.

The set of HTTP request parameters and dangerous elements of a vulnerable parameter can be defined both by bruteforcing and by using a generic function (typically based on regular expressions). Let us look at a fragment of code from an ASP.NET page that is vulnerable to XSS attacks:

01  var condition = Request.Params["condition"];
02  var param = Request.Params["param"];
03
04  if (condition == null || param == null)
05  {
06      Response.Write("Wrong parameters!");
07      return;
08  }
09
10  string response;
11  if (condition == "secret")
12  {
13    response = "Parameter value is `" + param + "`";
14  }
15  else
16  {
17    response = "Secret not found!";
18  }
19
20  Response.Write("<b>" + response + "</b>");</source>

By analyzing this attack vector code, we can generate a symbolic formula for the set of attack vector values:

{condition = "secret" ⇒ param ∈ { XSShtml-text }}, where XSShtml-text is the set of possible vectors of an XSS attack in the context of TEXT, as described in the HTML grammar.

This formula may yield both an exploit and a virtual patch. The descriptor of the WAF virtual patch can be used to generate filtering rules to block all HTTP requests capable of exploiting the relevant vulnerability.

Although this approach surely heads off certain attacks, it has some substantial drawbacks:

to demonstrate any given vulnerability, SAST needs to discover just one of the possible attack vectors. But to ensure true elimination of a vulnerability, it is necessary to address all possible attack vectors. Passing such information to the WAF is difficult, because the set of vectors is not only infinite but cannot even be expressed in regular expressions due to the irregularity of attack vector grammars;
the same is true for values of all additional request parameters that are necessary for vulnerability exploitation;
information regarding dangerous elements of a vulnerable parameter becomes useless if an attack vector, between the entry point and vulnerable execution point, undergoes intermediate transformations that change the context of its grammar or even its entire grammar (such as with Base64, URL, or HTML encoding, or string transformations).

Due to these flaws, VP technology—which is designed for piecemeal protection—is incapable of offering protection against all possible attacks on SAST-detected vulnerabilities. Attempts to create such “all-encompassing” traffic filtering rules often lead to blocking of legitimate HTTP requests and disrupt operation of the web application. Let us slightly modify the vulnerable code:

01  var condition = Request.Params["condition"];
02  var param = Request.Params["param"];
03 
04  if (condition == null || param == null)
05  {
06      Response.Write("Wrong parameters!");
07      return;
08  }
09 
10  string response;
11  // CustomDecode implements chain data transformation base64-URL-base64
12  if (CustomDecode(condition).Contains("secret"))
13  {
14      response = "Parameter value is `" + CustomDecode(param) + "`";
15  }
16  else
17  {
18       response = "Secret not found!";
19  }
20 
21  Response.Write(response);

The only difference from the previous example is that both request parameters now undergo a transformation and the condition for the secret parameter is weakened until the sub-string is included back in. The attack vector formula, based on analysis of this new code, is as follows:

(String.Contains (CustomDecode (condition)) ("secret")) ⇒ param ∈ (CustomDecode { XSShtml-text })

The analyzer will derive a formula at the relevant computation flow graph (CompFG) node for the CustomDecode function to describe the Base64—URL—Base64 transformation chain:

(Base64Decode (UrlDecode (Base64Decode argument)))

It is still possible to build an exploit on the basis of such formulas (we have considered this issue in a previous article), but the classical approach to generating virtual patches cannot be applied here for the following reasons:

the vulnerability may be exploited only if the decoded condition parameter of the request contains the “secret” substring (String 12). However, this parameter’s set of values is quite large and expressing this set via regular expressions is infeasible due to the irregularity of decoding functions;
request parameter that is, in fact, an attack vector also is decoded (String 14). Therefore, SAST cannot describe that set of dangerous elements to the WAF.

Since all the problems of traditional VP stem from the inability to interact with an application at the WAF level based on the white-box approach, the obvious solution is to implement this capability and make further improvements so that:

SAST provides the WAF with full information about all transformations to which a vulnerable parameter and variables of attack conditions are subjected, from entry point to vulnerable execution point. This enables the WAF to compute argument values based on the values of the parameters of a given HTTP request;
for attack detection, heuristics are replaced with formal methods that are based on rigorous proof of all statements and describe the exploitation conditions for any particular vulnerability in the most general case, instead of haphazardly describing a limited number of cases.

Thus was born runtime virtual patching.

Runtime virtual patching

Runtime virtual patching (RVP) is based on the computation flow graph model used in PT Application Inspector (PT AI). The model is built using abstract interpretation of an application’s code, expressed in semantics similar to conventional symbolic computations. Nodes of this graph contain generating formulas in the target language. The formulas yield the set of all allowable values associated with all data flows at the relevant execution points:

These flows are called execution point arguments. CompFG is evaluable, and thus able to compute sets of specific values for all arguments at any execution point, based on the values that have been set for input parameters.

RVP occurs in two stages, which correspond to the application lifecycle: Deployment (D) and Run (R):

Deployment stage

Before a new version of an application is deployed, the application is analyzed by PT AI. Three formulas are computed for each CompFG node that describes a vulnerable execution point:

conditions for reaching the vulnerable execution point;
conditions for reaching values of all its arguments;
sets of values of all arguments and corresponding grammars.

All formula sets are grouped by the application entry point to whose control flow the vulnerability relates. The very notion of entry point is specific to each web framework supported by PT AI and is defined in the analyzer’s database.

Then a report containing the list of vulnerabilities and related formulas is extracted in the form of code written in a special language based on S-expression syntax. This language describes CompFG formulas in a form that does not depend on the target language. For instance, the formula describing the value of an argument for the vulnerable point in the above code sample is as follows:

(+ ("Parameter value is ``") (FromBase64Str (UrlDecodeStr (FromBase64Str (GetParameterData (param))))) ("``"))

The formula for reaching the vulnerable point is:

(Contains (FromBase64Str (UrlDecodeStr (FromBase64Str (GetParameterData (condition))))) ("secret"))

The report is then uploaded to PT Application Firewall (PT AF). On the basis of the report, a binary module is generated, which can compute all the formulas contained in the report. For example, the decompiled code for computing the condition for reaching the above-mentioned vulnerable point is as follows:

To make formula computation possible, PT AF must have one of the following:

pre-compute database of all functions that may occur in the report;
an isolated sandbox with runtime environment for the language or platform on which the web application runs (such as CLR, JVM, or PHP, Python, or Ruby interpreter), and libraries used in the application.

The first method ensures maximum speed but requires a huge volume of manual work by the WAF developers to describe the pre-compute database even if the developers restrict the scope to standard library functions). The second method allows computing all the functions that may occur in the report, but increases the time needed to process each HTTP request, because the WAF needs to access the runtime environment to compute each function. The most appropriate solution here would be to use the first approach for the most common functions while using the second approach for the rest.

It is quite possible for a formula to contain a function that the analyzer cannot process (for instance, calling a method that involves a missing project dependency or native code) and/or a function that PT AF is unable to compute (for instance, a function for reading data from external sources or the server environment). Such functions are flagged “unknown” in formulas and processed in a special way as described below.

Run stage

At the run stage, the WAF delegates processing of each HTTP request to the binary module. The module analyzes a request and detects the relevant entry point in the web application. For this point, formulas of all detected vulnerabilities are selected and then computed in a specific way.

First, formulas are computed for both conditions: 1) reaching the vulnerable point and 2) reaching values of all its arguments. In each formula, variables are substituted with values of the relevant request parameters, after which the formula value is computed. If a formula contains expressions that are flagged “unknown”, it is processed as follows:

each “unknown” flag spreads bottom-up through the formula expression tree until a Boolean expression is found;
in the formula, such expressions (“unknown” regions) are substituted with Boolean variables, so the Boolean satisfiability problem is solved;
the assumption formula generates n conditions by substituting possible values of unknown regions from all the solutions found in the previous step;
the value of each formula is computed. If at least one formula is satisfiable, the assumption is deemed satisfiable as well.

If computations show that the assumption is false, then the HTTP request in question cannot lead the application to a vulnerable point even with dangerous values of all request arguments. In this case, RVP simply returns request processing to the WAF’s core module.

If attack conditions are satisfiable, the value of the argument of the vulnerable point is then computed. Algorithms used depend on the vulnerability class to which the analyzed point belongs. Their only similarity is the logic used to process formulas that contain unknown nodes: unlike assumption formulas, argument formulas cannot possibly be computed, which is immediately communicated to the WAF. Then the next vulnerable point is computed. To better flesh this out, we shall now review the most complicated algorithm, which is used for detecting injection attacks.

Detecting injections

Injections include any attacks that target the integrity of text written in a formal language (including HTML, XML, JavaScript, SQL, URLs, and file paths) on the basis of data controlled by the attacker. The attack is carried out by passing specifically formed input data to the application. When this data is “plugged in” to the target text, the boundaries of the token are exceeded and the text now includes syntactic constructions not intended by the application logic.

If a vulnerable point belongs to this attack class, its argument value is determined using incremental computation with abstract interpretation using taint analysis semantics. The idea behind this method is that each expression is computed separately, from bottom to top, while the computation results obtained at each step are additionally marked with taint intervals, given the semantics of each function and rules of traditional taint checking. This makes it possible to pinpoint all fragments that are the result of transformation of input data (tainted fragments).

For instance, for the code above and the following HTTP request parameter ?condition=YzJWamNtVjA%3d&param=UEhOamNtbHdkRDVoYkdWeWRDZ3hLVHd2YzJOeWFYQjBQZyUzRCUzRA%3d%3d, the result of applying this algorithm to the formula of a vulnerable point argument is as follows (tainted arguments are marked in red):

The value is then tokenized in accordance with the grammar of the vulnerable point argument. If any tainted fragment matches more than one token, this is a formal sign of an injection attack (based on the definition of injection given at the beginning of this section).

Once formulas have been computed for all vulnerabilities pertaining to the current entry point, request processing is passed on to the WAF’s core module together with detection results.

RVP advantages and specific features

This approach to application protection based on code analysis has a range of substantial advantages as compared to traditional VP:

the shortcomings of traditional VP are addressed, thanks to the formal approach described above and the ability to take into account any and all intermediate transformations;
the formal approach also completely rules out the possibility of false positives, so long as the formulas do not contain unknown nodes;
there is no adverse impact on web application functionality, because protection is built on the functions of the application, as opposed to simply trying to work around them.

For testing the technology and confirming its effectiveness, we have developed a prototype of an integration module for PT Application Inspector and PT Application Firewall, in the form of a .NET HTTP module for IIS web server. A video of the prototype handling the code example above is on YouTube. Performance tests on around fifteen open-source content management systems (CMSs) have shown great results: the time required for processing HTTP requests with RVP is comparable to the time that it takes to process such requests with traditional (heuristic) WAF methods. The average performance hit for web applications was as follows:

0% for requests that do not lead to a vulnerable point;
6–10% for requests that lead to a vulnerable point and are not an attack (depending on complexity of the grammar of the vulnerable point);
4–7% for requests that lead to a vulnerable point and are an attack.

Despite obvious advantages over traditional VP, RVP still has several conceptual shortcomings:

it is not possible to compute formulas that contain data from external sources absent on the WAF (including file resources, databases, and server environment);
the quality of formulas directly depends on the quality of approximation of code fragments during analysis (including loops, recursion, and calls to external library methods);
to describe semantics of transformation functions for the pre-compute database, some engineering work from the developers is required. The description process is difficult to automate and is prone to human error.

However, we have managed to mitigate these weaknesses by offloading some RVP functions to the application and by applying the technologies that underlie runtime application self-protection (RASP).

Advanced RASP

In essence, the ARASP approach consists of using the application itself to compute the formula fragments that cannot be computed by the RVP. An additional instrumentation module is deployed on the application side in order to integrate detector sensors into the web application. These sensors allow getting the values of any fragments of the formulas computed by the RVP.

The ARASP workflow is effectively an extension of the RVP’s one, and is different from it in the following ways:

In the report exported from PT AI, each expression in a formula is supplemented by additional attribute: its coordinates in the code:

+ ("Parameter value is `")  
  (Default.aspx.cs:36:7:FromBase64Str   
    (Default.aspx.cs:35:13:UrlDecodeStr 
      (Default.aspx.cs:32:11:FromBase64Str   
        (Default.aspx.cs:31:10:GetParameterData
          ("param")))))

The report is used to generate not only a formula computation module, but also an instrumentation module that runs on the application side. This module embeds detector sensors at all execution points in the application that correspond uncertain expressions in the report. The module also inserts breakpoints that hand off control to the RVP before proceeding to the vulnerable execution point:

RVP does not take control when processing an HTTP request, instead allowing the application to process the request up until the breakpoint that precedes the vulnerable execution point (when this point is reached, the RVP already has collected information from all of the detector sensors that have been triggered up to that point).
When the breakpoint is reached, processing of the HTTP request is handed over to the RVP and formulas are computed in a way equivalent to the description in the previous section, with one important difference: if the formula contains an uncertain expression, or an expression that cannot be computed by the RVP (because of references to external data sources or the absence of a necessary transformation function in the knowledge base), then the value of the expression is taken from the pool of information that has been accumulated in the application as of when detector sensors were triggered.
If an attack is detected, processing of the request is stopped (and therefore the application does not reach the vulnerable execution point).
If no attack is detected, processing of the request is returned to the application until the next breakpoint is reached or until processing of the request is completed.

This approach significantly expands upon the abilities of RVP, eliminating the drawbacks of RVP as regards the quality of application protection.

Advantages of ARASP: More than just virtual patching

PT AI can be configured to export formulas for all potentially vulnerable execution points, without detecting vulnerabilities in them, thereby assuring full coverage of all dangerous fragments of application code. This is what in fact makes ARASP a comprehensive solution for application protection. This next-generation WAF takes a white-box approach to the application and uses formal detection methods instead of heuristic ones. Compared to the traditional RASP approach, this solution has several advantages: * negligible performance penalty. Processing of a request by application fragments occurs in parallel with processing of the same request by the WAF module running ARASP: * minimal hit to application stability. Instrumentation is used only for those execution points that are truly needed for computing formulas; * precise (close to 100%) detection of attacks, thanks to use of CompFG model elements and formal methods for operating on these elements.

That it RVP and ARASP represent the most promising way forward in application protection, and will continue to develop them as the main vector for improving integration between PT Application Inspector and PT Application Firewall.