Correlation: Dynamic data handling using Apache JMeter

In this article, We will talk about how to handle dynamic data in JMeter which is  basically Correlation. Correlation is the important aspect of scripting. Proper handling of dynamic data results in building effective test scripts.

What is Correlation and why is it required ?

Correlation is an act of fetching dynamic data from preceding requests/calls and posting it to the subsequent requests/calls. Lets envision an example to find out why we need correlation. Suppose we have recorded a scenario in which user does the following transactions

• User enters login details and click submit button,
• Home page opens and user take further actions,
• And then Logs out of the system.

Soon after recording if you try to run the script, it may even fail for a single user. This might be because of numerous reasons, lets assume that one of the reason might be authentication mechanism implemented. When a user logged into the system, a unique session is created for user and it will be active until the user logs out of the system. Lets imagine, a unique tokenId is dynamically generated up on user login and is passed to the subsequent requests to validate the user actions. So, one cannot just record and playback the requests having these tokenId’s which are unique for every user sessions. Here, we need to correlate the web requests with the dynamic variables (tokenId). And for correlation, we need to use the “Regular Expression Extractor” which makes use of regular expressions or most commonly known as RegEx.

Examples of Dynamic data in the response body

Server sends unique tokenID in the body of a response, which is then used by the browser when it submits a form in the next request. For instance below is the example:

When you work with Microsoft® ASP.NET web applications, the most commonly seen dynamic value in the response body is __VIEWSTATE. Viewstate is an ASP.NET page-level state management mechanism. It stands for the method which is being used to preserve page and controls between round-trips by holding the state of an ASP web form across postbacks. In short, the current state of the page, controls, values, etc. is being converted to a Base64-encoded string and is bi-directionally passed between client and server.

value=”Xobex3z0jnc29axbnsFZbbRnLLUeZw0Nnycn50qN7pafSBsEe2xG8FRGPdzVi6KNNfCLm7V/FGkJiDbFeopvkBNXXHx/gwJs1UONXqQm/YhTNraTb2B0fKzfbHcJ/2lTco+jQ/fPbNr6rPWrg+DGC5DohH1MdXb9Rtw9LDzJ0SUPC5B7kf+6uswY3jkrQPKYnp9hrhdqwvygjMe55Df0t6Um9voXv/vRR8HK60EZQQ8tYcG0Qnot/fYZw+Kt9lkY47mEZjL9YXoJDgQlNHK2uFBzCKB+L3CU1v7TanITLqWrOf7n6nujpUiQ5J0hbY9iPQpyvsyntA/cHGYr3vjF2OxmvurWAoZFA4f0r1Y2Ig0X16bz6OFJnY5IuJrzJuKTnGHHlTMyRkbbtSqnDN2yxYttHOQfrgXe5Y8pRK05vEHLtk8wBsZHl9xxzRJcBt6GzqUJAF651pJ8WTL0r7dcB+HwCIu5roI89rxbRJhCalaDd5WwuJBR4XnqrSL+1ntHDz4JXmZX3J+uH1Z0yMMqHoN9lwc4qsduHqyB5IkMPTQtH7R7RLf3y+0JrfvE48s10Jo2WZ5X6kc3QM2jbBzG2VR1Fbnn9ZN9IV5nbN7Jc4+UQ3O8PuqpY+vG6hLdWsZOzo6FXAVa5ibL57KW7pPcUDzO+Zzi196o0WTz79HVUf2eQVK9uZEX4kWHOJcNmUcd8kyTU+XcIFNa/5BiI8+cbiSP6ssdCYHsqumKevMotKFHdLRwY2OVijtKBrJFSDhKbtBPP3RM8zzc2KtY11+PKQGN58=” />

How do you know when to correlate or handle dynamic data?

If you are experienced tester you will always have an eye for this and also by close monitoring of the network activity using some network debugging tools, it will be easier to identify the dynamic data such as tokenID, Viewstate, JSESSIONID etc which can be handled using regular expressions.

For starters, it advisable to record your transactions twice with same users and use any file monitoring tools to detect the differences. If can’t make out the differences, any seniors officials help comes to your aid.

Correlation using JMeter

Using the above tokenID example, lets understand in detail how correlation is done using JMeter. Lets envision a scenario where a unique tokenID is dynamically created and passed to the subsequent requests to validate the user actions that are performed. Imagine that our thread group contains two http request samplers i.e Login Page and Home Page. Login Page response body contains a tokenId which has to be correlated and  posted it to the Home Page. The set up can be as follows:

Now the first step is, Add a Regular expression extractor to Login Page request as the we know that the response body contains the tokenId. Right click on the Login Page sampler then Add -> Post Processors -> Select Regular Expression Extractor. 

Now, the extractor has to be configure to extract the dynamic value as shown below:

Reference Name:The name of the JMeter variable in which to store the result. Also note that each group is stored as [refname]_g#, where [refname] is the string you entered as the reference name, and # is the group number, where group 0 is the entire match, group 1 is the match from the first set of parentheses, etc. From the above configuration, the result is stored in “tokenId”.

Regular Expression: The regular expression used to parse the response data. This must contain at least one set of parentheses “()” to capture a portion of the string, unless using the group $0$. Do not enclose the expression in / / – unless of course you want to match these characters as well. From the above configuration, the value is extracted using this expression name=”token” value=”([a-zA-Z0-9]+)“. For more details on regular expressions refer here.

Template: The template used to create a string from the matches found. This is an arbitrary string with special elements to refer to groups within the regular expression. The syntax to refer to a group is: ‘$1$’ to refer to group 1, ‘$2$’ to refer to group 2, etc. $0$ refers to whatever the entire expression matches. Since the template value is set to $1$ in the above configuration, it is clearly defined to refer group 1.

Match No: Indicates which match to use. The regular expression may match multiple times.

• Use a value of zero to indicate JMeter should choose a match at random.
• A positive number N means to select the nth match.
• Negative numbers are used in conjunction with the ForEach controller

Since the the MatchNo is set to 1, we are instructing JMeter to match the first occurrence.

Default Value: If the regular expression does not match, then the reference variable will be set to the default value. This is particularly useful for debugging tests. If no default is provided, then it is difficult to tell whether the regular expression did not match, or the RE element was not processed or maybe the wrong variable is being used.

Another important note can be, we should be very clear about the section that we are extracting the value from. If the value is extracted from response body of the sampler then under response file check configuration body option has to be selected. If the value is extracted from the headers of the sampler then Response Headers option has to be selected and so on.

We can also define to which sample the regular expression extractor can apply to, If the want the extractor to check only the main sample then under Apply to configuration select Main Sample only option and so on.

Now pass ${tokenId} variable to the Home Page sampler by overriding the already present hard coded value of tokenId variable as shown below:

Save and run the test plan again check if the extracted value is being passed correctly to the subsequent request. All the request are marked green ? that’s it you have successfully handled the dynamic values using regular expression extractor and regular expressions.

Variations of Regular expression extractor

Some time we end up extracting multiple values from the response. For instance the ID and Name attributes associated with a particular class as shown below:

Our goal here is extract both id and name from the above response body. As stated before, add the regular expression extractor to the sampler where you want to extract data as shown below:

Have you noticed the regular expression in the above configuration, there are two sets of () in the regular expression which is used to match both the id and name attributes from the response of the sampler. The above regular expression will yield the following results:

colors=blu BLUE
colors_g=2
colors_g0=class="colors" id="blu" name="BLUE"
colors_g1=blu
colors_g2=BLUE

Lets understand what exactly the above results define, the results of the expression is stored in a JMeter variable ${colors}. If you observe the template we used was the first group $1$ is immediately followed by $2$ the second group, so in effect:

${colors} now equals blu BLUE

JMeter also breaks the expression up into other variables which is handy when we only want parts of the matched expression. For example ${colors_g1} says the 1st group equals blu and likewise ${colors_g2} equals the 2nd group BLUE. Indeed we can also get the entire matched string from the regular expression via $colors_g0.

What happens if we wanted all the matches on a page? This is where Match No. comes into play. In previous examples we used the 1st match found on the page.

Match No. is not a zero based index!
If you specify 0 then you will get a random match from the page.

The Match No. value should be set to negative number (-1) if you want all the matches on the page. Lets take the above example to under in detail. The regular expression extractor should be like this:

The above regular expression will yield the following results:

colors=BLACK
colors_1=blu
colors_1_g=2
colors_1_g0=class="colors" id="blu" name="BLUE"
colors_1_g1=blu
colors_1_g2=BLUE
colors_2=blk
colors_2_g=2
colors_2_g0=class="colors" id="blk" name="BLACK"
colors_2_g1=blk
colors_2_g2=BLACK
colors_3=red
colors_3_g=2
colors_3_g0=class="colors" id="red" name="RED"
colors_3_g1=red
colors_3_g2=RED
colors_matchNr=3

JMeter creates a variable called refName_MatchNr during runtime which basically returns the total numbers of matches in the response or page. In our case the variable can be ${colors_MatchNr} which is equal to 3. For example if we wanted to loop through all the matches in the response and include them in the next request, we could do something like this using a ForEach controller. or also you can use Beanshell Pre or post processor depending on your requirement.

The following BeanShell script executes a basic for loop, from 1 up to colors_MatchNr which equals 3, and for each iteration of the loop, adds a HTTP request parameter as follows:

int count = Integer.parseInt(vars.get("colors_matchNr"));
for(int i=1; i<=count; i++){ 
id = vars.get("colors_"+i+"_g1");
value=vars.get("colors_"+i+"_g2");
log.info("ID:"+id);
log.info("value:"+value);
}

This is pretty much all we have on correlation right now. If you enjoy reading this article then you can subscribe our updates for FREE, just add your email id . I will keep on updating the article for latest testing information. Subscribe and stay tuned for updates, there’s lot more to come.

🙂 Happy Performance Testing !!  🙂