Correlation: Dynamic data handling using Apache JMeter

In this article, We will talk about how to handle dynamic data in JMeter which is  basically Correlation. Correlation is the important aspect of scripting. Proper handling of dynamic data results in building effective test scripts.

What is Correlation and why is it required ?

Correlation is an act of fetching dynamic data from preceding requests/calls and posting it to the subsequent requests/calls. Lets envision an example to find out why we need correlation. Suppose we have recorded a scenario in which user does the following transactions

  • User enters login details and click submit button,
  • Home page opens and user take further actions,
  • And then Logs out of the system.

Soon after recording if you try to run the script, it may even fail for a single user. This might be because of numerous reasons, lets assume that one of the reason might be authentication mechanism implemented. When a user logged into the system, a unique session is created for user and it will be active until the user logs out of the system. Lets imagine, a unique tokenId is dynamically generated up on user login and is passed to the subsequent requests to validate the user actions. So, one cannot just record and playback the requests having these tokenId’s which are unique for every user sessions. Here, we need to correlate the web requests with the dynamic variables (tokenId). And for correlation, we need to use the “Regular Expression Extractor” which makes use of regular expressions or most commonly known as RegEx.

Examples of Dynamic data in the response body

Server sends unique tokenID in the body of a response, which is then used by the browser when it submits a form in the next request. For instance below is the example:

<input name="token" value="a14f956c63e19d35c2272d5d1e1a15cff1ba1f01"/>

When you work with Microsoft® ASP.NET web applications, the most commonly seen dynamic value in the response body is __VIEWSTATE. Viewstate is an ASP.NET page-level state management mechanism. It stands for the method which is being used to preserve page and controls between round-trips by holding the state of an ASP web form across postbacks. In short, the current state of the page, controls, values, etc. is being converted to a Base64-encoded string and is bi-directionally passed between client and server.

<input type=”hidden” name=”__VIEWSTATE” id=”__VIEWSTATE”

value=”Xobex3z0jnc29axbnsFZbbRnLLUeZw0Nnycn50qN7pafSBsEe2xG8FRGPdzVi6KNNfCLm7V/FGkJiDbFeopvkBNXXHx/gwJs1UONXqQm/YhTNraTb2B0fKzfbHcJ/2lTco+jQ/fPbNr6rPWrg+DGC5DohH1MdXb9Rtw9LDzJ0SUPC5B7kf+6uswY3jkrQPKYnp9hrhdqwvygjMe55Df0t6Um9voXv/vRR8HK60EZQQ8tYcG0Qnot/fYZw+Kt9lkY47mEZjL9YXoJDgQlNHK2uFBzCKB+L3CU1v7TanITLqWrOf7n6nujpUiQ5J0hbY9iPQpyvsyntA/cHGYr3vjF2OxmvurWAoZFA4f0r1Y2Ig0X16bz6OFJnY5IuJrzJuKTnGHHlTMyRkbbtSqnDN2yxYttHOQfrgXe5Y8pRK05vEHLtk8wBsZHl9xxzRJcBt6GzqUJAF651pJ8WTL0r7dcB+HwCIu5roI89rxbRJhCalaDd5WwuJBR4XnqrSL+1ntHDz4JXmZX3J+uH1Z0yMMqHoN9lwc4qsduHqyB5IkMPTQtH7R7RLf3y+0JrfvE48s10Jo2WZ5X6kc3QM2jbBzG2VR1Fbnn9ZN9IV5nbN7Jc4+UQ3O8PuqpY+vG6hLdWsZOzo6FXAVa5ibL57KW7pPcUDzO+Zzi196o0WTz79HVUf2eQVK9uZEX4kWHOJcNmUcd8kyTU+XcIFNa/5BiI8+cbiSP6ssdCYHsqumKevMotKFHdLRwY2OVijtKBrJFSDhKbtBPP3RM8zzc2KtY11+PKQGN58=” />

How do you know when to correlate or handle dynamic data?

If you are experienced tester you will always have an eye for this and also by close monitoring of the network activity using some network debugging tools, it will be easier to identify the dynamic data such as tokenID, Viewstate, JSESSIONID etc which can be handled using regular expressions.

For starters, it advisable to record your transactions twice with same users and use any file monitoring tools to detect the differences. If can’t make out the differences, any seniors officials help comes to your aid.

Correlation using JMeter

Using the above tokenID example, lets understand in detail how correlation is done using JMeter. Lets envision a scenario where a unique tokenID is dynamically created and passed to the subsequent requests to validate the user actions that are performed. Imagine that our thread group contains two http request samplers i.e Login Page and Home Page. Login Page response body contains a tokenId which has to be correlated and  posted it to the Home Page. The set up can be as follows:

re1

Now the first step is, Add a Regular expression extractor to Login Page request as the we know that the response body contains the tokenId. Right click on the Login Page sampler then Add -> Post Processors -> Select Regular Expression Extractor. 

ExpReg

Now, the extractor has to be configure to extract the dynamic value as shown below:

extractor

Reference Name:The name of the JMeter variable in which to store the result. Also note that each group is stored as [refname]_g#, where [refname] is the string you entered as the reference name, and # is the group number, where group 0 is the entire match, group 1 is the match from the first set of parentheses, etc. From the above configuration, the result is stored in “tokenId”.

Regular Expression: The regular expression used to parse the response data. This must contain at least one set of parentheses “()” to capture a portion of the string, unless using the group $0$. Do not enclose the expression in / / – unless of course you want to match these characters as well. From the above configuration, the value is extracted using this expression name=”token” value=”([a-zA-Z0-9]+)“. For more details on regular expressions refer here.

Template: The template used to create a string from the matches found. This is an arbitrary string with special elements to refer to groups within the regular expression. The syntax to refer to a group is: ‘$1$’ to refer to group 1, ‘$2$’ to refer to group 2, etc. $0$ refers to whatever the entire expression matches. Since the template value is set to $1$ in the above configuration, it is clearly defined to refer group 1.

Match No: Indicates which match to use. The regular expression may match multiple times.

  • Use a value of zero to indicate JMeter should choose a match at random.
  • A positive number N means to select the nth match.
  • Negative numbers are used in conjunction with the ForEach controller

Since the the MatchNo is set to 1, we are instructing JMeter to match the first occurrence.

Default Value: If the regular expression does not match, then the reference variable will be set to the default value. This is particularly useful for debugging tests. If no default is provided, then it is difficult to tell whether the regular expression did not match, or the RE element was not processed or maybe the wrong variable is being used.

Another important note can be, we should be very clear about the section that we are extracting the value from. If the value is extracted from response body of the sampler then under response file check configuration body option has to be selected. If the value is extracted from the headers of the sampler then Response Headers option has to be selected and so on.

We can also define to which sample the regular expression extractor can apply to, If the want the extractor to check only the main sample then under Apply to configuration select Main Sample only option and so on.

Now pass ${tokenId} variable to the Home Page sampler by overriding the already present hard coded value of tokenId variable as shown below:

val

Save and run the test plan again check if the extracted value is being passed correctly to the subsequent request. All the request are marked green ? that’s it you have successfully handled the dynamic values using regular expression extractor and regular expressions.

Variations of Regular expression extractor

Some time we end up extracting multiple values from the response. For instance the ID and Name attributes associated with a particular class as shown below:

<input class="colors" id="blu" name="BLUE">
<input class="colors" id="blk" name="BLACK">
<input class="colors" id="red" name="RED">

Our goal here is extract both id and name from the above response body. As stated before, add the regular expression extractor to the sampler where you want to extract data as shown below:

exp

Have you noticed the regular expression in the above configuration, there are two sets of () in the regular expression which is used to match both the id and name attributes from the response of the sampler. The above regular expression will yield the following results:

colors=blu BLUE
colors_g=2
colors_g0=class="colors" id="blu" name="BLUE"
colors_g1=blu
colors_g2=BLUE

Lets understand what exactly the above results define, the results of the expression is stored in a JMeter variable ${colors}. If you observe the template we used was the first group $1$ is immediately followed by $2$ the second group, so in effect:

${colors} now equals blu BLUE

JMeter also breaks the expression up into other variables which is handy when we only want parts of the matched expression. For example ${colors_g1} says the 1st group equals blu and likewise ${colors_g2} equals the 2nd group BLUE. Indeed we can also get the entire matched string from the regular expression via $colors_g0.

What happens if we wanted all the matches on a page? This is where Match No. comes into play. In previous examples we used the 1st match found on the page.

Match No. is not a zero based index!
If you specify 0 then you will get a random match from the page.

The Match No. value should be set to negative number (-1) if you want all the matches on the page. Lets take the above example to under in detail. The regular expression extractor should be like this:

exp

The above regular expression will yield the following results:

colors=BLACK
colors_1=blu
colors_1_g=2
colors_1_g0=class="colors" id="blu" name="BLUE"
colors_1_g1=blu
colors_1_g2=BLUE
colors_2=blk
colors_2_g=2
colors_2_g0=class="colors" id="blk" name="BLACK"
colors_2_g1=blk
colors_2_g2=BLACK
colors_3=red
colors_3_g=2
colors_3_g0=class="colors" id="red" name="RED"
colors_3_g1=red
colors_3_g2=RED
colors_matchNr=3

JMeter creates a variable called refName_MatchNr during runtime which basically returns the total numbers of matches in the response or page. In our case the variable can be ${colors_MatchNr} which is equal to 3. For example if we wanted to loop through all the matches in the response and include them in the next request, we could do something like this using a ForEach controller. or also you can use Beanshell Pre or post processor depending on your requirement.

The following BeanShell script executes a basic for loop, from 1 up to colors_MatchNr which equals 3, and for each iteration of the loop, adds a HTTP request parameter as follows:

int count = Integer.parseInt(vars.get("colors_matchNr"));
     for (int i=1; i<=count; i++)
       {
         id = vars.get("colors_" + i + "_g1");
         value = vars.get("colors_" + i + "_g2");
         log.info("ID:"+id);
         log.info("value:"+value);
       }

This is pretty much all we have on correlation right now. If you enjoy reading this article then you can subscribe our updates for FREE, just add your email id . I will keep on updating the article for latest testing information. Subscribe and stay tuned for updates, there’s lot more to come.

🙂 Happy Performance Testing !!  🙂

Advertisements

30 thoughts on “Correlation: Dynamic data handling using Apache JMeter

  1. Hi Gopi,

    Such a nice article and it’s very good to understand about the correlations.

    This one talks about extract the data from the previous request and i have a situation where in each request generates different session id’s and it would be used only for that request. Do you have any idea, how to handle this kind of situation.

    Regards,
    Karthik

    1. Hi Karthik,

      Thanks for visiting my blog 🙂 . Would you to care to explain your problem in detail ? As per my understanding, session id will be unique per each user session. I believe that the ID which you are receiving might be like a tokenID or something to be secure while sending the data to server.

      You can overcome this situation by normal approach i.e extract the data from the previous request and pass it subsequent request. For instance, lets assume that there are 3 requests r1,r2,r3 in your script and assume that each request generates a different id then follow the below approach.

      1. The Id which is generated will be in previous request’s response for sure. So check the request r1 response for such dynamic data. Extract the data and pass it to the request r2.

      2. Now check the request r2 response for such dynamic data. Extract the data and pass it to the request r2 and so on.

      You can reach me on gkp.anandi@gmail.com for any help. Thank you.
      Regards,
      Gopikrishna

      1. Thanks Gopi 🙂 Let me try this approach.

        I have a problem where in CSRF tokens are enabled in the application. So whenever user navigates to subsequent requests, it generates a token id’s and each one is unique.

        The article about the regular expressions is easy to understand and kudos for your job 🙂

  2. Thanks a lot Karthik, Just follow the above mentioned approach. You will be able to over come your problem. If you are facing any difficulties, you know how to approach me 😛 By the way article on regular expressions is one of my favorite. 🙂

  3. Hi Gopi,

    I tried looping all the matches in the response and include them in the next request using for each controller ,But it’s not working
    Below is the pattern i have extracted

    “UserId”:”e7cd6c4f-5c4a-4596-9133-56a9a70aaa6b”,”FirstName”:”Synthesis1″
    “UserId”:”b97dec3-3c52-4f2c-a604-f778a55e8286″,”FirstName”:”Synthesis2″
    Likewise 50 entries are there in response

    Regular expression i have tried was :
    Ref Name: uid
    Exp: “UserID”:”(.+?)”,”FirstName”:”Synthesis(.+?)”
    Template :$1$$2$
    Match No : -1

    My test Plan structure is like

    Thread Group
    Get User Id
    Reg Exp Extractor
    For each Controller

    I am not aware about how to insert input and output variable in for each controller since it is a multipair value

    Could you please help me on the same

    Thanks,
    Sindhu

  4. Hi,

    Thanks for your valuable stuffs,

    I tried looping all the matches in the response and include them in the next request, using a ForEach controller.

    Below listed responses and test plan structure

    Responses : “UserId”:”e7cd6c4f-5c4a-4596-9133-56a9a70aaa6b”: “FirstName”: “Synthesis41”,etc (like n entries r there )

    I just want to extract userid with the names starting with “Synthesis” so i used expression as

    Ref name : uid
    Exp :”UserID”:”(.+?)”,”FirstName”:”Synthesis(.+?)”
    Template : $1$$2$
    Match No : -1 (to extract all the matches)

    Test Plan Structure
    ——
    Thread Group
    HTTP Request
    Reg Exp Extractor to extract user id
    For each controller —- > Hereby i stucked up like the requests placed inside the loop is not getting executed while running it ,I just inserted the input variable as ‘refno_g1’ and output as var1

    Could you please help me regarding this

    Thanks in Advance !

  5. Responses : “UserId”:”e7cd6c4f-5c4a-4596-9133-56a9a70aaa6b”: “FirstName”: “Synthesis41”,etc (like n entries r there )

    I just want to extract userid with the names starting with “Synthesis” so i used expression as

    Ref name : uid
    Exp :”UserID”:”(.+?)”,”FirstName”:”Synthesis(.+?)”
    Template : $1$$2$
    Match No : -1 (to extract all the matches)

    Test Plan Structure
    ——
    Thread Group
    HTTP Request
    Reg Exp Extractor to extract user id
    For each controller —- > Hereby i stucked up like the requests placed inside the loop is not getting executed while running it ,I just inserted the input variable as ‘refno_g1’ and output as var1

    Could you please help me regarding this

    Thanks in Advance !

    1. Hi Sindhu,

      As per my understanding, you are trying to extract list of userid’s and trying to pass the same to subsequent requests. Use the following RegEx configuration

      Ref name : uids
      Exp :”UserID”:”(.+?)”,”FirstName”:”Synthesis.+?”
      Template : $1$
      Match No : -1 (to extract all the matches)

      () purpose is to extract the value and save it the parameter. Your agenda is to extract only userId starting with Synthesis. In that case why do u need to extract ”Synthesis(.+?)” values as well.The above expression will matches,extract list of usersIds and saves in parameter called ${uids}.

      Under the ForEach controller setting, you need to set the input variable prefix and output variable name which will be available on each iteration. All found user id will be named as uids_X, where X – index number of found by RegExp piece of data. That’s why we may set an input prefix as the uids parameter. set the foreach configuration as below:

      Input variable prefix = uids (Ref Name)
      output variable name = uid and tick add “_” before number. Leave all other fields empty.

      Now add all the http samplers inside the forEach controller and replace the occurrences of UserID with ${uid} . The variable ${uid} will contain one of the found UserIds on each iteration. Let me know if this is working for you.

      Thanks for visiting my blog. Please subscribe for future updates. 🙂

      Cheers,
      Gopikrishna

  6. Hi gopi,

    I tried using response assertions with one of my request

    This is my response for one of my request passed : {“ResultCode”:”Success”,”Message”:””,”Payload”:[{“UserID”:”9102d720-9ab6-4d17-a05f-9ea6ff5f0456″,

    My objective is to verify whether the userid with some specific patterns are being displayed , so i tried appending response assertion to my request with the details furnished below

    Apply to : Main Samples only
    Response field to test : Text response
    patterns to test : {“ResultCode”:”Success”,”Message”:null,”Payload”:{“UserProjectID”:”([0-9,a-z,-].*)”

    but it fails out

    could you please help me regarding this

    Thanks in advance,

    Sindhuja

  7. Hi gopi,

    I tried using response assertions with one of my request

    This is my response for one of my request passed : {“ResultCode”:”Success”,”Message”:””,”Payload”:[{“UserID”:”9102d720-9ab6-4d17-a05f-9ea6ff5f0456″,

    My objective is to verify whether the userid with some specific patterns are being displayed , so i tried appending response assertion to my request with the details furnished below

    Apply to : Main Samples only
    Response field to test : Text response
    patterns to test : {“ResultCode”:”Success”,”Message”:null,”Payload”:{“UserProjectID”:”([0-9,a-z,-].*)”

    but it fails out

    could you please help me regarding this

    Thanks in advance,

    Sindhuja

  8. Sorry , wrongly inserted the patterns to test field

    I tried using response assertions with one of my request

    This is my response for one of my request passed : {“ResultCode”:”Success”,”Message”:””,”Payload”:[{“UserID”:”9102d720-9ab6-4d17-a05f-9ea6ff5f0456″,

    My objective is to verify whether the userid with some specific patterns are being displayed , so i tried appending response assertion to my request with the details furnished below

    Apply to : Main Samples only
    Response field to test : Text response
    patterns to test : {“ResultCode”:”Success”,”Message”:null,”Payload”:{“UserID”:”([0-9,a-z,-].*)”

    but it fails out

    could you please help me regarding this

    Thanks in advance,

    Sindhuja

    1. Hi Sindhu,

      I would suggest you to understand more about regular expressions and its usage. The problem is very simple here. Use the below assertion:

      patterns to test : {“ResultCode”:”Success”,”Message”:””,”Payload”:{“UserID”:${uid}”

      I believe that you have already correlated the Userid from response body. You just need to use the same here. If not use the below pattern.

      patterns to test : {“ResultCode”:”Success”,”Message”:””,”Payload”:{“UserID”:[0-9a-z-].+”

      The problem here is ‘()’, you cannot use these parentheses every where as they are used to group the RegEx and extract the certain text from the response. You have used these parentheses in the response assertion which are not present in the actual response body.i believe that’s the issue that you are facing.

      For better understanding of regular expressions follow the below link:

      https://iamgopikrishna.wordpress.com/2014/12/02/using-regular-expressions/

      Regards,
      Gopikrishna

  9. Hello Gopi,
    Such a great technical blog.Really helps a lot to simplify the correlation terminology.I am trying to develop a scalable testing tool and for this i need to understand the correlation part.Once again thanking you and looking forward for more help from your side.

  10. Hi Gopi

    I am stuck in a situation where the Name field contains the dynamic value and that dynamic value contain the fix value 1. Please see the details. I mailed the .jmx file on your email id.

    Name
    username nishant.saxena79@gmail.com true true
    password saxena@007 true true
    Submit false true
    option com_users false true
    task user.login false true
    return aW5kZXgucGhwP0l0ZW1pZD0xMTc= true true
    b5b4429a499c9f208d31756485b4129b 1 false true

    Thanks
    Nishant Saxena

  11. Hi
    I am stuck in middle of Token and Authtoken. My load test scenario is first i need to login into the system. At that time a token is generate. Let’say – {“ModuleCode”:”Widget”,”SubModuleCode”:”CompanyAnnouncement”,”ActionCode”:””,”ReasonCode”:””,”SubscriptionName”:”hdfc”,”Token”:”1e1ce692-63b4-411f-afc6-722ccd5d7e4e”,”TokenValue”:”hdfc”}.

    And then their is panel in system- when i click on that it redirect me to new tab with a new token. Let say – {“Token”:”9bc2fd50-d022-4124-9975-50a68f9e9dff”,”TokenValue”:”hdfc”,”CategoryID”:”2″,”Module”:”EmployeeGeneralInfo”,”Method”:”TimeStampDifference”,”Program”:””,”TransType”:””}

    Again i need to go back previous tab to do logout.

    How can correlate these 2 token?

    Thanks in advance.

    1. Hi,
      The problem here is very simple. Lets assume that the request hierarchy is as below.
      1. Landing Page (where login details are entered) and submit
      2. Home page (where there is panel in system)

      Now Correlation is nothing but extract the value from previous request and pass it to subsequent request. Add Regular expression extractor as stated below.

      1. Landing Page
      1.1 Look for token value in this request response body (either in the body or in the header) and the add Regular Expression extractor with this expression Token”:”(.+?)” and pass the extracted value in the below request i,e home page request.

      Repeat the same procedure for other tokens as well. Please subscribe for further updates

      Regards,
      Gopikrishna

  12. Hi Gopi,

    Nice Blog,

    I am trying to hit a webservice through soap request and am getting response like
    java.io.IOException: Stream closed
    at java.io.BufferedInputStream.getBufIfOpen(Unknown Source)
    at java.io.BufferedInputStream.read(Unknown Source)
    at org.apache.commons.httpclient.ContentLengthInputStream.read(ContentLengthInputStream.java:170)
    at java.io.FilterInputStream.read(Unknown Source)
    at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)
    at java.io.FilterInputStream.read(Unknown Source)
    at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)
    at org.apache.jmeter.protocol.http.sampler.SoapSampler.sample(SoapSampler.java:293)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1146)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1135)
    at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:434)
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
    at java.lang.Thread.run(Unknown Source)

    Could you please help me on this regards
    Thanks Much in Advance
    Sindhuja

  13. Hi All.
    Please help me to get token from this respond body

    I have tried the value is extracted using this expression name=”ecTokenId” value=”([a-zA-Z0-9]+)“
    But it return null

      1. Hi Gopi,

        I am trying to extract the numbers from the below string , i tried with many possible ways but no luck with tat

        “ImageUrl”:”\/%7B636197926870000359%7D\/WebResources\/cpt_SalesOrderLineItem32″,”NodeType”:0,”ParentId”:”ActiveBusiness”,”SiteMapPath”:”SFA|ActiveBusiness|nav_orderlines”,”StripeClass”:null,”StripeUrl”:null,”ToolTip”:”Sales Order Line Items”},{“Action”:{“ActionType”:”NavigateToPageAction”,”EntityName”:”cpt_consumption”,”EntityTypeCode”:10125,”PageUrl”:”\/_root\/homepage.aspx?etc=10125″,”QuickCreateTimestamp”:null},

        Just want to extract the string ‘%7B636202209280000359%7D’ from the above line

        Could you please help me on this

        Thanks in Advance

        Sindhuja

      2. Hi Sindhuja,
        You can use the below Regex to extract the value

        ImageURL”:”\\/(.+?)\\/

        But if the response body has multiple such occurrences, you might have to handle that situation.

  14. Hi Gopi,

    Thanks for the wonderful post. It is really helpful. Though I have a question. In the example which you have mentioned, there were only2 requests and it was obvious that correlated data will come from the previous requests. But what if we have a large number of requests. Then how should be we decide from which request we should extract the value? I am not able to see the response for each request. Is there any way for this, as it can be done in LR.

    Thanks much,
    Sumi

    1. Hi Sumi,

      In most of the cases, you can extract the dynamic value from previous request only. You have mentioned that, you are unable to see the response for requests, Check if the requests are redirected to some other target.

  15. Hi Gopi,

    I am trying to extract the token number from link below that generates dynamically when move on to checkout page from its previous pages.

    I need to pass this last “359028′ number in HTTP:header manager.
    Regular expression i used is.. :<link rel="(.*?)" href="(.*?)" but it's not working .

    Could you please help me how to handle this..

    Thankq

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s