jMeter and script automation

jMeter provides a command line interface that allows to automate performance testing. This blog post demonstrates some useful techniques that take advantage of this interface.

How to run jMeter in command line mode?

Simply add the non-gui switch and specify the benchmark file.

jmeter -n -t benchmark.jmx

How to parameterize benchmarks?

Use the following scheme for setting the values of a benchmark file:

$(__P(paramname))

jmeter_parameters

These values can be parameterized at launch by using the following command line convention:

jmeter -Jusers=100 -Jrampup=5 -Jcount=1000

How to collect statistics?

One convenient way is to set up a Summariser with the following parameters:

jmeter -Jsummariser.name=sum Jsummariser.interval=10 Jsummariser.out=true

The standard output of the jMeter process will look like this:

Creating summariser
Created the tree successfully using benchmark.jmx
Starting the test @ Mon May 13 00:23:27 CEST 2013 (1368397407319)
Waiting for possible shutdown message on port 4445
sum +      1 in   0,1s =   14,5/s Avg:    69 Min:    69 Max:    69 Err:     0 (0,00%) Active: 1 Started: 1 Finished: 0
sum +  23892 in   5,2s = 4574,4/s Avg:     1 Min:     0 Max:     7 Err:     0 (0,00%) Active: 10 Started: 10 Finished: 0
sum =  23893 in   5,3s = 4512,4/s Avg:     1 Min:     0 Max:    69 Err:     0 (0,00%)
sum +  45184 in     6s = 7529,4/s Avg:     1 Min:     0 Max:     6 Err:     0 (0,00%) Active: 10 Started: 10 Finished: 0
sum =  69077 in  11,3s = 6116,3/s Avg:     1 Min:     0 Max:    69 Err:     0 (0,00%)
sum +  45512 in     6s = 7584,1/s Avg:     1 Min:     0 Max:     7 Err:     0 (0,00%) Active: 10 Started: 10 Finished: 0
sum = 114589 in  17,3s = 6626,3/s Avg:     1 Min:     0 Max:    69 Err:     0 (0,00%)

Some notes:

  • The summeriserName + output shows statistics for the current interval
  • The summariserName = output shows total statistics.

Regular expression can be used to parse the valuable information from the output. The following Powershell example does the trick:

cmd.exe /c jmeter -n `
    -t benchmark.jmx `
    -Jsummariser.name=sum `
    -Jsummariser.interval=10 `
    -Jsummariser.out=true | ForEach-Object {
    $line = $_
    # Parse the listening port
    if ($line -match "^Waiting for possible shutdown message on port ([0-9]+)") {
        $jMeterListenerPort = $matches[1];
        return
    }
    # Parse the statistics of the current interval
    if ($line -match "^sum \+ +" + `
                     "([0-9]+) in +([0-9,]+)s = +" + `
                     "([0-9,]+)/s " + `
                     "Avg: +([0-9]+) " + `
                     "Min: +([0-9]+) " + `
                     "Max: +([0-9]+) " + `
                     "Err: +([0-9]+) \(([0-9,]+)%\)") {
        $requests = $matches[1]
        $throughput = $matches[3]
        $latency = $matches[4]
        $errors = $matches[7]
        $errors_percentage = $matches[8]
        return
   }
}

This script parses the standard output of jMeter line by line. The parsed statistics can be collected and evaluated in any way the tester wants. However, keep in mind the variables hold string values, so they have to be parsed into number variables.

How to stop benchmarks on demand?

The script block of the ForEach-Object cmdlet – in the previously presented script – will run until the jMeter process exits. This happens only if the test is finished or shut down. The later can be done in graceful way with scripts too:

cmd.exe /c "@echo | call stoptest $jMeterListenerPort" | Out-Null

Note that this tricky invocation bypasses the pause command in the batch file and ignores the standard output.

Sample script

As a bonus I would like to share a Powershell script that I use to micro benchmark different kind of server solutions. I am not going into the details how it exactly works, but it uses the automation techniques described in this post.

The following sample demonstrates how can it be executed.

$path = "http://localhost:8080"
$users = @(1, 250, 500, 750, 1000, 1250, 1500, 2000, 3000)

.\benchmark.ps1 `
    -TargetAddress $path `
    -JMeterFile "benchmark.jmx" `
    -OutFile "test-out.txt" `
    -Users $users `
    -UserPerSecRampup 50 `
    -Warmup 10 `
    -Duration 30 `
    -Cooldown 0 `
    -Verbose

This will execute the specified jMeter benchmark file multiple times with different size of workload (1, 250, 500…) on the specified target. It collects and aggregates benchmark data from the jMeter summarizer and writes into the specified text file in spreadsheet-friendly format. The data rows contain the total request and error count, as well as the average throughput and latency measured under various workload.

http://localhost:8080
274	63260	120285	167832	202466	205907	195140	202406	193983
0	0	0	0	0	0	0	0	0
9.2	2083.3	3957.87	5530.27	6643.4	6744.7	6378.83	6541.93	6139.53
108	118	124	133.67	147.67	181.67	230	296.67	464

Some script parameters might need a little explanation: The UserPerSecRampup value is used to calculate the ramp up period length. The Warmup, Duration and Cooldown parameters set the duration of periods in which no benchmark data is collected after ramp up, data is collected, the workflow sleeps until the next iteration, respectively. The Verbose switch makes the standard output of the jMeter process appear as verbose output.

workflow

The script has prerequisites too: some settings of the jMeter benchmark file – specified by the JMeterFile parameter – should be set up like below:

Number of Threads (users) ${__P(users)}
Ramp-Up Period (in seconds) ${__P(rampup)}
Loop count Forever
Server Name or IP ${__P(webhost)}
Port Number ${__P(webhostport)}
Path ${__P(webhostpath)}

I hope you will find this script useful and serves as a good starting point for building more complex testing environments. Happy benchmarking!

How to enable remote Powershell?

Interestingly, it was quite confusing to find out how to setup it properly, so I decided to create a blog post about it.

On the server side, open Powershell with elevated privileges (Run as Administrator), an execute this command:

Enable-PsRemoting

Unfortunately, this will not be enough if any of your current network interfaces are not in a private or domain network. For example, if VMWare Workstation is installed on your system, its virtual network interfaces are categorized as members of unidentified networks. To overcome this issue, check the solution introduced by this post.

Now we can move to the client side. In the case of not being joined to a domain network, we have to mark the servers as trusted hosts. To achieve this, open cmd.exe (not Powershell) with elevated privileges and run this command:

winrm set winrm/config/client @{TrustedHosts="server1, server2"}

You can read or write the TrustedHosts property with Powershell too: run the Powershell command prompt with elevated trust and use the Get-Item and Set-Item cmdlets with the appropriate wsman path. The path is autocompleted, so it is much easier to type it here. You can also use the Concatenate switch to simply add a new host. In order to allow the client connect to any server, use the wildcard character.

Get-Item WSMan:\localhost\Client\TrustedHosts | select -ExpandProperty Value
Set-Item WSMan:\localhost\Client\TrustedHosts "server1,server2"
Set-Item WSMan:\localhost\Client\TrustedHosts "server3" -Concatenate -Force
Set-Item WSMan:\localhost\Client\TrustedHosts *

Now we can connect to the server with a Powershell session. On a domain network this might be enough:

Enter-PsSession server1

If you are on a non-domain network or your current user does not have the appropriate rights, you have to specify the appropriate credentials explicitly.

Enter-PsSession server1 -Credential (Get-Credentials)

You can also run multiple sessions simultaneously, and switch between them on demand.

$s = @("server1", "server2") | New-PsSession
Enter-PsSession $s[0]
Exit-PsSession
Enter-PsSession $s[1]
exit

Unfortunately these commands are only for interactive usage, so you are not able to use them in scripts. For remote script execution Invoke-Command is the way to go, but this a much longer story…

Exchange well-known folder hell

During an Exchange Server migration I have encountered an annoying problem. I have exported the mailboxes from an old Exchange 2003 server into PST files. After that, these files were imported into the newly created mailboxes of a new Exchange 2010 server. This process did not go absolutely smoothly, the following problem occurred:

The former figure presents what happens if the language of the new and exported mailbox are not identical (e.g. hu/HU => en/US), the latter shows how the mailbox look like if the the languages are identical (e.g. en/US => en/US).

owa2        owa1

I searched around the internet and I found out that this is a really common phenomenon and is said to be a bug. It seems that the well-known folders are referenced by their localized name instead of an invariant name in the PST files. Furthermore, if the folder already exists, the imported folder will be renamed (number is concatenated). Fortunately, I found a very impressive Powershell script on the Microsoft Technet forum that is meant to solve this issue.

After a bit examination and preparation the script worked flawlessly, it cleaned up the mess in the mailboxes nicely. I really appreciate the author‘s effort and generosity. I also worked on the script a bit too. I think it is much easier to use now.

First of all, the script is based on EWS 1.1, so you have to download and install it on the Exchange Server. (Update: It seems the EWS 1.1 link is dead, try EWS 1.2, it should work too)

Now, create a CSV file that describes the folder mappings:

folder,source
Inbox,"Beérkezett üzenetek"
Inbox,"Beérkezett üzenetek1"
Inbox,Inbox
Inbox,Inbox1
SentItems,"Elküldött üzenetek"
SentItems,"Elküldött üzenetek1"
SentItems,"Sent Items"
SentItems,"Sent Items1"
Notes,Feljegyzések
...

The first element of each row is the invariant name of a well-known folder (my sample file contains all of them), the second is a possible localized name that the script should look for.

In the following example, the script is going to be executed on my mailbox. First, import the Exchange Powershell snapin and the cleanup script.

Add-PSSnapin Microsoft.Exchange.Management.Powershell.E2010
. ./Cleanup-Mailbox.ps1

Query the problematic mailboxes, in this case: mine.

$m = Get-Mailbox tom

The user that executes the script needs Full Access permission on the mailboxes.

$m | Add-MailboxPermission -User Admin -Accessright Fullaccess

Now, run the cleanup script on the mailboxes by simply piping them to the Cleanup-Mailbox function. The Mapping parameter should point to the previously created CSV file. If the Test switch is specified, than the script will only output the diagnostics messages without doing the modifications. I suggest to try it before production use.

$m | Cleanup-Mailbox -Mapping ./folders.csv -Test

That’s it, my mailbox has been cleaned up! It is recommended to remove the Full Access permission.

$m | Remove-MailboxPermission -User Admin -Accessright Fullaccess

You can download the cleanup script from here, and the previously mentioned sample mapping CSV file here.