jMeter and script automation

jMeter provides a command line interface that allows to automate performance testing. This blog post demonstrates some useful techniques that take advantage of this interface.

How to run jMeter in command line mode?

Simply add the non-gui switch and specify the benchmark file.

jmeter -n -t benchmark.jmx

How to parameterize benchmarks?

Use the following scheme for setting the values of a benchmark file:

$(__P(paramname))

jmeter_parameters

These values can be parameterized at launch by using the following command line convention:

jmeter -Jusers=100 -Jrampup=5 -Jcount=1000

How to collect statistics?

One convenient way is to set up a Summariser with the following parameters:

jmeter -Jsummariser.name=sum Jsummariser.interval=10 Jsummariser.out=true

The standard output of the jMeter process will look like this:

Creating summariser
Created the tree successfully using benchmark.jmx
Starting the test @ Mon May 13 00:23:27 CEST 2013 (1368397407319)
Waiting for possible shutdown message on port 4445
sum +      1 in   0,1s =   14,5/s Avg:    69 Min:    69 Max:    69 Err:     0 (0,00%) Active: 1 Started: 1 Finished: 0
sum +  23892 in   5,2s = 4574,4/s Avg:     1 Min:     0 Max:     7 Err:     0 (0,00%) Active: 10 Started: 10 Finished: 0
sum =  23893 in   5,3s = 4512,4/s Avg:     1 Min:     0 Max:    69 Err:     0 (0,00%)
sum +  45184 in     6s = 7529,4/s Avg:     1 Min:     0 Max:     6 Err:     0 (0,00%) Active: 10 Started: 10 Finished: 0
sum =  69077 in  11,3s = 6116,3/s Avg:     1 Min:     0 Max:    69 Err:     0 (0,00%)
sum +  45512 in     6s = 7584,1/s Avg:     1 Min:     0 Max:     7 Err:     0 (0,00%) Active: 10 Started: 10 Finished: 0
sum = 114589 in  17,3s = 6626,3/s Avg:     1 Min:     0 Max:    69 Err:     0 (0,00%)

Some notes:

  • The summeriserName + output shows statistics for the current interval
  • The summariserName = output shows total statistics.

Regular expression can be used to parse the valuable information from the output. The following Powershell example does the trick:

cmd.exe /c jmeter -n `
    -t benchmark.jmx `
    -Jsummariser.name=sum `
    -Jsummariser.interval=10 `
    -Jsummariser.out=true | ForEach-Object {
    $line = $_
    # Parse the listening port
    if ($line -match "^Waiting for possible shutdown message on port ([0-9]+)") {
        $jMeterListenerPort = $matches[1];
        return
    }
    # Parse the statistics of the current interval
    if ($line -match "^sum \+ +" + `
                     "([0-9]+) in +([0-9,]+)s = +" + `
                     "([0-9,]+)/s " + `
                     "Avg: +([0-9]+) " + `
                     "Min: +([0-9]+) " + `
                     "Max: +([0-9]+) " + `
                     "Err: +([0-9]+) \(([0-9,]+)%\)") {
        $requests = $matches[1]
        $throughput = $matches[3]
        $latency = $matches[4]
        $errors = $matches[7]
        $errors_percentage = $matches[8]
        return
   }
}

This script parses the standard output of jMeter line by line. The parsed statistics can be collected and evaluated in any way the tester wants. However, keep in mind the variables hold string values, so they have to be parsed into number variables.

How to stop benchmarks on demand?

The script block of the ForEach-Object cmdlet – in the previously presented script – will run until the jMeter process exits. This happens only if the test is finished or shut down. The later can be done in graceful way with scripts too:

cmd.exe /c "@echo | call stoptest $jMeterListenerPort" | Out-Null

Note that this tricky invocation bypasses the pause command in the batch file and ignores the standard output.

Sample script

As a bonus I would like to share a Powershell script that I use to micro benchmark different kind of server solutions. I am not going into the details how it exactly works, but it uses the automation techniques described in this post.

The following sample demonstrates how can it be executed.

$path = "http://localhost:8080"
$users = @(1, 250, 500, 750, 1000, 1250, 1500, 2000, 3000)

.\benchmark.ps1 `
    -TargetAddress $path `
    -JMeterFile "benchmark.jmx" `
    -OutFile "test-out.txt" `
    -Users $users `
    -UserPerSecRampup 50 `
    -Warmup 10 `
    -Duration 30 `
    -Cooldown 0 `
    -Verbose

This will execute the specified jMeter benchmark file multiple times with different size of workload (1, 250, 500…) on the specified target. It collects and aggregates benchmark data from the jMeter summarizer and writes into the specified text file in spreadsheet-friendly format. The data rows contain the total request and error count, as well as the average throughput and latency measured under various workload.

http://localhost:8080
274	63260	120285	167832	202466	205907	195140	202406	193983
0	0	0	0	0	0	0	0	0
9.2	2083.3	3957.87	5530.27	6643.4	6744.7	6378.83	6541.93	6139.53
108	118	124	133.67	147.67	181.67	230	296.67	464

Some script parameters might need a little explanation: The UserPerSecRampup value is used to calculate the ramp up period length. The Warmup, Duration and Cooldown parameters set the duration of periods in which no benchmark data is collected after ramp up, data is collected, the workflow sleeps until the next iteration, respectively. The Verbose switch makes the standard output of the jMeter process appear as verbose output.

workflow

The script has prerequisites too: some settings of the jMeter benchmark file – specified by the JMeterFile parameter – should be set up like below:

Number of Threads (users) ${__P(users)}
Ramp-Up Period (in seconds) ${__P(rampup)}
Loop count Forever
Server Name or IP ${__P(webhost)}
Port Number ${__P(webhostport)}
Path ${__P(webhostpath)}

I hope you will find this script useful and serves as a good starting point for building more complex testing environments. Happy benchmarking!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: