# API Gateway Benchmarking with KrakenD

> Explore the API gateway benchmarking capabilities of KrakenD, measure performance, and optimize your API gateway for high throughput and low latency

Source: https://www.krakend.io/docs/enterprise/benchmarks/api-gateway-benchmark/

We wanted to compare our own product with other similar products in the market. In order to do so we used the same environment and conditions and tested the following products:

- Kong
- Vulcand
- Tyk
- KrakenD

For the benchmarks, we based the tests on the benchmarking project [varnish/api-gateway-benchmarks](https://github.com/varnish/api-gateway-benchmarks).

**At the time of writing, KrakenD does not support auth features, so we just did the benchmark with *test01***

## Hardware

hardware specs 

```
$Model MacBook Pro (MacBookPro11,4) - August 2015
Processor: Intel Core i7 2,2 GHz
```

## Setup

For this test, we stored this configuration at `krakend.json`

```json
  {
    "version": 1,
    "host": [
      "http://webserver:8888"
    ],
    "endpoints": [
      {
        "endpoint": "/test01",
        "method": "GET",
        "concurrent_calls": "1",
        "backend": [
          {
            "url_pattern": "/test01"
          }
        ]
      },
      {
        "endpoint": "/test03",
        "method": "GET",
        "backend": [
          {
            "url_pattern": "/test03"
          }
        ],
        "max_rate": "1000000"
      },
      {
        "endpoint": "/test04",
        "method": "GET",
        "max_rate": "1",
        "backend": [
          {
            "url_pattern": "/test04"
          }
        ]
      }
    ],
    "oauth": {
      "disable": true
    },
    "cache_ttl": "5m",
    "timeout": "5s"
  }
```

And we started the environment as described in the [README](https://github.com/varnish/api-gateway-benchmarks/blob/master/README.md#deployment-example)

Start the environment 

```
$cd deployment/vagrant
$ vagrant up

$ vagrant ssh webserver
[vagrant@webserver ~]$ cd /opt/benchmarks/webservers/dummy-api
[vagrant@webserver dummy-api]$ sudo ./deploy
[vagrant@webserver dummy-api]$ exit

$ vagrant ssh gateway
[vagrant@gateway ~]$ cd /opt/benchmarks/gateways/krakend
[vagrant@gateway krakend]$ sudo ./deploy
[vagrant@gateway krakend]$ exit

$ vagrant ssh consumer
[vagrant@consumer ~]$ cd /opt/benchmarks/consumers/boom
[vagrant@consumer boom]$ sudo ./deploy
[vagrant@consumer boom]$ /usr/local/bin/test00
[vagrant@consumer boom]$ /usr/local/bin/test01
```

## Results

### Test 00

```
[vagrant@consumer ~]$ /usr/local/bin/test00
100000 / 100000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %

Summary:
  Total:  13.2722 secs.
  Slowest:  0.1720 secs.
  Fastest:  0.0002 secs.
  Average:  0.0132 secs.
  Requests/sec: 7534.5524
  Total Data Received:  10800000 bytes.
  Response Size per Request:  108 bytes.

Status code distribution:
  [200] 100000 responses

Response time histogram:
  0.000 [1]     |
  0.017 [72298] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.035 [26760] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.052 [522]   |
  0.069 [227]   |
  0.086 [93]    |
  0.103 [28]    |
  0.120 [4]     |
  0.138 [10]    |
  0.155 [24]    |
  0.172 [33]    |

Latency distribution:
  10% in 0.0039 secs.
  25% in 0.0076 secs.
  50% in 0.0129 secs.
  75% in 0.0180 secs.
  90% in 0.0218 secs.
  95% in 0.0233 secs.
  99% in 0.0333 secs.
```

### Test 01

#### KrakenD

```
[vagrant@consumer ~]$ /usr/local/bin/test01
100000 / 100000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %

Summary:
  Total:  28.7424 secs.
  Slowest:  0.2781 secs.
  Fastest:  0.0009 secs.
  Average:  0.0287 secs.
  Requests/sec: 3479.1863
  Total Data Received:  10900000 bytes.
  Response Size per Request:  109 bytes.

Status code distribution:
  [200] 100000 responses

Response time histogram:
  0.001 [1]     |
  0.029 [71546] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.056 [26536] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.084 [1061]  |
  0.112 [392]   |
  0.140 [248]   |
  0.167 [93]    |
  0.195 [51]    |
  0.223 [12]    |
  0.250 [27]    |
  0.278 [33]    |

Latency distribution:
  10% in 0.0213 secs.
  25% in 0.0231 secs.
  50% in 0.0245 secs.
  75% in 0.0295 secs.
  90% in 0.0432 secs.
  95% in 0.0469 secs.
  99% in 0.0771 secs.
```

#### Vulcand

```
[vagrant@consumer ~]$ /usr/local/bin/test01
100000 / 100000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %

Summary:
  Total:  50.5294 secs.
  Slowest:  0.3426 secs.
  Fastest:  0.0010 secs.
  Average:  0.0505 secs.
  Requests/sec: 1979.0451
  Total Data Received:  10800000 bytes.
  Response Size per Request:  108 bytes.

Status code distribution:
  [200] 100000 responses

Response time histogram:
  0.001 [1] |
  0.035 [21120] |∎∎∎∎∎∎∎∎∎∎∎
  0.069 [71365] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.103 [5946]  |∎∎∎
  0.138 [168] |
  0.172 [74]  |
  0.206 [329] |
  0.240 [496] |
  0.274 [388] |
  0.308 [88]  |
  0.343 [25]  |

Latency distribution:
  10% in 0.0290 secs.
  25% in 0.0378 secs.
  50% in 0.0490 secs.
  75% in 0.0571 secs.
  90% in 0.0665 secs.
  95% in 0.0733 secs.
  99% in 0.2057 secs.
```

#### Kong

```
[vagrant@consumer ~]$ /usr/local/bin/test01
100000 / 100000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %

Summary:
  Total:  57.0194 secs.
  Slowest:  1.5978 secs.
  Fastest:  0.0076 secs.
  Average:  0.0569 secs.
  Requests/sec: 1753.7883
  Total Data Received:  13600000 bytes.
  Response Size per Request:  136 bytes.

Status code distribution:
  [200] 100000 responses

Response time histogram:
  0.008 [1]     |
  0.167 [97506] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.326 [2290]  |
  0.485 [103]   |
  0.644 [0]     |
  0.803 [0]     |
  0.962 [0]     |
  1.121 [0]     |
  1.280 [0]     |
  1.439 [17]    |
  1.598 [83]    |

Latency distribution:
  10% in 0.0407 secs.
  25% in 0.0435 secs.
  50% in 0.0461 secs.
  75% in 0.0497 secs.
  90% in 0.0816 secs.
  95% in 0.1076 secs.
  99% in 0.2355 secs.
```

#### Tyk

```
[vagrant@consumer ~]$ /usr/local/bin/test01
100000 / 100000 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00 %

Summary:
  Total:  221.5803 secs.
  Slowest:  5.6482 secs.
  Fastest:  0.0012 secs.
  Average:  0.2215 secs.
  Requests/sec: 451.3037
  Total Data Received:  10800000 bytes.
  Response Size per Request:  108 bytes.

Status code distribution:
  [200] 100000 responses

Response time histogram:
  0.001 [1]     |
  0.566 [90838] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.131 [4890]  |∎∎
  1.695 [2208]  |
  2.260 [1380]  |
  2.825 [383]   |
  3.389 [161]   |
  3.954 [86]    |
  4.519 [40]    |
  5.084 [6]     |
  5.648 [7]     |

Latency distribution:
  10% in 0.0355 secs.
  25% in 0.0521 secs.
  50% in 0.0823 secs.
  75% in 0.1785 secs.
  90% in 0.5263 secs.
  95% in 0.9231 secs.
  99% in 2.1054 secs.
```

## Summary

### Requests per second

| Gateway | RPS       |
|---------|-----------|
| None    | 7534.5524 |
| KrakenD | 3479.1863 |
| Vulcand | 1979.0451 |
| Kong    | 1753.7883 |
| Tyk     | 451.3037  |

### Response time distribution

*Time in milliseconds*

| Gateway | 10%  | 25%  | 50%  | 75%   | 90%   | 95%   | 99%    | Min | Max    | Avg   |
|---------|------|------|------|-------|-------|-------|--------|-----|--------|-------|
| KrakenD | 21.3 | 23.1 | 24.5 | 29.5  | 43.2  | 46.9  | 77.1   | 0.9 | 278.1  | 28.7  |
| Vulcand | 29.0 | 37.8 | 49.0 | 57.1  | 66.5  | 73.3  | 205.7  | 1.0 | 342.6  | 50.5  |
| Kong    | 40.7 | 43.5 | 46.1 | 49.7  | 81.6  | 107.6 | 235.5  | 7.6 | 1597.8 | 56.9  |
| Tyk     | 35.5 | 52.1 | 82.3 | 178.5 | 526.3 | 923.1 | 2105.4 | 1.2 | 5648.2 | 221.5 |