profile
viewpoint

Ask questionsDistributed minio low read performance

Hi, thank you for you hard work.

I'm requesting 1 file index.html with Hello world! content from minio using ab

Expected Behavior

5-10 ms 98 percentile for head/getObject request(small object).

Current Behavior

Part one

$ ab -c 10 -n 1000 "https://minio/rvadim-test/index.html?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=rvadim-test%2F20200324%2F%2Fs3%2Faws4_request&X-Amz-Date=20200324T085719Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=..."
Document Length:        13 bytes

Concurrency Level:      10
Time taken for tests:   3.440 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      490000 bytes
HTML transferred:       13000 bytes
Requests per second:    290.68 [#/sec] (mean)
Time per request:       34.402 [ms] (mean)
Time per request:       3.440 [ms] (mean, across all concurrent requests)
Transfer rate:          139.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        3    5   1.3      4      14
Processing:    17   29   8.9     28      86
Waiting:       17   29   8.9     28      86
Total:         21   34   9.3     32      95

Percentage of the requests served within a certain time (ms)
  50%     32
  66%     36
  75%     38
  80%     40
  90%     45
  95%     51
  98%     60
  99%     64
 100%     95 (longest request)

Also mc admin trace minio reports 15-30ms time:

09:40:00.340 [200 OK] s3.GetObject minio./rvadim-test/index.html?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=rvadim-test%2F20200324%2F%2Fs3%2Faws4_request&X-Amz-Date=20200324T085719Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=...  123.123.123.123    24.692ms

Part two

If I try to increase ab concurrency more then 20, response time begins to grow. For example, with ab -c 100 105ms 50 percentile and 200+ms 98 percentile, minio trace reports 30-200ms time

Questions

  1. Is it normal for my configuration to have ~15-30ms response on minio?
  2. Is it normal that with the increase of concurrent requests increases the response time of the minio?

Your Environment

  • Version used (minio version): RELEASE.2020-03-14T02-21-58Z
  • Environment name and version (e.g. nginx 1.9.1): Kubernetes 1.15.9
  • Server type and version: --
  • Operating System and version (uname -a): Centos 7 4.4.213-1
  • Configuration: 2 zones by 16 disks
  • 20M objects in storage
minio_args:
    - minio
    - server
    - https://minio-z0-{0...3}.minio-z0:9000/data{0...3}
    - https://minio-z1-{0...3}.minio-z1:9000/data{0...3}

4 nodes with 8 disks each especially for minio, 2 minio pods on each node. 80 cores(16 available for each minio) 32 Gb RAM available for each minio 3.6 Tb SSD disks, 4 disk per minio pod

minio/minio

Answer questions klauspost

This doesn't seem completely unreasonable. For very small files like this a regular http server will likely outperform it. MinIO has to coordinate between all your servers on each request, so small payloads will have a considerable overhead.

'ab' is generally not considered a good benchmark, even for HTTP servers. You can give warp a try. In particular you can see request times for different payload sizes using the --obj.randsize option.

Maybe try warp get --obj.randsize --autoterm --concurrent=10 --obj.size=50MB --requests to get an idea of the performance. It will test up to 50MiB objects. You can set host and credentials with env variables.

useful!

Related questions

minio heal broken after delete both part.1 and xl.json file hot 1
Bad status in distributed cluster when nodes are restarted hot 1
Distributed: PutObject io: read/write on closed pipe hot 1
codebase imports an old repository github.com/coreos/etcd hot 1
MINIO_HTTP_TRACE is not working hot 1
ssl protocol error hot 1
[TLS]use:s3cmd ls s3:// error:ERROR: SSL certificate verification failure: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:59 hot 1
Minio behind reverse proxy breaks presigned URLs hot 1
Curl user-agent is giving an HTTP 403 on all endpoints hot 1
Can not add custom policy to user hot 1
presignedGetObject's url return SignatureDoesNotMatch error hot 1
'TLS handshake error from 172.17.0.1:59810: remote error: tls: unknown certificate' error when using minio with https in kubernetes hot 1
Bucket not resolved when using MINIO_DOMAIN hot 1
Minio server is not starting hot 1
build github.com/minio/minio: cannot load archive/zip: cannot find module providing package archive/zip hot 1
Github User Rank List