mysql压力测试实践

mysql 压力测试

实验环境

系统： centos 7.6

地区：香港

配置：2核8G 磁盘 essd

mysql版本：5.7.29

压测工具：mysqlslap

常用参数 [options] 详细说明：

--auto-generate-sql, -a 自动生成测试表和数据，表示用mysqlslap工具自己生成的SQL脚本来测试并发压力。
--auto-generate-sql-load-type=type 测试语句的类型。代表要测试的环境是读操作还是写操作还是两者混合的。取值包括：read，key，write，update和mixed(默认)。
--auto-generate-sql-add-auto-increment 代表对生成的表自动添加auto_increment列，从5.1.18版本开始支持。
--number-char-cols=N, -x N 自动生成的测试表中包含多少个字符类型的列，默认1
--number-int-cols=N, -y N 自动生成的测试表中包含多少个数字类型的列，默认1
--number-of-queries=N 总的测试查询次数(并发客户数×每客户查询次数)
--query=name,-q 使用自定义脚本执行测试，例如可以调用自定义的一个存储过程或者sql语句来执行测试。
--create-schema 代表自定义的测试库名称，测试的schema，MySQL中schema也就是database。
--commint=N 多少条DML后提交一次。
--compress, -C 如果服务器和客户端支持都压缩，则压缩信息传递。
--concurrency=N, -c N 表示并发量，也就是模拟多少个客户端同时执行select。可指定多个值，以逗号或者--delimiter参数指定的值做为分隔符。例如：--concurrency=100,200,500。
--engine=engine_name, -e engine_name 代表要测试的引擎，可以有多个，用分隔符隔开。例如：--engines=myisam,innodb。
--iterations=N, -i N 测试执行的迭代次数，代表要在不同并发环境下，各自运行测试多少次。
--only-print 只打印测试语句而不实际执行。
--detach=N 执行N条语句后断开重连。
--debug-info, -T 打印内存和CPU的相关信息。

第一次测试

# 单线程访问测试
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -uroot -p'QianFeng@123'
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.003 seconds
        Minimum number of seconds to run all queries: 0.003 seconds
        Maximum number of seconds to run all queries: 0.003 seconds
        Number of clients running queries: 1
        Average number of queries per client: 0

 测试结果显示单线程连接一次服务器需要3ms。
 
# 100 个线程同时连接一次
 [root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c 100 -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.281 seconds
        Minimum number of seconds to run all queries: 0.281 seconds
        Maximum number of seconds to run all queries: 0.281 seconds
        Number of clients running queries: 100
        Average number of queries per client: 0

可以看到使用100个线程同时连接一次服务器需要218ms，同单线程比，响应时间大幅度提升，这个不知道是什么原因，需要后续查询下。

# 1000个线程同时连接一次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c 1000 -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
mysqlslap: Error when connecting to server: 1040 Too many connections
mysqlslap: Error when connecting to server: 1040 Too many connections
mysqlslap: Error when connecting to server: 1040 Too many connections

这里我们发现出现了错误，初步猜测是由于我们没有对服务进行任何优化造成的，mysql的默认线程只有151个，这里我们将他进行修改。

[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c 1000 -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 12.480 seconds
        Minimum number of seconds to run all queries: 12.480 seconds
        Maximum number of seconds to run all queries: 12.480 seconds
        Number of clients running queries: 1000
        Average number of queries per client: 0

总结：1000并发连接的处理时间是12.5秒

第二次测试

# 单线程10次迭代测试平均值
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -i 10 -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.003 seconds
        Minimum number of seconds to run all queries: 0.003 seconds
        Maximum number of seconds to run all queries: 0.003 seconds
        Number of clients running queries: 1
        Average number of queries per client: 0

总结：与上次结果一样，约3ms

# 多线程10次迭代测试平均值
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c 100 -i 10 -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.208 seconds
        Minimum number of seconds to run all queries: 0.200 seconds
        Maximum number of seconds to run all queries: 0.220 seconds
        Number of clients running queries: 100
        Average number of queries per client: 0
总结：较上次测试快一些，应该是优化得原因

# 多线程10次迭代测试平均值
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c 1000 -i 10 -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 11.156 seconds
        Minimum number of seconds to run all queries: 10.944 seconds
        Maximum number of seconds to run all queries: 11.631 seconds
        Number of clients running queries: 1000
        Average number of queries per client: 0

第三次测试

# 读操作测试  并发10  共1000次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c10 --number-of-queries=1000 --auto-generate-sql-load-type=read -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.093 seconds
        Minimum number of seconds to run all queries: 0.093 seconds
        Maximum number of seconds to run all queries: 0.093 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

# 读操作测试 并发1 共1000次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a --number-of-queries=1000 --auto-generate-sql-load-type=read -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.141 seconds
        Minimum number of seconds to run all queries: 0.141 seconds
        Maximum number of seconds to run all queries: 0.141 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

# 读操作测试 并发500 共10000次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c500 --number-of-queries=10000 --auto-generate-sql-load-type=read -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 1.177 seconds
        Minimum number of seconds to run all queries: 1.177 seconds
        Maximum number of seconds to run all queries: 1.177 seconds
        Number of clients running queries: 500
        Average number of queries per client: 20

# 写操作测试  并发10  共1000次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c10 --number-of-queries=1000 --auto-generate-sql-load-type=write -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.171 seconds
        Minimum number of seconds to run all queries: 0.171 seconds
        Maximum number of seconds to run all queries: 0.171 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

# 写操作测试  并发1  共1000次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a --number-of-queries=1000 --auto-generate-sql-load-type=write -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.603 seconds
        Minimum number of seconds to run all queries: 0.603 seconds
        Maximum number of seconds to run all queries: 0.603 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

# 写操作测试  并发500  共10000次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c 500 --number-of-queries=10000 --auto-generate-sql-load-type=write -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.871 seconds
        Minimum number of seconds to run all queries: 0.871 seconds
        Maximum number of seconds to run all queries: 0.871 seconds
        Number of clients running queries: 500
        Average number of queries per client: 20

第四次测试

# 混合操作 并发10  共1000次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c10 --number-of-queries=1000 --auto-generate-sql-load-type=mixed -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.204 seconds
        Minimum number of seconds to run all queries: 0.204 seconds
        Maximum number of seconds to run all queries: 0.204 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

# 混合操作 并发1  共1000次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a --number-of-queries=1000 --auto-generate-sql-load-type=mixed -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 0.524 seconds
        Minimum number of seconds to run all queries: 0.524 seconds
        Maximum number of seconds to run all queries: 0.524 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

# 混合操作 并发500  共10000次
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a -c500 --number-of-queries=10000 --auto-generate-sql-load-type=mixed -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Average number of seconds to run all queries: 9.179 seconds
        Minimum number of seconds to run all queries: 9.179 seconds
        Maximum number of seconds to run all queries: 9.179 seconds
        Number of clients running queries: 500
        Average number of queries per client: 20

第五次测试

[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a --concurrency=1,10,500 --number-of-queries 1000 --engine=myisam,innodb,memory --iterations=5 -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Running for engine myisam
        Average number of seconds to run all queries: 0.151 seconds
        Minimum number of seconds to run all queries: 0.144 seconds
        Maximum number of seconds to run all queries: 0.165 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

Benchmark
        Running for engine myisam
        Average number of seconds to run all queries: 0.105 seconds
        Minimum number of seconds to run all queries: 0.098 seconds
        Maximum number of seconds to run all queries: 0.123 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

Benchmark
        Running for engine myisam
        Average number of seconds to run all queries: 0.216 seconds
        Minimum number of seconds to run all queries: 0.205 seconds
        Maximum number of seconds to run all queries: 0.232 seconds
        Number of clients running queries: 500
        Average number of queries per client: 2

Benchmark
        Running for engine innodb
        Average number of seconds to run all queries: 0.524 seconds
        Minimum number of seconds to run all queries: 0.495 seconds
        Maximum number of seconds to run all queries: 0.550 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

Benchmark
        Running for engine innodb
        Average number of seconds to run all queries: 0.198 seconds
        Minimum number of seconds to run all queries: 0.186 seconds
        Maximum number of seconds to run all queries: 0.206 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

Benchmark
        Running for engine innodb
        Average number of seconds to run all queries: 0.326 seconds
        Minimum number of seconds to run all queries: 0.294 seconds
        Maximum number of seconds to run all queries: 0.363 seconds
        Number of clients running queries: 500
        Average number of queries per client: 2

Benchmark
        Running for engine memory
        Average number of seconds to run all queries: 0.119 seconds
        Minimum number of seconds to run all queries: 0.115 seconds
        Maximum number of seconds to run all queries: 0.122 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

Benchmark
        Running for engine memory
        Average number of seconds to run all queries: 0.085 seconds
        Minimum number of seconds to run all queries: 0.083 seconds
        Maximum number of seconds to run all queries: 0.090 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

Benchmark
        Running for engine memory
        Average number of seconds to run all queries: 0.205 seconds
        Minimum number of seconds to run all queries: 0.189 seconds
        Maximum number of seconds to run all queries: 0.221 seconds
        Number of clients running queries: 500
        Average number of queries per client: 2


----------------------------------------------------------------
[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a --concurrency=1,10,500 --number-of-queries 1000 --engine=myisam,innodb,memory --auto-generate-sql-load-type=read --iterations=5 -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Running for engine myisam
        Average number of seconds to run all queries: 0.113 seconds
        Minimum number of seconds to run all queries: 0.106 seconds
        Maximum number of seconds to run all queries: 0.128 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

Benchmark
        Running for engine myisam
        Average number of seconds to run all queries: 0.072 seconds
        Minimum number of seconds to run all queries: 0.067 seconds
        Maximum number of seconds to run all queries: 0.082 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

Benchmark
        Running for engine myisam
        Average number of seconds to run all queries: 0.168 seconds
        Minimum number of seconds to run all queries: 0.160 seconds
        Maximum number of seconds to run all queries: 0.178 seconds
        Number of clients running queries: 500
        Average number of queries per client: 2

Benchmark
        Running for engine innodb
        Average number of seconds to run all queries: 0.142 seconds
        Minimum number of seconds to run all queries: 0.137 seconds
        Maximum number of seconds to run all queries: 0.150 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

Benchmark
        Running for engine innodb
        Average number of seconds to run all queries: 0.098 seconds
        Minimum number of seconds to run all queries: 0.094 seconds
        Maximum number of seconds to run all queries: 0.106 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

Benchmark
        Running for engine innodb
        Average number of seconds to run all queries: 0.202 seconds
        Minimum number of seconds to run all queries: 0.194 seconds
        Maximum number of seconds to run all queries: 0.207 seconds
        Number of clients running queries: 500
        Average number of queries per client: 2

Benchmark
        Running for engine memory
        Average number of seconds to run all queries: 0.092 seconds
        Minimum number of seconds to run all queries: 0.091 seconds
        Maximum number of seconds to run all queries: 0.093 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

Benchmark
        Running for engine memory
        Average number of seconds to run all queries: 0.056 seconds
        Minimum number of seconds to run all queries: 0.055 seconds
        Maximum number of seconds to run all queries: 0.060 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

Benchmark
        Running for engine memory
        Average number of seconds to run all queries: 0.158 seconds
        Minimum number of seconds to run all queries: 0.154 seconds
        Maximum number of seconds to run all queries: 0.161 seconds
        Number of clients running queries: 500
        Average number of queries per client: 2

------------------------------------------------------------

[root@iZj6cd4tdhfozkz2n0b6dqZ ~]# mysqlslap -a --concurrency=1,10,500 --number-of-queries 1000 --engine=myisam,innodb,memory --auto-generate-sql-load-type=write --iterations=5 -uroot -pQianFeng@123
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
        Running for engine myisam
        Average number of seconds to run all queries: 0.054 seconds
        Minimum number of seconds to run all queries: 0.053 seconds
        Maximum number of seconds to run all queries: 0.060 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

Benchmark
        Running for engine myisam
        Average number of seconds to run all queries: 0.038 seconds
        Minimum number of seconds to run all queries: 0.035 seconds
        Maximum number of seconds to run all queries: 0.052 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

Benchmark
        Running for engine myisam
        Average number of seconds to run all queries: 0.137 seconds
        Minimum number of seconds to run all queries: 0.127 seconds
        Maximum number of seconds to run all queries: 0.150 seconds
        Number of clients running queries: 500
        Average number of queries per client: 2

Benchmark
        Running for engine innodb
        Average number of seconds to run all queries: 0.603 seconds
        Minimum number of seconds to run all queries: 0.595 seconds
        Maximum number of seconds to run all queries: 0.621 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

Benchmark
        Running for engine innodb
        Average number of seconds to run all queries: 0.141 seconds
        Minimum number of seconds to run all queries: 0.119 seconds
        Maximum number of seconds to run all queries: 0.171 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

Benchmark
        Running for engine innodb
        Average number of seconds to run all queries: 0.192 seconds
        Minimum number of seconds to run all queries: 0.164 seconds
        Maximum number of seconds to run all queries: 0.223 seconds
        Number of clients running queries: 500
        Average number of queries per client: 2

Benchmark
        Running for engine memory
        Average number of seconds to run all queries: 0.046 seconds
        Minimum number of seconds to run all queries: 0.045 seconds
        Maximum number of seconds to run all queries: 0.052 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1000

Benchmark
        Running for engine memory
        Average number of seconds to run all queries: 0.038 seconds
        Minimum number of seconds to run all queries: 0.031 seconds
        Maximum number of seconds to run all queries: 0.047 seconds
        Number of clients running queries: 10
        Average number of queries per client: 100

Benchmark
        Running for engine memory
        Average number of seconds to run all queries: 0.121 seconds
        Minimum number of seconds to run all queries: 0.116 seconds
        Maximum number of seconds to run all queries: 0.127 seconds
        Number of clients running queries: 500
        Average number of queries per client: 2

mysql 监控

工具：mytop
介绍：
man mytop
-u / --user <USERNAME>：指定 username，预设是 root
-p / --pass / --password <PASSWORD>：指定password，预设是none
-h / --host <HOSTNAME[:PORT]>：指定 MySQL server的hostname，预设是localhost
-P / --port <PORT>：指定连接 MySQL server的port，预设是3306
-s / --delay <SECONDS>：更新的秒数，预设是5秒
-d / --db / --database <DATABASE>：指定连接的资料库，预设是test
-b / --batch / --batchmode：指定为 batch mode，每次更新不会清除旧的显示结果，会将更新资料显示上最上方，预设是unset
-S / --socket <PATH_TO_SOCKET>：指定使用MySQL socket直接连线，而不使用TCP/IP连线，预设是none(当mytop和MySQL在同一台时才能使用)
--header or -noheader：是否要显示表头，预设是header
--color or --nocolor：是否要使用颜色，预设是color
-i / -idle or -noidle：idle 的thread是否要出现在清单上，预设是idle


安装：
yum install yum-plugin-protectbase.noarch -y
yum install mytop -y
vim /root/.mytop
host=localhost
user=root
pass=QianFeng@123
db=mysql
port=3306
socket=/tmp/mysql.sock
delay=5
batchmode=0
color=1 
idle=1

fio 磁盘性能压测

常用参数的说明
--debug 调试模式 
--version 显示版本信息 
测试任务相关的参数 
--output= 文件名 输出到文件 
--runtime= 限制运行时间 单位为秒 
-- name= 字符串 测试项目名称 
--description= 字符串 测试项目描述 
--filename= 测试文件，也可以制定到裸设备上 
--size= 测试文件大小 
--filesize= 单独的文件大小，也可以是一个范围，fio在设定的size只内随机的选择大小，如果没有指定，每个子文件大小相同。 
读写相关的参数 
--readwrite= 
read 顺序读 
write 顺序写 
trim 顺序trim，只支持Linux块设备 
randread 随机读 
randwrite 随机写 
randtrim 随机trim 只支持Linux块设备 
rw，readnwrite 混合读写 
randrw 随机混合读写 
如果是混合读写，默认是 50/50 即50%读加50%的写，对于特定的类型，结果会有些出入，因为速度可能不一样。可以在测试的指定一个数字， 
rwmixwrite=20 混合读写，写占20% 
direct=1 或者0 ，1为绕过文件系统的缓存 
--blocksize= 
--bs= 测试块大小，4K 8K 1M 等 
--bssplit=字符串 需要测试混合块大小的时候使用，可以按照如下的格式 
bssplit=4k/10:64k/50:32k/40 4K占10% 64K占50% 32k占40% 
也可以不指定百分比，fio会平均分配，比如 
bssplit=4k/50:1k/:32k/ 就是4k占50% 1K 32看各占25% 
bssplit 读写的时候都可以使用 
运行相关参数 
--max-job= 运行的最高线程 
--zero_buffers 如果使用这个参数，fio会初始化IO缓存，并默认使用随机的数据填满缓存。 
--refill_buffers 强制重新填写读写缓存 
--iodepth= io队列深度 默认是1 
--direct= 0或者1 1为关闭文件系统缓存 
--numjobs= 线程数量 
--group_reporting 汇总每个进程的信息 
--lockmem= 固件测试的时候内存大小 比如2G 
--nrfiles=8 每个进程生成文件的数量

下面几个参数在libaio的引擎下的作用： 
--iodepth_batch=当队列里面的IO个数达到该值的时候，就调用io_submit批次提交请求 
--iodepth_batch_complete=一次最多可收割的io数 
--iodepth_low=队列的io数降到该值，就重新填充

2.fio使用案例
文件系统4k顺序读测试 
fio --filename=/tmp/test --direct=1 --rw=read --bs=4k --size=10G --numjobs=64 --runtime=300 --group_reporting --name=test-read 
文件系统4k随机写测试 
fio --filename=/tmp/test --direct=1 --rw=randwrite --bs=4k --size=15G --numjobs=64 --runtime=300 --group_reporting --name=test-read 
裸设备4k顺序写测试 
fio --filename=/dev/sdb --direct=1 --rw=write --bs=4k --size=15G --numjobs=64 --runtime=300 --group_reporting --name=test-read

3.测试结果解读
fio --filename=/test --direct=1 --rw=randwrite --bs=4k --size=1G --numjobs=64 --runtime=300 --group_reporting --name=test-randwrite 
… 
write: io=2850.2MB, bw=9726.1KB/s, iops=2431 , runt=300048msec 
clat (usec): min=159 , max=448315 , avg=410.20, stdev=1208.19 
lat (usec): min=159 , max=448315 , avg=411.15, stdev=1208.27 
clat percentiles (usec): 
| 1.00th=[ 203], 5.00th=[ 278], 10.00th=[ 326], 20.00th=[ 346], 
| 30.00th=[ 366], 40.00th=[ 386], 50.00th=[ 402], 60.00th=[ 418], 
| 70.00th=[ 434], 80.00th=[ 454], 90.00th=[ 486], 95.00th=[ 516], 
| 99.00th=[ 724], 99.50th=[ 1032], 99.90th=[ 1368], 99.95th=[ 2224], 
| 99.99th=[ 8256] 
bw (KB/s) : min= 3, max= 667, per=1.57%, avg=152.30, stdev=27.63 
lat (usec) : 250=3.92%, 500=89.07%, 750=6.14%, 1000=0.32% 
lat (msec) : 2=0.49%, 4=0.03%, 10=0.02%, 20=0.01%, 50=0.01% 
lat (msec) : 500=0.01% 
cpu : usr=0.04%, sys=0.23%, ctx=1548182, majf=0, minf=1844 
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% 
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
issued : total=r=0/w=729637/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs): 
WRITE: io=2850.2MB, aggrb=9726KB/s, minb=9726KB/s, maxb=9726KB/s, mint=300048msec, maxt=300048msec

Disk stats (read/write): 
dm-0: ios=0/766673, merge=0/0, ticks=0/341324, in_queue=341316, util=86.15%, aggrios=0/731075, aggrmerge=0/35722, aggrticks=0/268135, aggrin_queue=267337, aggrutil=85.33% 
vda: ios=0/731075, merge=0/35722, ticks=0/268135, in_queue=267337, util=85.33%

I/O延迟包括三种：slat，clat，lat 
关系是 lat = slat + clat。 
slat 表示fio submit某个I/O的延迟。 
clat 表示fio complete某个I/O的延迟。 
lat 表示从fio将请求提交给内核，再到内核完成这个I/O为止所需要的时间。

4.通过脚本执行fio测试
fio 也可以将测试用例写好脚本，通过运行脚本，完成多项测试任务 
; -- start -- 
[global] 
[random-writers] 
rw=randwrite 
bs=4K 
direct=1 
size=30G 
numjobs=63 
[random-read] 
rw=randread 
bs=4K 
direct=1 
size=30G 
numjobs=63