我的书签
添加书签
移除书签Luminous版本的ceph新增了一个crush class的功能，可以为新增的设备指定为类，创建rule的时候直接指定class即可。
实验环境
# cat /etc/redhat-releaseCentOS Linux release 7.5.1804 (Core)# ceph --versionceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous (stable)
由于实验环境中并没有ssd磁盘，所以这里操作时假设每一个主机有一块ssd盘，操作时手动把对应的osd调整class标签。
修改crush class
1、查看当前osd分布情况
# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0.74797 root default-15 0.21999 rack rack1-9 0.08800 host node13 hdd 0.04399 osd.3 up 1.00000 1.0000010 hdd 0.04399 osd.10 up 0.98001 1.0000011 hdd 0.04399 osd.11 up 0.96002 1.00000-3 0.13199 host storage-04 hdd 0.04399 osd.4 up 0.96002 1.000005 hdd 0.04399 osd.5 up 0.96002 1.0000014 hdd 0.04399 osd.14 up 0.98001 1.00000-16 0.26399 rack rack2-5 0.13199 host node20 hdd 0.04399 osd.0 up 0.98628 1.000006 hdd 0.04399 osd.6 up 1.00000 1.0000016 hdd 0.04399 osd.16 up 1.00000 1.00000-7 0.13199 host storage-12 hdd 0.04399 osd.2 up 1.00000 1.000008 hdd 0.04399 osd.8 up 1.00000 1.0000012 hdd 0.04399 osd.12 up 1.00000 1.00000-17 0.26399 rack rack3-11 0.13199 host node31 hdd 0.04399 osd.1 up 1.00000 1.000007 hdd 0.04399 osd.7 up 1.00000 1.0000015 hdd 0.04399 osd.15 up 1.00000 1.00000-13 0.13199 host storage-29 hdd 0.04399 osd.9 up 1.00000 1.0000013 hdd 0.04399 osd.13 up 1.00000 1.0000017 hdd 0.04399 osd.17 up 0.96002 1.00000
2、查看当前集群的crush class
# ceph osd crush class ls["hdd",]
3、删除osd.0，osd.1，osd.2，osd.3，osd.4，osd.9的class
# for i in 0 1 2 3 4 9 ; do ceph osd crush rm-device-class osd.$i ; donedone removing class of osd(s): 0done removing class of osd(s): 1done removing class of osd(s): 2done removing class of osd(s): 3done removing class of osd(s): 4done removing class of osd(s): 9# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0.74797 root default-15 0.21999 rack rack1-9 0.08800 host node13 0.04399 osd.3 up 1.00000 1.0000010 hdd 0.04399 osd.10 up 0.98001 1.0000011 hdd 0.04399 osd.11 up 0.96002 1.00000-3 0.13199 host storage-04 0.04399 osd.4 up 0.96002 1.000005 hdd 0.04399 osd.5 up 0.96002 1.0000014 hdd 0.04399 osd.14 up 0.98001 1.00000-16 0.26399 rack rack2-5 0.13199 host node20 0.04399 osd.0 up 0.98628 1.000006 hdd 0.04399 osd.6 up 1.00000 1.0000016 hdd 0.04399 osd.16 up 1.00000 1.00000-7 0.13199 host storage-12 0.04399 osd.2 up 1.00000 1.000008 hdd 0.04399 osd.8 up 1.00000 1.0000012 hdd 0.04399 osd.12 up 1.00000 1.00000-17 0.26399 rack rack3-11 0.13199 host node31 0.04399 osd.1 up 1.00000 1.000007 hdd 0.04399 osd.7 up 1.00000 1.0000015 hdd 0.04399 osd.15 up 1.00000 1.00000-13 0.13199 host storage-29 0.04399 osd.9 up 1.00000 1.0000013 hdd 0.04399 osd.13 up 1.00000 1.0000017 hdd 0.04399 osd.17 up 0.96002 1.00000
4、设置osd.0，osd.1，osd.2，osd.3，osd.4，osd.9的class为ssd
# for i in 0 1 2 3 4 9 ; do ceph osd crush set-device-class ssd osd.$i ; doneset osd(s) 0 to class 'ssd'set osd(s) 1 to class 'ssd'set osd(s) 2 to class 'ssd'set osd(s) 3 to class 'ssd'set osd(s) 4 to class 'ssd'set osd(s) 9 to class 'ssd'# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0.74797 root default-15 0.21999 rack rack1-9 0.08800 host node110 hdd 0.04399 osd.10 up 0.98001 1.0000011 hdd 0.04399 osd.11 up 0.96002 1.000003 ssd 0.04399 osd.3 up 1.00000 1.00000-3 0.13199 host storage-05 hdd 0.04399 osd.5 up 0.96002 1.0000014 hdd 0.04399 osd.14 up 0.98001 1.000004 ssd 0.04399 osd.4 up 0.96002 1.00000-16 0.26399 rack rack2-5 0.13199 host node26 hdd 0.04399 osd.6 up 1.00000 1.0000016 hdd 0.04399 osd.16 up 1.00000 1.000000 ssd 0.04399 osd.0 up 0.98628 1.00000-7 0.13199 host storage-18 hdd 0.04399 osd.8 up 1.00000 1.0000012 hdd 0.04399 osd.12 up 1.00000 1.000002 ssd 0.04399 osd.2 up 1.00000 1.00000-17 0.26399 rack rack3-11 0.13199 host node37 hdd 0.04399 osd.7 up 1.00000 1.0000015 hdd 0.04399 osd.15 up 1.00000 1.000001 ssd 0.04399 osd.1 up 1.00000 1.00000-13 0.13199 host storage-213 hdd 0.04399 osd.13 up 1.00000 1.0000017 hdd 0.04399 osd.17 up 0.96002 1.000009 ssd 0.04399 osd.9 up 1.00000 1.00000
5、再次查看crush class，多了个ssd
# ceph osd crush class ls["hdd","ssd"]
6、创建一个优先使用ssd的crush rule
# ceph osd crush rule create-replicated rule-ssd default rack ssd# ceph osd crush rule dump rule-ssd{"rule_id": 1,"rule_name": "rule-ssd","ruleset": 1,"type": 1,"min_size": 1,"max_size": 10,"steps": [{"op": "take","item": -30,"item_name": "default~ssd"},{"op": "chooseleaf_firstn","num": 0,"type": "rack"},{"op": "emit"}]}
验证
方法一：
1、获取crushmap
# ceph osd getcrushmap -o monmap60
2、反编译crushmap
# crushtool -d monmap -o monmap.txt# cat monmap.txt…………rule rule-ssd {id 1type replicatedmin_size 1max_size 10step take default class ssdstep chooseleaf firstn 0 type rackstep emit}
可以看到在crushmap中多了一项rule-ssd的crush rule
3、测试
# crushtool -i monmap --test --min-x 0 --max-x 9 --num-rep 3 --ruleset 1 --show_mappingsCRUSH rule 1 x 0 [3,2,9]CRUSH rule 1 x 1 [2,4,9]CRUSH rule 1 x 2 [1,4,0]CRUSH rule 1 x 3 [9,0,3]CRUSH rule 1 x 4 [2,9,3]CRUSH rule 1 x 5 [1,2,4]CRUSH rule 1 x 6 [1,3,0]CRUSH rule 1 x 7 [1,0,4]CRUSH rule 1 x 8 [0,4,1]CRUSH rule 1 x 9 [0,1,3]
从测试结果可以看到一共10次的测试中，所有的副本所在的OSD都是前面设置了class为ssd的OSD
方法二：
1、创建名为ssdtest的pool，并设置crush rule为rule-ssd
# ceph osd pool create ssdtest 64 64 rule-ssdpool 'ssdtest' created[root@node1 ~]# ceph osd pool get ssdtest crush_rulecrush_rule: rule-ssd
2、上传对象
# rados -p ssdtest put init.txt init.sh
3、查询对象的OSD组
# ceph osd map ssdtest init.txtosdmap e286 pool 'ssdtest' (10) object 'init.txt' -> pg 10.66387d99 (10.19) -> up ([9,2,3], p9) acting ([9,2,3], p9)
从查询结果可以看出，对象所在的OSD，是class为ssd的OSD
