{"id":1529,"date":"2014-08-23T05:57:51","date_gmt":"2014-08-23T10:57:51","guid":{"rendered":"http:\/\/www.shainmiley.com\/wordpress\/?p=1529"},"modified":"2014-08-23T05:58:07","modified_gmt":"2014-08-23T10:58:07","slug":"benchmarking-ceph","status":"publish","type":"post","link":"https:\/\/www.shainmiley.com\/wordpress\/2014\/08\/23\/benchmarking-ceph\/","title":{"rendered":"Benchmarking Ceph"},"content":{"rendered":"<p>This is a post that I have had in draft mode for quite some time.  At this point some of this information is out of date, so I am planning on writing a &#8216;part II&#8217; post shortly, which will include some updated information.<\/p>\n<p><strong>Benchmarking Ceph:<\/strong><\/p>\n<p>Ever since we got our ceph cluster up and running, I&#8217;ve been running various benchmarking applications against different cluster configurations.  Just to review, the cluster that we recently built has the following specs:<\/p>\n<p><strong>Cluster specs:<\/strong><\/p>\n<ul>\n<li>3 x Dell R-420;32 GB of RAM; for MON\/RADOSGW\/MDS nodes<\/li>\n<li>6 x Dell R-720xd;64 GB of RAM; for OSD nodes<\/li>\n<li>72 x 4TB SAS drives as OSD&#8217;s<\/li>\n<li>2 x Force10 S4810 switches<\/li>\n<li>4 x 10 GigE LCAP bonded Intel cards<\/li>\n<li>Ubuntu 12.04 (AMD64)<\/li>\n<li>Ceph 0.72.1 (emperor)<\/li>\n<li>2400 placement groups<\/li>\n<li>261TB of usable space<\/li>\n<\/ul>\n<p>The main role for this cluster will be one primarily tied to archiving audio and video assets.  This being the case, we decided to try and maximize total cluster capacity (4TB drives, no ssd&#8217;s, etc), while at the same time being able to achieve and maintain reasonable cluster throughput (10 GigE, 12 drives per osd nodes, etc).<\/p>\n<p>Most of my benchmarking focused on rbd and radosgw, because either of these is most likely to be what we introduce into production when we are ready. \u00c2\u00a0We are very much awaiting a stable and supported cephfs release (which will hopefully be available sometime in mid-late 2014), which will allow us to switch out our rbd + samba setup, for on based on cephfs.<\/p>\n<p><strong>Rados Benchmarks:\u00c2\u00a0<\/strong><\/p>\n<p>I setup a pool called &#8216;test&#8217; with 1600 pg&#8217;s in order to run some benchmarks using the &#8216;rados bench&#8217; tool that came with Ceph. \u00c2\u00a0I started with a replication level of &#8216;1&#8217; and worked my way up to a replication level of &#8216;3&#8217;.<\/p>\n<div class=\"ex\"><strong>root@hqceph1:\/# rados -p test bench 20 write (rep size=1)<\/strong><br \/>\nTotal time run: 20.241290<br \/>\nTotal writes made: 5646<br \/>\nWrite size: 4194304<br \/>\n<strong>Bandwidth (MB\/sec): 1115.739<\/strong><br \/>\nStddev Bandwidth: 246.027<br \/>\nMax bandwidth (MB\/sec): 1136<br \/>\nMin bandwidth (MB\/sec): 0<br \/>\nAverage Latency: 0.0571572<br \/>\nStddev Latency: 0.0262513<br \/>\nMax latency: 0.336378<br \/>\nMin latency: 0.02248<\/div>\n<div class=\"ex\"><strong>root@hqceph1:\/# rados -p test bench 20 write (rep size=2)<\/strong><br \/>\nTotal time run: 20.547026<br \/>\nTotal writes made: 2910<br \/>\nWrite size: 4194304<br \/>\n<strong>Bandwidth (MB\/sec): 566.505<\/strong><br \/>\nStddev Bandwidth: 154.643<br \/>\nMax bandwidth (MB\/sec): 764<br \/>\nMin bandwidth (MB\/sec): 0<br \/>\nAverage Latency: 0.112384<br \/>\nStddev Latency: 0.198579<br \/>\nMax latency: 2.5105<br \/>\nMin latency: 0.025391<\/div>\n<div class=\"ex\"><strong>root@hqceph1:\/# rados -p test bench 20 write (rep size=3)<\/strong><br \/>\nTotal time run: 20.755272<br \/>\nTotal writes made: 2481<br \/>\nWrite size: 4194304<br \/>\n<strong>Bandwidth (MB\/sec): 478.144 <\/strong><br \/>\nStddev Bandwidth: 147.064<br \/>\nMax bandwidth (MB\/sec): 728<br \/>\nMin bandwidth (MB\/sec): 0<br \/>\nAverage Latency: 0.133827<br \/>\nStddev Latency: 0.229962<br \/>\nMax latency: 3.32957<br \/>\nMin latency: 0.029481<\/div>\n<p><strong>RBD Benchmarks:<\/strong><\/p>\n<p>Next I setup a 10GB block device using rbd:<\/p>\n<div class=\"ex\"><strong>root@ceph1:\/blockdev# dd bs=1M count=256 if=\/dev\/zero of=test1 conv=fdatasync<\/strong> (rep size=1)<br \/>\n256+0 records in<br \/>\n256+0 records out<br \/>\n268435456 bytes (268 MB) copied, 0.440333 s, 610 MB\/s<\/div>\n<div class=\"ex\"><strong>root@ceph1:\/blockdev# dd bs=4M count=256 if=\/dev\/zero of=test1 conv=fdatasync<\/strong> (rep size=1)<br \/>\n256+0 records in<br \/>\n256+0 records out<br \/>\n1073741824 bytes (1.1 GB) copied, 1.07413 s, 1000 MB\/s<\/div>\n<div class=\"ex\"><strong>root@ceph1:\/mnt\/blockdev# hdparm -Tt \/dev\/rbd1<\/strong> (rep size=1)<br \/>\n\/dev\/rbd1:<br \/>\nTiming cached reads: 16296 MB in 2.00 seconds = 8155.69 MB\/sec<br \/>\nTiming buffered disk reads: 246 MB in 3.10 seconds = 79.48 MB\/sec<\/div>\n<div class=\"ex\"><strong>root@ceph1:\/mnt\/blockdev# dd bs=1M count=256 if=\/dev\/zero of=test conv=fdatasync<\/strong> (rep size=2)<br \/>\n256+0 records in<br \/>\n256+0 records out<br \/>\n268435456 bytes (268 MB) copied, 1.29985 s, 207 MB\/s<\/div>\n<div class=\"ex\"><strong>root@ceph1:\/mnt\/blockdev# dd bs=4M count=256 if=\/dev\/zero of=test2 conv=fdatasync<\/strong>(rep size=2)<br \/>\n256+0 records in<br \/>\n256+0 records out<br \/>\n1073741824 bytes (1.1 GB) copied, 4.02375 s, 267 MB\/s<\/strong><\/div>\n<div class=\"ex\"><strong>root@cephmount1:\/mnt\/ceph-block-device\/test# hdparm -Tt \/dev\/rbd1 <\/strong> (rep size=2)<br \/>\n\/dev\/rbd1:<br \/>\nTiming cached reads: 16434 MB in 2.00 seconds = 8225.55 MB\/sec<br \/>\nTiming buffered disk reads: 152 MB in 3.01 seconds = 50.55 MB\/sec<\/div>\n<p><strong>Radosgw Benchmarks:<\/strong><\/p>\n<p>Using s3cmd (s3tools) I was able to achieve about 70MB\/s when pushing files to ceph via the s3 restful API.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is a post that I have had in draft mode for quite some time. At this point some of this information is out of date, so I am planning on writing a &#8216;part II&#8217; post shortly, which will include some updated information. Benchmarking Ceph: Ever since we got our ceph cluster up and running, [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[35,3,23,32],"tags":[],"_links":{"self":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts\/1529"}],"collection":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/comments?post=1529"}],"version-history":[{"count":56,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts\/1529\/revisions"}],"predecessor-version":[{"id":1588,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts\/1529\/revisions\/1588"}],"wp:attachment":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/media?parent=1529"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/categories?post=1529"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/tags?post=1529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}