{"id":887,"date":"2011-08-27T17:48:17","date_gmt":"2011-08-27T22:48:17","guid":{"rendered":"http:\/\/www.shainmiley.com\/wordpress\/?p=887"},"modified":"2012-03-07T20:36:25","modified_gmt":"2012-03-08T01:36:25","slug":"zfs-crash-during-high-io","status":"publish","type":"post","link":"https:\/\/www.shainmiley.com\/wordpress\/2011\/08\/27\/zfs-crash-during-high-io\/","title":{"rendered":"ZFS crash during high I\/O"},"content":{"rendered":"<p>After successfully completing a &#8216;zfs replace&#8217; I was not so pleased to get the following error message back from &#8216;zfs detach&#8217;:<\/p>\n<p>cannot detach c5t17d0: no valid replicas<\/p>\n<p>I decided that I would <a href=\"http:\/\/www.kamiogi.net\/Kamiogi\/Frame_Dragging\/Entries\/2009\/6\/5_A_Smooth_Upgrade_-_OpenSolaris_2009.06_snv_111b.html\" target=\"_blank\">upgrade<\/a> this OpenSolaris 2008.11 instance to OpenSolaris 2009.06 in order to see if the obvious bug that I was encountering was resolved in the newest version. Since upgrading in OpenSolaris supports automatic <a href=\"http:\/\/developers.sun.com\/developer\/technicalArticles\/opensolaris\/boot-environments.html\" target=\"_blank\">boot environment<\/a> creation, there really is not much danger at all in updating because you can always boot back into the other environment at any time.<\/p>\n<p>The upgrade was a success, and after I booted into 2009.06 I was able to simply detach the failed drive from the pool and thus remove it from the system.<\/p>\n<p>I recompiled gluster and I ran 2009.06 for a couple of days, until I started noticing that the server was rebooting during times of high I\/O. A peek inside &#8216;\/var\/adm\/messages&#8217; revealed the following errors:<\/p>\n<div class=\"ex\">Aug 15 22:33:04 cybertron unix: [ID 836849 kern.notice]<br \/>\nAug 15 22:33:04 cybertron ^Mpanic[cpu0]\/thread=ffffff091060c900:<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 783603 kern.notice] Deadlock: cycle in blocking chain<br \/>\nAug 15 22:33:04 cybertron unix: [ID 100000 kern.notice]<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d9651f0 genunix:turnstile_block+795 ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965250 unix:mutex_vector_enter+261 ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d9652f0 zfs:zfs_zget+be ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965380 zfs:zfs_zaccess+7c ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965400 zfs:zfs_lookup+333 ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d9654a0 genunix:fop_lookup+ed ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965550 genunix:xattr_dir_realdir+8b ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d9655a0 genunix:xattr_dir_realvp+5e ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d9655f0 genunix:fop_realvp+32 ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965640 genunix:vn_compare+31 ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965860 genunix:lookuppnvp+94c ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965900 genunix:lookuppnatcred+11b ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965990 genunix:lookuppnat+69 ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965b30 genunix:vn_createat+13a ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965cf0 genunix:vn_openat+1fb ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965e50 genunix:copen+435 ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965e80 genunix:openat64+25 ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965ec0 genunix:fsat32+f5 ()<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 655072 kern.notice] ffffff003d965f10 unix:brand_sys_sysenter+1e0 ()<br \/>\nAug 15 22:33:04 cybertron unix: [ID 100000 kern.notice]<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 672855 kern.notice] syncing file systems&#8230;<br \/>\nAug 15 22:33:04 cybertron genunix: [ID 904073 kern.notice] done<\/div>\n<p>My efforts to find any further detailes about this bug are ongoing, so at this point I have booted back into 2008.11 and I will be running that until a fix or a workaround is found.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>After successfully completing a &#8216;zfs replace&#8217; I was not so pleased to get the following error message back from &#8216;zfs detach&#8217;: cannot detach c5t17d0: no valid replicas I decided that I would upgrade this OpenSolaris 2008.11 instance to OpenSolaris 2009.06 in order to see if the obvious bug that I was encountering was resolved in [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[13,14],"tags":[],"_links":{"self":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts\/887"}],"collection":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/comments?post=887"}],"version-history":[{"count":17,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts\/887\/revisions"}],"predecessor-version":[{"id":900,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts\/887\/revisions\/900"}],"wp:attachment":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/media?parent=887"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/categories?post=887"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/tags?post=887"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}