{"id":1309,"date":"2012-06-26T22:58:54","date_gmt":"2012-06-27T03:58:54","guid":{"rendered":"http:\/\/www.shainmiley.com\/wordpress\/?p=1309"},"modified":"2012-06-26T23:09:15","modified_gmt":"2012-06-27T04:09:15","slug":"using-strace-to-debug-issues-with-apache","status":"publish","type":"post","link":"https:\/\/www.shainmiley.com\/wordpress\/2012\/06\/26\/using-strace-to-debug-issues-with-apache\/","title":{"rendered":"Using strace to debug issues with apache"},"content":{"rendered":"

Today I had to track down the cause of an issue we were having with a server where shortly after restarting the server, requests would start to hang, and the number of Apache processes seemed to be growing rather large, rather quickly.<\/p>\n

I started out using Apache’s mod_status to get some details about the state of each process.<\/p>\n

I noticed that many of the processes ended up \u00c2\u00a0in a ‘”W<\/code><\/strong>” \u00c2\u00a0or “Sending Reply” state. \u00c2\u00a0I choose a random Apache process and fired up ‘strace’ to try to get some more information:<\/p>\n

server7:\/root# strace -p 11574
\nProcess 11574 attached – interrupt to quit
\nflock(26, LOCK_EX <unfinished …><\/div>\n

This process was stuck waiting for an\u00c2\u00a0exclusive lock on some file. \u00c2\u00a0I used ‘readlink’ to find out the name of the file in question:<\/p>\n

server7:\/root# readlink \/proc\/11574\/fd\/26
\n\/mnt\/Pages\/xml\/0\/1\/list1055.xml<\/div>\n

Once I had the name of the file I used ‘lsof’ to see if there were any other processes trying to access that file as well:<\/p>\n

server7:\/root#lsof |grep\u00c2\u00a0list1055.xml
\nhttpd 11574 nobody 26w REG 0,31 4232 925874559 \/mnt\/Pages\/xml\/0\/1\/list1055.xml (storage1.npr.org:\/files\/data)
\nhttpd 11579 nobody 26w REG 0,31 4232 925874559 \/mnt\/Pages\/xml\/0\/1\/list1055.xml (storage1.npr.org:\/files\/data)
\nhttpd 11629 nobody 26w REG 0,31 4232 925874559 \/mnt\/Pages\/xml\/0\/1\/list1055.xml (storage1.npr.org:\/files\/data)<\/div>\n

Here we have several other process waiting for an\u00c2\u00a0exclusive lock on the file as well.<\/p>\n

At this point it appears as though a recent code change maybe the cause of this issue…however a closer look at the recent source code commits will be required to know for sure.<\/p>\n","protected":false},"excerpt":{"rendered":"

Today I had to track down the cause of an issue we were having with a server where shortly after restarting the server, requests would start to hang, and the number of Apache processes seemed to be growing rather large, rather quickly. I started out using Apache’s mod_status to get some details about the state […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[4,3,1],"tags":[],"_links":{"self":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts\/1309"}],"collection":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/comments?post=1309"}],"version-history":[{"count":20,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts\/1309\/revisions"}],"predecessor-version":[{"id":1327,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/posts\/1309\/revisions\/1327"}],"wp:attachment":[{"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/media?parent=1309"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/categories?post=1309"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shainmiley.com\/wordpress\/wp-json\/wp\/v2\/tags?post=1309"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}