l i n u x - u s e r s - g r o u p - o f - d a v i s
L U G O D
 
Next Meeting:
December 2: Social gathering
Next Installfest:
TBD
Latest News:
Nov. 18: Club officer elections
Page last updated:
2010 Nov 30 13:18

The following is an archive of a post made to our 'vox-tech mailing list' by one of its subscribers.

Report this post as spam:

(Enter your email address)
[vox-tech] php-cgi hanging on semop()
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[vox-tech] php-cgi hanging on semop()



So I'm noticing some stability issues with a website.

* lighttpd 1.4.x
* php 5.3.x
* mysql 5.1.x

The symptoms to the outside world include lagging connections and,
eventually, Error 500s from the webserver.

Internally, I run "mytop" (basically, a 'top'-like interface that
parses the results of a "show full processlist" query).  mytop shows
me what connections are active.

Typically, most of our connections are relatively active (only idle a
couple of seconds, at most; If the website's not very busy, the idle
time in some of them can climb in idleness).

However, in the situation I'm seeing, all of the connections from the
webserver are idle, and their idle times continue to increase.
Basically, the current connections between the website and the
database aren't doing anything (not running queries for the website),
and are just sitting there exhausting MySQL's connection pool.

Since this has only started happening recently, and usually after-hours,
my quick-fix has been to do this:

  sudo /etc/init.d/lighttpd stop   # stop webserver
  sudo killall php-cgi             # kill all php-cgi processes
  sudo killall -2 php-cgi          # no, REALLY kill them
  sudo /etc/init.d/lighttpd start  # restart webserver

And things have typically been fine for at least another day.


Today, it has happend a _FEW_ times during business hours, so I
get to drop everything and investigate.  In my job, I am a
literal Jack of All Trades, Master of None.  (On a good day,
I spend my time doing web app development, so I suppose that's
my "main" job ;) )

Google hasn't helped me much, sadly, so I've decided to post here :)


Briefly, here's what I get if I try to attach an strace to
one of the running php-cgi processes.  Note: I see the
same thing:

  - before stopping lighttpd
  - after stopping lighttpd, but before 'killall php-cgi'
  - after 'killall php-cgi', but (obviously), before 'killall -2...'

So here's the clue, gang.  Like, ZOINKS!

  Process 17180 attached - interrupt to quit
  semop(229383, {{0, -1, SEM_UNDO}}, 1^C <unfinished ...>
  Process 17180 detached                                 


In my investigation, I've learned that "ipcs" might be my
friend.  Sadly(?), the server's been chugging along fine
since the last restart.  In the meantime, in case it helps,
here are some logs from /var/log/lighttpd/error.log

[crash! restart]

  2010-11-24 15:36:07: (log.c.166) server started
  2010-11-24 15:38:00: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 17185 socket: unix:/tmp/php.socket-2
  2010-11-24 15:38:00: (mod_fastcgi.c.3382) response already sent out, but backend returned error on socket: unix:/tmp/php.socket-2 for /index.php?, terminating connection
  2010-11-24 15:45:19: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 17170 socket: unix:/tmp/php.socket-0
  2010-11-24 15:45:19: (mod_fastcgi.c.3367) response not received, request sent: 991 on socket: unix:/tmp/php.socket-0 for /index.php?, closing connection
  2010-11-24 15:49:11: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 17178 socket: unix:/tmp/php.socket-1
  2010-11-24 15:49:11: (mod_fastcgi.c.3367) response not received, request sent: 851 on socket: unix:/tmp/php.socket-1 for /index.php?, closing connection
  2010-11-24 15:52:32: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 17178 socket: unix:/tmp/php.socket-1
  2010-11-24 15:52:32: (mod_fastcgi.c.3367) response not received, request sent: 923 on socket: unix:/tmp/php.socket-1 for /index.php?, closing connection
  2010-11-24 15:53:33: (server.c.1503) server stopped by UID = 0 PID = 21022

[crash! restart]

  2010-11-24 15:53:48: (log.c.166) server started
  2010-11-24 15:59:16: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 537
  2010-11-24 15:59:16: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 1 load: 537
  2010-11-24 15:59:16: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 2 load: 537
  2010-11-24 15:59:16: (mod_fastcgi.c.3011) backend is overloaded; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 3 load: 537
  2010-11-24 15:59:16: (mod_fastcgi.c.3608) all handlers for /index.php? on .php are down.
  2010-11-24 15:59:18: (mod_fastcgi.c.2774) fcgi-server re-enabled:  0 /tmp/php.socket
  2010-11-24 15:59:18: (mod_fastcgi.c.2774) fcgi-server re-enabled:  0 /tmp/php.socket
  2010-11-24 15:59:18: (mod_fastcgi.c.2774) fcgi-server re-enabled:  0 /tmp/php.socket
  2010-11-24 15:59:18: (mod_fastcgi.c.2774) fcgi-server re-enabled:  0 /tmp/php.socket

["backend is overloaded", "all handlers... are down", "fcgi-server re-enabled"
 repeats a bunch of times...]

  2010-11-24 16:01:37: (mod_fastcgi.c.3608) all handlers for /index.php? on .php are down.
  2010-11-24 16:01:37: (server.c.1503) server stopped by UID = 0 PID = 23434
  2010-11-24 16:01:37: (log.c.166) server started
  2010-11-24 16:07:26: (mod_fastcgi.c.2582) unexpected end-of-file (perhaps the fastcgi process died): pid: 23459 socket: unix:/tmp/php.socket-0
  2010-11-24 16:07:26: (mod_fastcgi.c.3367) response not received, request sent: 1091 on socket: unix:/tmp/php.socket-0 for /index.php?, closing connection
  2010-11-24 16:08:29: (connections.c.294) SSL: 1 error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request

[so far, followed by some repearts of the "mod_fastcgi" errors...]


Any ideas?  Suggestions for next steps to determine the
cause of the issue?  In the meantime, I'm off to learn
about "ipcs" so that I'm prepared to weild it as a weapon
the next time this happens. :)

PS - Bonus hint, the closest thing I've seen to a similar
issue that somehow had seemed to have to do with SSL mutex cache.

Thanks in advance!

-- 
-bill!
Sent from my computer
_______________________________________________
vox-tech mailing list
vox-tech@lists.lugod.org
http://lists.lugod.org/mailman/listinfo/vox-tech



LinkedIn
LUGOD Group on LinkedIn
Sign up for LUGOD event announcements
Your email address:
facebook
LUGOD Group on Facebook
'Like' LUGOD on Facebook:

Hosting provided by:
Sunset Systems
Sunset Systems offers preconfigured Linux systems, remote system administration and custom software development.

LUGOD: Linux Users' Group of Davis
PO Box 2082, Davis, CA 95617
Contact Us

LUGOD is a 501(c)7 non-profit organization
based in Davis, California
and serving the Sacramento area.
"Linux" is a trademark of Linus Torvalds.

Sponsored in part by:
O'Reilly and Associates
For numerous book donations.