Server outage keeps occuring - lots of php-cgi processes ?

PaulK

Hi Guys, can anyone answer this, my server stops responding to http requests after a number of hours.
Everything seems to be up, but there seems to be a lot of php-cgi processes.
Here's a snip from the fast cgi log

19/Nov/2006:19:37:54 +0000	SUID: 49	PUID: 49	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:19:42:55 +0000	SUID: 51	PUID: 51	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:19:42:55 +0000	SUID: 51	PUID: 51	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:19:48:01 +0000	SUID: 52	PUID: 52	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:19:48:01 +0000	SUID: 52	PUID: 52	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:19:53:01 +0000	SUID: 54	PUID: 54	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:19:53:01 +0000	SUID: 54	PUID: 54	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:19:58:04 +0000	SUID: 56	PUID: 56	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:19:58:04 +0000	SUID: 56	PUID: 56	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:20:03:06 +0000	SUID: 58	PUID: 58	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:20:03:06 +0000	SUID: 58	PUID: 58	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:20:18:13 +0000	SUID: 60	PUID: 60	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:20:18:13 +0000	SUID: 60	PUID: 60	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:20:33:21 +0000	SUID: 62	PUID: 62	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:20:33:21 +0000	SUID: 62	PUID: 62	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:20:43:27 +0000	SUID: 64	PUID: 64	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:20:43:27 +0000	SUID: 64	PUID: 64	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:20:48:29 +0000	SUID: 66	PUID: 66	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:20:48:29 +0000	SUID: 66	PUID: 66	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:20:53:32 +0000	SUID: 68	PUID: 68	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:20:53:32 +0000	SUID: 68	PUID: 68	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:20:58:33 +0000	SUID: 69	PUID: 69	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:20:58:33 +0000	SUID: 69	PUID: 69	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:21:03:35 +0000	SUID: 71	PUID: 71	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:21:03:35 +0000	SUID: 71	PUID: 71	RUID: 0	URI: 	timeout-header2!
19/Nov/2006:21:08:37 +0000	SUID: 72	PUID: 72	RUID: 0	URI: 	Reading 56 bytes failed = The process cannot access the file because it is being used by another process.
19/Nov/2006:21:08:37 +0000	SUID: 72	PUID: 72	RUID: 0	URI: 	timeout-header2!

and here's a snip from the website log

89.234.6.166 - - [19/Nov/2006:20:43:27 +0000] "HEAD / HTTP/1.0" 200 0 "" "SiteUptime.com"
89.234.6.166 - - [19/Nov/2006:20:48:32 +0000] "HEAD / HTTP/1.0" 200 0 "" "SiteUptime.com"
89.234.6.166 - - [19/Nov/2006:20:53:32 +0000] "HEAD / HTTP/1.0" 200 0 "" "SiteUptime.com"
89.234.6.166 - - [19/Nov/2006:20:58:33 +0000] "HEAD / HTTP/1.0" 200 0 "" "SiteUptime.com"
66.249.72.84 - - [19/Nov/2006:21:01:35 +0000] "GET /robots.txt HTTP/1.1" 200 216 "" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.84 - - [19/Nov/2006:21:01:35 +0000] "GET /index.php?name=PNphpBB2&file=viewtopic&t=25&start=0&postdays=0&postorder=asc&highlight= HTTP/1.1" 200 44512 "" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
89.234.6.166 - - [19/Nov/2006:21:03:35 +0000] "HEAD / HTTP/1.0" 200 0 "" "SiteUptime.com"
202.108.11.106 - - [19/Nov/2006:21:05:35 +0000] "GET / HTTP/1.1" 200 9216 "" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
131.107.0.96 - - [19/Nov/2006:21:06:16 +0000] "GET /modules/AutoTheme/style/style.css HTTP/1.1" 200 26 "http://www.mywebsite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)"
131.107.0.96 - - [19/Nov/2006:21:06:25 +0000] "GET /themes/GP-Theme-safe/style/styleNN.css HTTP/1.1" 200 0 "http://www.mywebsite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)"
131.107.0.96 - - [19/Nov/2006:21:06:26 +0000] "GET /javascript/showimages.js HTTP/1.1" 200 1513 "http://www.mywebsite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)"
131.107.0.96 - - [19/Nov/2006:21:06:28 +0000] "GET /themes/GP-Theme-safe/style/style.css HTTP/1.1" 200 2819 "http://www.mywebsite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)"
131.107.0.96 - - [19/Nov/2006:21:06:28 +0000] "GET /javascript/openwindow.js HTTP/1.1" 200 409 "http://www.mywebsite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)"
131.107.0.96 - - [19/Nov/2006:21:06:28 +0000] "GET /themes/GP-Theme-safe/includes/javascript.js HTTP/1.1" 200 1460 "http://www.mywebsite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)"
131.107.0.96 - - [19/Nov/2006:21:06:31 +0000] "GET /themes/GP-Theme-safe/includes/liveclock_lite.js HTTP/1.1" 200 2706 "http://www.mywebsite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)"
131.107.0.96 - - [19/Nov/2006:21:06:03 +0000] "GET / HTTP/1.1" 200 26804 "http://search.live.com/result.aspx?q=www&mrt=en-us&FORM=LVSP" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; WOW64; SV1; .NET CLR 2.0.50727)"

Any pointers?
Thanks
Paul

AbyssUnderground

My server sometimes does this and I think its caused by a fastCGI bug and I beleive Aprelium is developing a workaround.
I implemented a remote-restart function which still seems to work even if the site is unresponsive. It runs a php script, which launches a bat file, which kills all fastcgi.exe's, then restarts the Abyss Web Server service. Works a treat.

PaulK

Well in a way I guess I'm pleased that this is a problem that has cropped up before. Just odd that it has started doing it over the weekend.
Any chance of you code Andy?

AbyssUnderground

Remind me tonight and Ill post the code.

PaulK

Happened again over night, 20 php-cgi processes, then server when down.
Spending a lot of money on SiteUpTime SMS messages :(

PaulK

Found you code here Andy, looks like you've helped on this before!
http://www.aprelium.com/forum/viewtopic.php?t=9040
Have sent logs and conf to aprelium and switch to std cgi for the time being.
Cheers Paul K

aprelium

PaulK,
A fix for this problem has been implemented in the coming version 2.4. Switching to CGI is the best solution until we make available the new release.

PaulK

Thanks for the update aprelium :)

AbyssUnderground

www.abyssunderground.co.uk/restart.zip
There is my code. Simply extract it to a folder you know the name of and then type www.yoururl.com/folder_name_you_chose and it should work.
Remember you need to run Abyss as a service for this to work. Editing to the bat file might be required since mine does php.exe not php-cgi.exe. Might be worth adding both :-)

TRUSTAbyss

I could write a PHP CRON program that can be run in the background that
monitors the number of PHP processes running. The function could kill the
PHP processes. Cool Huh? Anyone interested?
Note: This function can kill only a certain anount of PHP processes by
reading the PID from the tasklist command.

Daisyanne

i dont know what to do. no php will run on it.
im using 2.6

b/2009:02:24:32 -0500	SUID: 5	PUID: 5	RUID: 0	URI: 	Reading 8 bytes failed = The process cannot access the file because it is being used by another process.
08/Feb/2009:02:24:32 -0500	SUID: 5	PUID: 5	RUID: 0	URI: 	timeout-header1!
08/Feb/2009:02:24:32 -0500	SUID: 5	PUID: 5	RUID: 1	URI: /index.php	connect() = The system cannot find the file specified.
08/Feb/2009:02:27:11 -0500	SUID: 5	PUID: 5	RUID: 0	URI: 	Reading 8 bytes failed = The process cannot access the file because it is being used by another process.
08/Feb/2009:02:27:11 -0500	SUID: 5	PUID: 5	RUID: 0	URI: 	timeout-header1!
08/Feb/2009:02:27:11 -0500	SUID: 5	PUID: 5	RUID: 1	URI: /index.php	connect() = The system cannot find the file specified.
08/Feb/2009:02:27:38 -0500	SUID: 6	PUID: 6	RUID: 0	URI: 	Reading 8 bytes failed = The process cannot access the file because it is being used by another process.
08/Feb/2009:02:27:38 -0500	SUID: 6	PUID: 6	RUID: 0	URI: 	timeout-header1!
08/Feb/2009:02:27:38 -0500	SUID: 6	PUID: 6	RUID: 1	URI: /index.php	connect() = The system cannot find the file specified.
08/Feb/2009:02:27:48 -0500	SUID: 7	PUID: 7	RUID: 0	URI: 	Reading 8 bytes failed = The process cannot access the file because it is being used by another process.
08/Feb/2009:02:27:48 -0500	SUID: 7	PUID: 7	RUID: 0	URI: 	timeout-header1!
08/Feb/2009:02:27:48 -0500	SUID: 7	PUID: 7	RUID: 1	URI: /index.php	connect() = The system cannot find the file specified.
08/Feb/2009:02:28:53 -0500	SUID: 12	PUID: 12	RUID: 0	URI: 	Reading 8 bytes failed = The process cannot access the file because it is being used by another process.
08/Feb/2009:02:28:53 -0500	SUID: 12	PUID: 12	RUID: 0	URI: 	timeout-header1!
08/Feb/2009:02:28:53 -0500	SUID: 12	PUID: 12	RUID: 1	URI: /index.php	connect() = The system cannot find the file specified.
08/Feb/2009:02:29:05 -0500	SUID: 13	PUID: 13	RUID: 0	URI: 	Reading 8 bytes failed = The process cannot access the file because it is being used by another process.
08/Feb/2009:02:29:05 -0500	SUID: 13	PUID: 13	RUID: 0	URI: 	timeout-header1!
08/Feb/2009:02:29:05 -0500	SUID: 13	PUID: 13	RUID: 1	URI: /index.php	connect() = The system cannot find the file specified.
08/Feb/2009:02:33:53 -0500	SUID: 26	PUID: 26	RUID: 0	URI: 	Reading 8 bytes failed = The process cannot access the file because it is being used by another process.
08/Feb/2009:02:33:53 -0500	SUID: 26	PUID: 26	RUID: 0	URI: 	timeout-header1!
08/Feb/2009:02:33:53 -0500	SUID: 26	PUID: 26	RUID: 1	URI: /index.php	connect() = The system cannot find the file specified.
08/Feb/2009:02:34:29 -0500	SUID: 27	PUID: 27	RUID: 0	URI: 	Reading 8 bytes failed = The process cannot access the file because it is being used by another process.
08/Feb/2009:02:34:29 -0500	SUID: 27	PUID: 27	RUID: 0	URI: 	timeout-header1!
08/Feb/2009:02:34:29 -0500	SUID: 27	PUID: 27	RUID: 1	URI: /index.php	connect() = The system cannot find the file specified.
08/Feb/2009:02:34:35 -0500	SUID: 28	PUID: 28	RUID: 0	URI: 	Reading 8 bytes failed = The process cannot access the file because it is being used by another process.
08/Feb/2009:02:34:35 -0500	SUID: 28	PUID: 28	RUID: 0	URI: 	timeout-header1!
08/Feb/2009:02:34:35 -0500	SUID: 28	PUID: 28	RUID: 1	URI: /index.php	connect() = The system cannot find the file specified.

puertoblack2003

I'm having that same issue i may have to revert back to 2.5 never had that problem

aprelium

puertoblack2003 wrote
I'm having that same issue i may have to revert back to 2.5 never had that problem

That error should be ignored as explained above (turn FastCGI logging off if you do not need it).
By the way, 2.6 has not changed anything regarding FastCGI processing. It is the same module as the one which ships with 2.5.

puertoblack2003

aprelium wrote

puertoblack2003 wrote
I'm having that same issue i may have to revert back to 2.5 never had that problem

That error should be ignored as explained above (turn FastCGI logging off if you do not need it).
By the way, 2.6 has not changed anything regarding FastCGI processing. It is the same module as the one which ships with 2.5.

i take that back..I keep forgetting to disable apc on my test server.I always test product out before i go live..Its good now..my apology.