[Imc-web] Re: [Imc-tech] Re: Site down?

[Zachary C. Miller]

There is some, as yet unidentified, but in SOMETHING that our site
serves that causes apache processes to get stuck in a busy loop where
they suck up a ton of CPU and stop responding to requests. This is
happening pretty much CONSTANTLY.

We have enough apache children running that most of the time this just
means a low level drain on the CPU and the occasionally hit to the
site will segfault or hang, requiring a second click. But eventually
the broken processes build up to the point where the site is basically
down.

A cronjob automatically restarts apache every 5 hours so this
condition never lasts longer than that (but it also means there is a 2
minute downtime every 5 hours as the server restarts).

Arun and I have put some effort into identifying where this bug is. It
has persisted across MANY upgrades of Apache and PHP and the
Kernel. We think it must be in dada (but for all we know it could be
in ANY of the hundreds of other scripts that are hosted by this
server). We think that the bug is triggered BEFORE the logging
routines are called so we can't reliably analyze the log files to see
what kinds of hits are triggering the bug (though looking at
server-status it appears that many of the "hung" processes are simply
regular attempts by regular people to access the root of ucimc.org).

My remaining solution is to upgrade to Apache 2.0 and PHP5 and see if
the problem goes away. I have not had the time to do that yet (it's a
tall order because we host a LOT of vhosts and when I do it I need to
make sure that none of them have broken).

If that doesn't work, I will move ucimc.org off to another server and
see whether it is THAT server or this server that continues to be
broken, that'll tell us if dada is the culprit or not.

We've been hobbling along with the cronjob restarting apache for
MONTHS. I am INSANELY busy until after August 29th. I may find the
time to do an upgrade before then but I will DEFINITELY do it after
that if it isn't done yet.

Unfortunately upgrading to apache2 is one task I don't want to farm
out. But I am happy to facilitate anyone that wants to try to debug
what is wrong with the current configuration.

Daniel S. Lewart wrote:
> David,
> 
> > About the apache slowdown -- Dan, is this a fairly frequent occurence?
> > Any idea what causes it?
> 
> I don't know about either.  Zach may have a better idea.
> 
> Thanks!
> Dan
> _______________________________________________
> Imc-tech mailing list
> Imc-tech at lists.ucimc.org
> http://lists.chambana.net/cgi-bin/listinfo/imc-tech
> 

-- 
Zachary C. Miller - @= - http://zach.chambana.net/
IMSA 1995 - UIUC 2000 - Just Another Leftist Muppet - Ya Basta!
 Social Justice, Community, Nonviolence, Decentralization, Feminism,
 Sustainability, Responsibility, Diversity, Democracy, Ecology