[IMC-Tech] Re: rewrite rules

[Zachary C. Miller]

Note to all: please understand this well: read this all and don't skim
it...I feel like I've explained this problem in detail before and
people seem to still be making other theories about what is happening
without having addressed what I know to be the facts of the problem.
I'm not trying to be snarky, I'm EXTREMELY grateful that so many
people (especially David and Dan!!!! Thanks!!!!) are stepping forward
to take care of imsahp, I don't have time to do it and you guys are
the lifeline that the system needs. I'm just trying to make sure that
we REALLY understand the problem that needs to be solved, rather than
taking wild guesses about what will fix it.

The primary bog down that Dada seems to be causing (it might not be
Dada, it might be something else) on imsahp is that apache processes
become STUCK in a loop that consumes 100% of the available CPU and
never times out. The stuck processes build up until the server is so
bogged down that it is unusable. Nothing gets logged when apache
becomes stuck in this manner so it is impossible to know for sure
whether Dada is causing the problem but a lot of circumstantial clues
suggest Dada is the cause.

PHP is embedded in Apache so I believe the stuck processes are due to
Dada exposing some bug in PHP. Even badly written PHP code should time
out eventually unless there is a bug in PHP. This is why, even though
I believe the problem is caused by Dada, I've advocated in the past
for an upgrade of PHP and of Apache (as well as an upgrade of Dada) to
see if the bug has already been fixed in a newer version. These
upgrades are neccessary anyway for security and stability and
supportability purposes.

I suspect (this is WILD speculation) that the problem might be related
to memory allocation.  Some OJC people have told me they've seen
similar problems with older version of mod_perl and that they were
memory allocation related. (And honestly, if someone could check all
the websites we host and confirm that none of them are using mod_perl,
I'd suggest disabling mod_perl and see if that helps).

I do not believe the problem is related to rebuilding of indexes
because it is Apache and not MySQL that sucks up the CPU time. The
stuck apache processes do not respond to normal kill signals (they
must kill -9'ed) so we're really talking about the exposure of a BUG
and not just some rogue code that takes a lot of power to run.

Note that there are 3 such stuck processes on imsahp at this very
moment as I write this email. They'll go away as soon as the hourly
restart of apache happens. Then they'll inevitably come back over the
next hour. If you guys have stopped posting on Dada, it clearly has
NOT stopped the stuck apache process problem. It will continue to
plague us on a system that is running Dada in "archive" mode.

I am pretty sure that I have seen the stuck process thing happen while
I was making a web request to simply read the index page (not posting
an article) of ucimc.

Curently to address this problem there is a root cronjob that restarts
apache every hour. This causes about 5 minutes of downtime for every
site on the server every hour at 51 minutes past the hour. This is a
terrible hackish stop gap that has allowed the server to survive but
the server is still constantly a bit bogged down by a few of these
rogue apache processes. Performance on the server will skyrocket when
this fundamental apache problem is solved.
51 * * * * /home/wolfgang/bin/apache-restart >> /var/log/apache-restart.log 2>&1

If we're going to have an "archive" version of dada, it needs to run
on it's own separate dedicated server. Then we can isolate Dada and
confirm that it is the culprit causing the apache hangs and if it is
we can rest assured that it won't take the rest of imsahp down with
it. We might learn from this experiment that Dada has nothing to do
with causing the Apache hangs...but it'd be nice to learn that. We
need to use a process of elimination to figure out what is triggering
the apache hangs.

I can help configure rewrite rules if someone can set up the archive server.

On 9/27/06, David Gehrig <gehrigspamtrap at gmail.com> wrote:
> It seems to me that post-less dada isn't bogging down the system the way it
> usually does -- that dada's problem might be something to do with
> regenerating indexes or something else cycle-sucking every time someone
> posts, and that the spamming was therefore bringing it -- and everything
> else on imsahp -- to its knees.
>
> If there's agreement on that, then it's time to switch to the new site.
>
> I am not the master of rewrite rules, though.  Dan, could you set up
> rewrites to do what I've described in step (3) below?   The idea is that any
> URL that goes to www.ucimc.org/newswire/.... goes instead to
> archive.ucimc.org/newswire... etc.
>
>
> On 9/26/06, David Gehrig <gehrigspamtrap at gmail.com> wrote:
> > All right, here's my so-called thoughts.
> >
> > (1) I'm really very reluctant to try to find a way to graunch
> > the dada data into drupal.  I don't think it can be done well,
> > and I don't have the time to waste doing it badly.
> >
> > (2) Let's watch the lobotomized dada site for a few more
> > bounces -- if the drain on the system was related to the
> > process of posting to dada, then we should see better
> > response all around now that it no longer is.  Early signs
> > look good.
> >
> > (3) Here's a little more detail of what I'm suggesting.
> >
> > If a URL request comes in for
> >
> > www.ucimc.org/newswire/....
> > www.ucimc.org/mod/info/...
> > www.ucimc.org/editor/...
> > www.ucimc.org/library/...
> >
> > then redirect it to archive.ucimc.org.
> >
> > Otherwise have it go to the drupal site.
> >
> > I'm assuming this would have to be done in the main
> > apache config file. If this passes a basic sanity check, let's try it.
> >
> > @%<
> >
>
>
> _______________________________________________
> IMC-Tech mailing list
> IMC-Tech at lists.ucimc.org
> http://lists.chambana.net/cgi-bin/listinfo/imc-tech
>
>
>


--
Zachary C. Miller - @= - http://zach.chambana.net/
IMSA 1995 - UIUC 2000 - Just Another Leftist Muppet - Ya Basta!
 Social Justice, Community, Nonviolence, Decentralization, Feminism,
 Sustainability, Responsibility, Diversity, Democracy, Ecology