[IMC-Tech] Re: rewrite rules

[Zachary C. Miller]

The reason I harp on this so much is that it is the ROOT cause of all
the resource problems that imsahp has. System response too slow for
you? Mail gets backed up on the first of the month? Hourly downtime
annoying? Some web requests hang forever? These are all rooted in the
apache hangs. Once we solve the problem of the apache hangs there will
be a TON of available CPU power available and a definite increase in
speed and stability of all services on imsahp.

Anyone who is inspired to pour a ton of effort into helping fix the
problems that imsahp has really needs to pour their efforts into
solving this problem first and foremost. Moving to Drupal is a start.
Moving everything to Apache2 is a start. But as long as we're
continuing to run Dada in any form (or on the off chance that we
discover that Dada is not the cause) we have to be prepared to
continue to deal with the issue of apache hangs. If we can isolate
various potential causes (upgrade to Apache2 so we can't blame old
apache, upgrade to php5 so we can't blame old php, move Dada off the
system entirely so we can't blame Dada, stop even loading mod_perl so
we can't blame mod_perl, etc) we will figure out what the real cause
is. We're on the right track, I just want to make sure we're all on
the same page about what the track is.

Another backwards compatibility issue we have to consider: When Clint
ported us to Dada, he made sure that old URLs to the original "IMC
active" software still worked. I don't know if he did that by leaving
a crippled install of active in place or by porting all the old
articles into Dada and putting in a forwarding script, I've never
looked into it. We'll need to figure that out (or maybe I'm just
imagining that he ever even did it) and make sure those old old URLs
also remain valid if possible.

On 10/1/06, Zachary C. Miller <zach at zarfmouse.us> wrote:
> Note to all: please understand this well: read this all and don't skim
> it...I feel like I've explained this problem in detail before and
> people seem to still be making other theories about what is happening
> without having addressed what I know to be the facts of the problem.
> I'm not trying to be snarky, I'm EXTREMELY grateful that so many
> people (especially David and Dan!!!! Thanks!!!!) are stepping forward
> to take care of imsahp, I don't have time to do it and you guys are
> the lifeline that the system needs. I'm just trying to make sure that
> we REALLY understand the problem that needs to be solved, rather than
> taking wild guesses about what will fix it.
>
> The primary bog down that Dada seems to be causing (it might not be
> Dada, it might be something else) on imsahp is that apache processes
> become STUCK in a loop that consumes 100% of the available CPU and
> never times out. The stuck processes build up until the server is so
> bogged down that it is unusable. Nothing gets logged when apache
> becomes stuck in this manner so it is impossible to know for sure
> whether Dada is causing the problem but a lot of circumstantial clues
> suggest Dada is the cause.
>
> PHP is embedded in Apache so I believe the stuck processes are due to
> Dada exposing some bug in PHP. Even badly written PHP code should time
> out eventually unless there is a bug in PHP. This is why, even though
> I believe the problem is caused by Dada, I've advocated in the past
> for an upgrade of PHP and of Apache (as well as an upgrade of Dada) to
> see if the bug has already been fixed in a newer version. These
> upgrades are neccessary anyway for security and stability and
> supportability purposes.
>
> I suspect (this is WILD speculation) that the problem might be related
> to memory allocation.  Some OJC people have told me they've seen
> similar problems with older version of mod_perl and that they were
> memory allocation related. (And honestly, if someone could check all
> the websites we host and confirm that none of them are using mod_perl,
> I'd suggest disabling mod_perl and see if that helps).
>
> I do not believe the problem is related to rebuilding of indexes
> because it is Apache and not MySQL that sucks up the CPU time. The
> stuck apache processes do not respond to normal kill signals (they
> must kill -9'ed) so we're really talking about the exposure of a BUG
> and not just some rogue code that takes a lot of power to run.
>
> Note that there are 3 such stuck processes on imsahp at this very
> moment as I write this email. They'll go away as soon as the hourly
> restart of apache happens. Then they'll inevitably come back over the
> next hour. If you guys have stopped posting on Dada, it clearly has
> NOT stopped the stuck apache process problem. It will continue to
> plague us on a system that is running Dada in "archive" mode.
>
> I am pretty sure that I have seen the stuck process thing happen while
> I was making a web request to simply read the index page (not posting
> an article) of ucimc.
>
> Curently to address this problem there is a root cronjob that restarts
> apache every hour. This causes about 5 minutes of downtime for every
> site on the server every hour at 51 minutes past the hour. This is a
> terrible hackish stop gap that has allowed the server to survive but
> the server is still constantly a bit bogged down by a few of these
> rogue apache processes. Performance on the server will skyrocket when
> this fundamental apache problem is solved.
> 51 * * * * /home/wolfgang/bin/apache-restart >> /var/log/apache-restart.log 2>&1
>
> If we're going to have an "archive" version of dada, it needs to run
> on it's own separate dedicated server. Then we can isolate Dada and
> confirm that it is the culprit causing the apache hangs and if it is
> we can rest assured that it won't take the rest of imsahp down with
> it. We might learn from this experiment that Dada has nothing to do
> with causing the Apache hangs...but it'd be nice to learn that. We
> need to use a process of elimination to figure out what is triggering
> the apache hangs.
>
> I can help configure rewrite rules if someone can set up the archive server.
>
> On 9/27/06, David Gehrig <gehrigspamtrap at gmail.com> wrote:
> > It seems to me that post-less dada isn't bogging down the system the way it
> > usually does -- that dada's problem might be something to do with
> > regenerating indexes or something else cycle-sucking every time someone
> > posts, and that the spamming was therefore bringing it -- and everything
> > else on imsahp -- to its knees.
> >
> > If there's agreement on that, then it's time to switch to the new site.
> >
> > I am not the master of rewrite rules, though.  Dan, could you set up
> > rewrites to do what I've described in step (3) below?   The idea is that any
> > URL that goes to www.ucimc.org/newswire/.... goes instead to
> > archive.ucimc.org/newswire... etc.
> >
> >
> > On 9/26/06, David Gehrig <gehrigspamtrap at gmail.com> wrote:
> > > All right, here's my so-called thoughts.
> > >
> > > (1) I'm really very reluctant to try to find a way to graunch
> > > the dada data into drupal.  I don't think it can be done well,
> > > and I don't have the time to waste doing it badly.
> > >
> > > (2) Let's watch the lobotomized dada site for a few more
> > > bounces -- if the drain on the system was related to the
> > > process of posting to dada, then we should see better
> > > response all around now that it no longer is.  Early signs
> > > look good.
> > >
> > > (3) Here's a little more detail of what I'm suggesting.
> > >
> > > If a URL request comes in for
> > >
> > > www.ucimc.org/newswire/....
> > > www.ucimc.org/mod/info/...
> > > www.ucimc.org/editor/...
> > > www.ucimc.org/library/...
> > >
> > > then redirect it to archive.ucimc.org.
> > >
> > > Otherwise have it go to the drupal site.
> > >
> > > I'm assuming this would have to be done in the main
> > > apache config file. If this passes a basic sanity check, let's try it.
> > >
> > > @%<
> > >
> >
> >
> > _______________________________________________
> > IMC-Tech mailing list
> > IMC-Tech at lists.ucimc.org
> > http://lists.chambana.net/cgi-bin/listinfo/imc-tech
> >
> >
> >
>
>
> --
> Zachary C. Miller - @= - http://zach.chambana.net/
> IMSA 1995 - UIUC 2000 - Just Another Leftist Muppet - Ya Basta!
>  Social Justice, Community, Nonviolence, Decentralization, Feminism,
>  Sustainability, Responsibility, Diversity, Democracy, Ecology
>


--
Zachary C. Miller - @= - http://zach.chambana.net/
IMSA 1995 - UIUC 2000 - Just Another Leftist Muppet - Ya Basta!
 Social Justice, Community, Nonviolence, Decentralization, Feminism,
 Sustainability, Responsibility, Diversity, Democracy, Ecology