[IMC-Tech] Re: rewrite rules

[dan blah]

On 10/1/06, Mike Lehman <rebelmike at earthlink.net> wrote:
> I want to also express my appreciation for the work being done by
> everyone on this big, but very much needed project.
>
> Here's a few suggestions from the user perspective.
>
> We've already agreed that the new site needs to be up, due to spam, etc.
> So if the old Dada site needs to go down for the time being, as long as
> the new site goes up in its place at www.ucimc.org, that's cool. And
> very much needed. There are lots of things going on in our community
> right now, so we badly need to get something usable up where people are
> used to finding it.
>
> It will be good to be able to access the old site, sooner rather than
> later, so if Dan can get it going on an isolated server where it won't
> continue to cause problems and can be easily diagnosed, that sounds like
> a plan. In the meantime, we can manage without it, as long as people
> find the new site in its place.
>
i have been trolling around the indymedia global server and have found
there is a dada2drupal script being written to import old dada info.
i havent found this yet but am digging.
> AFAIK, the patches on Dada that were made by Clint were due to the
> legacy system (SFActive?) needing to be patched into Dada. Is it
> possible to just pull from the Dada database since then and do a
> relatively clean and easy update to whatever the latest build of Dada is
> with just that part of the database? If so, then we can leave the oldest
> part of ucimc.org -- the one that was patched in and has most of the
> potential issues that might snag a Dada update -- for last. I think
> we're talking about just the first two years, as I think the last four
> years were since the update to Dada was done.
>
> But maybe I don't know WTF I'm talking about with that.
>
> Anyway, I'm just saying that people are badly itching for a usable site
> ASAP -- and the rest can come along when it's figured out. No one is
> going to mind if archives of older stuff aren't available right away.
> Regards,
> Mike Lehman
>
> dan blah wrote:
> > On 10/1/06, Zachary C. Miller <zach at zarfmouse.us> wrote:
> >> The reason I harp on this so much is that it is the ROOT cause of all
> >> the resource problems that imsahp has. System response too slow for
> >> you? Mail gets backed up on the first of the month? Hourly downtime
> >> annoying? Some web requests hang forever? These are all rooted in the
> >> apache hangs. Once we solve the problem of the apache hangs there will
> >> be a TON of available CPU power available and a definite increase in
> >> speed and stability of all services on imsahp.
> >>
> >> Anyone who is inspired to pour a ton of effort into helping fix the
> >> problems that imsahp has really needs to pour their efforts into
> >> solving this problem first and foremost. Moving to Drupal is a start.
> >> Moving everything to Apache2 is a start. But as long as we're
> >> continuing to run Dada in any form (or on the off chance that we
> >> discover that Dada is not the cause) we have to be prepared to
> >> continue to deal with the issue of apache hangs. If we can isolate
> >> various potential causes (upgrade to Apache2 so we can't blame old
> >> apache, upgrade to php5 so we can't blame old php, move Dada off the
> >> system entirely so we can't blame Dada, stop even loading mod_perl so
> >> we can't blame mod_perl, etc) we will figure out what the real cause
> >> is. We're on the right track, I just want to make sure we're all on
> >> the same page about what the track is.
> >>
> > my plan is to do just this but after we get the old dada site working
> > on the new server.  as there are no resource issues at all on that
> > server with most all of hosted sites working its a fairly safe
> > assumption that apache issues on imsahp are related to the current
> > ucimc dada installation.  provided we can get the old dada install
> > work on the new server w/o causing any resource issues that pretty
> > much solves our problems.  from there i will start going through the
> > steps you have have listed just for the sake of knowing (i am curious
> > exactly what the problem is).
> >
> >> Another backwards compatibility issue we have to consider: When Clint
> >> ported us to Dada, he made sure that old URLs to the original "IMC
> >> active" software still worked. I don't know if he did that by leaving
> >> a crippled install of active in place or by porting all the old
> >> articles into Dada and putting in a forwarding script, I've never
> >> looked into it. We'll need to figure that out (or maybe I'm just
> >> imagining that he ever even did it) and make sure those old old URLs
> >> also remain valid if possible.
> >>
> > as for the rewite script, awesome.  i cped your pub key on imsahp to
> > zeco.chambana.net (new web server).
> >> On 10/1/06, Zachary C. Miller <zach at zarfmouse.us> wrote:
> >> > Note to all: please understand this well: read this all and don't skim
> >> > it...I feel like I've explained this problem in detail before and
> >> > people seem to still be making other theories about what is happening
> >> > without having addressed what I know to be the facts of the problem.
> >> > I'm not trying to be snarky, I'm EXTREMELY grateful that so many
> >> > people (especially David and Dan!!!! Thanks!!!!) are stepping forward
> >> > to take care of imsahp, I don't have time to do it and you guys are
> >> > the lifeline that the system needs. I'm just trying to make sure that
> >> > we REALLY understand the problem that needs to be solved, rather than
> >> > taking wild guesses about what will fix it.
> >> >
> >> > The primary bog down that Dada seems to be causing (it might not be
> >> > Dada, it might be something else) on imsahp is that apache processes
> >> > become STUCK in a loop that consumes 100% of the available CPU and
> >> > never times out. The stuck processes build up until the server is so
> >> > bogged down that it is unusable. Nothing gets logged when apache
> >> > becomes stuck in this manner so it is impossible to know for sure
> >> > whether Dada is causing the problem but a lot of circumstantial clues
> >> > suggest Dada is the cause.
> >> >
> >> > PHP is embedded in Apache so I believe the stuck processes are due to
> >> > Dada exposing some bug in PHP. Even badly written PHP code should time
> >> > out eventually unless there is a bug in PHP. This is why, even though
> >> > I believe the problem is caused by Dada, I've advocated in the past
> >> > for an upgrade of PHP and of Apache (as well as an upgrade of Dada) to
> >> > see if the bug has already been fixed in a newer version. These
> >> > upgrades are neccessary anyway for security and stability and
> >> > supportability purposes.
> >> >
> >> > I suspect (this is WILD speculation) that the problem might be related
> >> > to memory allocation.  Some OJC people have told me they've seen
> >> > similar problems with older version of mod_perl and that they were
> >> > memory allocation related. (And honestly, if someone could check all
> >> > the websites we host and confirm that none of them are using mod_perl,
> >> > I'd suggest disabling mod_perl and see if that helps).
> >> >
> >> > I do not believe the problem is related to rebuilding of indexes
> >> > because it is Apache and not MySQL that sucks up the CPU time. The
> >> > stuck apache processes do not respond to normal kill signals (they
> >> > must kill -9'ed) so we're really talking about the exposure of a BUG
> >> > and not just some rogue code that takes a lot of power to run.
> >> >
> >> > Note that there are 3 such stuck processes on imsahp at this very
> >> > moment as I write this email. They'll go away as soon as the hourly
> >> > restart of apache happens. Then they'll inevitably come back over the
> >> > next hour. If you guys have stopped posting on Dada, it clearly has
> >> > NOT stopped the stuck apache process problem. It will continue to
> >> > plague us on a system that is running Dada in "archive" mode.
> >> >
> >> > I am pretty sure that I have seen the stuck process thing happen while
> >> > I was making a web request to simply read the index page (not posting
> >> > an article) of ucimc.
> >> >
> >> > Curently to address this problem there is a root cronjob that restarts
> >> > apache every hour. This causes about 5 minutes of downtime for every
> >> > site on the server every hour at 51 minutes past the hour. This is a
> >> > terrible hackish stop gap that has allowed the server to survive but
> >> > the server is still constantly a bit bogged down by a few of these
> >> > rogue apache processes. Performance on the server will skyrocket when
> >> > this fundamental apache problem is solved.
> >> > 51 * * * * /home/wolfgang/bin/apache-restart >>
> >> /var/log/apache-restart.log 2>&1
> >> >
> >> > If we're going to have an "archive" version of dada, it needs to run
> >> > on it's own separate dedicated server. Then we can isolate Dada and
> >> > confirm that it is the culprit causing the apache hangs and if it is
> >> > we can rest assured that it won't take the rest of imsahp down with
> >> > it. We might learn from this experiment that Dada has nothing to do
> >> > with causing the Apache hangs...but it'd be nice to learn that. We
> >> > need to use a process of elimination to figure out what is triggering
> >> > the apache hangs.
> >> >
> >> > I can help configure rewrite rules if someone can set up the
> >> archive server.
> >> >
> >> > On 9/27/06, David Gehrig <gehrigspamtrap at gmail.com> wrote:
> >> > > It seems to me that post-less dada isn't bogging down the system
> >> the way it
> >> > > usually does -- that dada's problem might be something to do with
> >> > > regenerating indexes or something else cycle-sucking every time
> >> someone
> >> > > posts, and that the spamming was therefore bringing it -- and
> >> everything
> >> > > else on imsahp -- to its knees.
> >> > >
> >> > > If there's agreement on that, then it's time to switch to the new
> >> site.
> >> > >
> >> > > I am not the master of rewrite rules, though.  Dan, could you set up
> >> > > rewrites to do what I've described in step (3) below?   The idea
> >> is that any
> >> > > URL that goes to www.ucimc.org/newswire/.... goes instead to
> >> > > archive.ucimc.org/newswire... etc.
> >> > >
> >> > >
> >> > > On 9/26/06, David Gehrig <gehrigspamtrap at gmail.com> wrote:
> >> > > > All right, here's my so-called thoughts.
> >> > > >
> >> > > > (1) I'm really very reluctant to try to find a way to graunch
> >> > > > the dada data into drupal.  I don't think it can be done well,
> >> > > > and I don't have the time to waste doing it badly.
> >> > > >
> >> > > > (2) Let's watch the lobotomized dada site for a few more
> >> > > > bounces -- if the drain on the system was related to the
> >> > > > process of posting to dada, then we should see better
> >> > > > response all around now that it no longer is.  Early signs
> >> > > > look good.
> >> > > >
> >> > > > (3) Here's a little more detail of what I'm suggesting.
> >> > > >
> >> > > > If a URL request comes in for
> >> > > >
> >> > > > www.ucimc.org/newswire/....
> >> > > > www.ucimc.org/mod/info/...
> >> > > > www.ucimc.org/editor/...
> >> > > > www.ucimc.org/library/...
> >> > > >
> >> > > > then redirect it to archive.ucimc.org.
> >> > > >
> >> > > > Otherwise have it go to the drupal site.
> >> > > >
> >> > > > I'm assuming this would have to be done in the main
> >> > > > apache config file. If this passes a basic sanity check, let's
> >> try it.
> >> > > >
> >> > > > @%<
> >> > > >
> >> > >
> >> > >
> >> > > _______________________________________________
> >> > > IMC-Tech mailing list
> >> > > IMC-Tech at lists.ucimc.org
> >> > > http://lists.chambana.net/cgi-bin/listinfo/imc-tech
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Zachary C. Miller - @= - http://zach.chambana.net/
> >> > IMSA 1995 - UIUC 2000 - Just Another Leftist Muppet - Ya Basta!
> >> >  Social Justice, Community, Nonviolence, Decentralization, Feminism,
> >> >  Sustainability, Responsibility, Diversity, Democracy, Ecology
> >> >
> >>
> >>
> >> --
> >> Zachary C. Miller - @= - http://zach.chambana.net/
> >> IMSA 1995 - UIUC 2000 - Just Another Leftist Muppet - Ya Basta!
> >>  Social Justice, Community, Nonviolence, Decentralization, Feminism,
> >>  Sustainability, Responsibility, Diversity, Democracy, Ecology
> >> _______________________________________________
> >> IMC-Tech mailing list
> >> IMC-Tech at lists.ucimc.org
> >> http://lists.chambana.net/cgi-bin/listinfo/imc-tech
> >>
> >
> >
>
> _______________________________________________
> IMC-Tech mailing list
> IMC-Tech at lists.ucimc.org
> http://lists.chambana.net/cgi-bin/listinfo/imc-tech
>


-- 
Daniel