[IMC-Tech] Re: rewrite rules

[Joe Futrelle]

The Greens need a usable hosting service *yesterday*--our mailing  
lists and sites have been down repeatedly over the past few weeks,  
exactly when we're using them to coordinate time-critical activities  
and get information out about our campaigns. It looks really bad when  
we pass out flyers with a web address and then when people try to go  
there it's unreachable.

I understand that dada and the old IMC site is a problem, but isn't  
keeping the other sites up an easier, more tractable problem?

--
Joe Futrelle
Person


On Oct 1, 2006, at 11:35 PM, Mike Lehman wrote:

> I want to also express my appreciation for the work being done by  
> everyone on this big, but very much needed project.
>
> Here's a few suggestions from the user perspective.
>
> We've already agreed that the new site needs to be up, due to spam,  
> etc. So if the old Dada site needs to go down for the time being,  
> as long as the new site goes up in its place at www.ucimc.org,  
> that's cool. And very much needed. There are lots of things going  
> on in our community right now, so we badly need to get something  
> usable up where people are used to finding it.
>
> It will be good to be able to access the old site, sooner rather  
> than later, so if Dan can get it going on an isolated server where  
> it won't continue to cause problems and can be easily diagnosed,  
> that sounds like a plan. In the meantime, we can manage without it,  
> as long as people find the new site in its place.
>
> AFAIK, the patches on Dada that were made by Clint were due to the  
> legacy system (SFActive?) needing to be patched into Dada. Is it  
> possible to just pull from the Dada database since then and do a  
> relatively clean and easy update to whatever the latest build of  
> Dada is with just that part of the database? If so, then we can  
> leave the oldest part of ucimc.org -- the one that was patched in  
> and has most of the potential issues that might snag a Dada update  
> -- for last. I think we're talking about just the first two years,  
> as I think the last four years were since the update to Dada was done.
>
> But maybe I don't know WTF I'm talking about with that.
>
> Anyway, I'm just saying that people are badly itching for a usable  
> site ASAP -- and the rest can come along when it's figured out. No  
> one is going to mind if archives of older stuff aren't available  
> right away.
> Regards,
> Mike Lehman
>
> dan blah wrote:
>> On 10/1/06, Zachary C. Miller <zach at zarfmouse.us> wrote:
>>> The reason I harp on this so much is that it is the ROOT cause of  
>>> all
>>> the resource problems that imsahp has. System response too slow for
>>> you? Mail gets backed up on the first of the month? Hourly downtime
>>> annoying? Some web requests hang forever? These are all rooted in  
>>> the
>>> apache hangs. Once we solve the problem of the apache hangs there  
>>> will
>>> be a TON of available CPU power available and a definite increase in
>>> speed and stability of all services on imsahp.
>>>
>>> Anyone who is inspired to pour a ton of effort into helping fix the
>>> problems that imsahp has really needs to pour their efforts into
>>> solving this problem first and foremost. Moving to Drupal is a  
>>> start.
>>> Moving everything to Apache2 is a start. But as long as we're
>>> continuing to run Dada in any form (or on the off chance that we
>>> discover that Dada is not the cause) we have to be prepared to
>>> continue to deal with the issue of apache hangs. If we can isolate
>>> various potential causes (upgrade to Apache2 so we can't blame old
>>> apache, upgrade to php5 so we can't blame old php, move Dada off the
>>> system entirely so we can't blame Dada, stop even loading  
>>> mod_perl so
>>> we can't blame mod_perl, etc) we will figure out what the real cause
>>> is. We're on the right track, I just want to make sure we're all on
>>> the same page about what the track is.
>>>
>> my plan is to do just this but after we get the old dada site working
>> on the new server.  as there are no resource issues at all on that
>> server with most all of hosted sites working its a fairly safe
>> assumption that apache issues on imsahp are related to the current
>> ucimc dada installation.  provided we can get the old dada install
>> work on the new server w/o causing any resource issues that pretty
>> much solves our problems.  from there i will start going through the
>> steps you have have listed just for the sake of knowing (i am curious
>> exactly what the problem is).
>>
>>> Another backwards compatibility issue we have to consider: When  
>>> Clint
>>> ported us to Dada, he made sure that old URLs to the original "IMC
>>> active" software still worked. I don't know if he did that by  
>>> leaving
>>> a crippled install of active in place or by porting all the old
>>> articles into Dada and putting in a forwarding script, I've never
>>> looked into it. We'll need to figure that out (or maybe I'm just
>>> imagining that he ever even did it) and make sure those old old URLs
>>> also remain valid if possible.
>>>
>> as for the rewite script, awesome.  i cped your pub key on imsahp to
>> zeco.chambana.net (new web server).
>>> On 10/1/06, Zachary C. Miller <zach at zarfmouse.us> wrote:
>>> > Note to all: please understand this well: read this all and  
>>> don't skim
>>> > it...I feel like I've explained this problem in detail before and
>>> > people seem to still be making other theories about what is  
>>> happening
>>> > without having addressed what I know to be the facts of the  
>>> problem.
>>> > I'm not trying to be snarky, I'm EXTREMELY grateful that so many
>>> > people (especially David and Dan!!!! Thanks!!!!) are stepping  
>>> forward
>>> > to take care of imsahp, I don't have time to do it and you guys  
>>> are
>>> > the lifeline that the system needs. I'm just trying to make  
>>> sure that
>>> > we REALLY understand the problem that needs to be solved,  
>>> rather than
>>> > taking wild guesses about what will fix it.
>>> >
>>> > The primary bog down that Dada seems to be causing (it might  
>>> not be
>>> > Dada, it might be something else) on imsahp is that apache  
>>> processes
>>> > become STUCK in a loop that consumes 100% of the available CPU and
>>> > never times out. The stuck processes build up until the server  
>>> is so
>>> > bogged down that it is unusable. Nothing gets logged when apache
>>> > becomes stuck in this manner so it is impossible to know for sure
>>> > whether Dada is causing the problem but a lot of circumstantial  
>>> clues
>>> > suggest Dada is the cause.
>>> >
>>> > PHP is embedded in Apache so I believe the stuck processes are  
>>> due to
>>> > Dada exposing some bug in PHP. Even badly written PHP code  
>>> should time
>>> > out eventually unless there is a bug in PHP. This is why, even  
>>> though
>>> > I believe the problem is caused by Dada, I've advocated in the  
>>> past
>>> > for an upgrade of PHP and of Apache (as well as an upgrade of  
>>> Dada) to
>>> > see if the bug has already been fixed in a newer version. These
>>> > upgrades are neccessary anyway for security and stability and
>>> > supportability purposes.
>>> >
>>> > I suspect (this is WILD speculation) that the problem might be  
>>> related
>>> > to memory allocation.  Some OJC people have told me they've seen
>>> > similar problems with older version of mod_perl and that they were
>>> > memory allocation related. (And honestly, if someone could  
>>> check all
>>> > the websites we host and confirm that none of them are using  
>>> mod_perl,
>>> > I'd suggest disabling mod_perl and see if that helps).
>>> >
>>> > I do not believe the problem is related to rebuilding of indexes
>>> > because it is Apache and not MySQL that sucks up the CPU time. The
>>> > stuck apache processes do not respond to normal kill signals (they
>>> > must kill -9'ed) so we're really talking about the exposure of  
>>> a BUG
>>> > and not just some rogue code that takes a lot of power to run.
>>> >
>>> > Note that there are 3 such stuck processes on imsahp at this very
>>> > moment as I write this email. They'll go away as soon as the  
>>> hourly
>>> > restart of apache happens. Then they'll inevitably come back  
>>> over the
>>> > next hour. If you guys have stopped posting on Dada, it clearly  
>>> has
>>> > NOT stopped the stuck apache process problem. It will continue to
>>> > plague us on a system that is running Dada in "archive" mode.
>>> >
>>> > I am pretty sure that I have seen the stuck process thing  
>>> happen while
>>> > I was making a web request to simply read the index page (not  
>>> posting
>>> > an article) of ucimc.
>>> >
>>> > Curently to address this problem there is a root cronjob that  
>>> restarts
>>> > apache every hour. This causes about 5 minutes of downtime for  
>>> every
>>> > site on the server every hour at 51 minutes past the hour. This  
>>> is a
>>> > terrible hackish stop gap that has allowed the server to  
>>> survive but
>>> > the server is still constantly a bit bogged down by a few of these
>>> > rogue apache processes. Performance on the server will  
>>> skyrocket when
>>> > this fundamental apache problem is solved.
>>> > 51 * * * * /home/wolfgang/bin/apache-restart >> /var/log/apache- 
>>> restart.log 2>&1
>>> >
>>> > If we're going to have an "archive" version of dada, it needs  
>>> to run
>>> > on it's own separate dedicated server. Then we can isolate Dada  
>>> and
>>> > confirm that it is the culprit causing the apache hangs and if  
>>> it is
>>> > we can rest assured that it won't take the rest of imsahp down  
>>> with
>>> > it. We might learn from this experiment that Dada has nothing  
>>> to do
>>> > with causing the Apache hangs...but it'd be nice to learn that. We
>>> > need to use a process of elimination to figure out what is  
>>> triggering
>>> > the apache hangs.
>>> >
>>> > I can help configure rewrite rules if someone can set up the  
>>> archive server.
>>> >
>>> > On 9/27/06, David Gehrig <gehrigspamtrap at gmail.com> wrote:
>>> > > It seems to me that post-less dada isn't bogging down the  
>>> system the way it
>>> > > usually does -- that dada's problem might be something to do  
>>> with
>>> > > regenerating indexes or something else cycle-sucking every  
>>> time someone
>>> > > posts, and that the spamming was therefore bringing it -- and  
>>> everything
>>> > > else on imsahp -- to its knees.
>>> > >
>>> > > If there's agreement on that, then it's time to switch to the  
>>> new site.
>>> > >
>>> > > I am not the master of rewrite rules, though.  Dan, could you  
>>> set up
>>> > > rewrites to do what I've described in step (3) below?   The  
>>> idea is that any
>>> > > URL that goes to www.ucimc.org/newswire/.... goes instead to
>>> > > archive.ucimc.org/newswire... etc.
>>> > >
>>> > >
>>> > > On 9/26/06, David Gehrig <gehrigspamtrap at gmail.com> wrote:
>>> > > > All right, here's my so-called thoughts.
>>> > > >
>>> > > > (1) I'm really very reluctant to try to find a way to graunch
>>> > > > the dada data into drupal.  I don't think it can be done well,
>>> > > > and I don't have the time to waste doing it badly.
>>> > > >
>>> > > > (2) Let's watch the lobotomized dada site for a few more
>>> > > > bounces -- if the drain on the system was related to the
>>> > > > process of posting to dada, then we should see better
>>> > > > response all around now that it no longer is.  Early signs
>>> > > > look good.
>>> > > >
>>> > > > (3) Here's a little more detail of what I'm suggesting.
>>> > > >
>>> > > > If a URL request comes in for
>>> > > >
>>> > > > www.ucimc.org/newswire/....
>>> > > > www.ucimc.org/mod/info/...
>>> > > > www.ucimc.org/editor/...
>>> > > > www.ucimc.org/library/...
>>> > > >
>>> > > > then redirect it to archive.ucimc.org.
>>> > > >
>>> > > > Otherwise have it go to the drupal site.
>>> > > >
>>> > > > I'm assuming this would have to be done in the main
>>> > > > apache config file. If this passes a basic sanity check,  
>>> let's try it.
>>> > > >
>>> > > > @%<
>>> > > >
>>> > >
>>> > >
>>> > > _______________________________________________
>>> > > IMC-Tech mailing list
>>> > > IMC-Tech at lists.ucimc.org
>>> > > http://lists.chambana.net/cgi-bin/listinfo/imc-tech
>>> > >
>>> > >
>>> > >
>>> >
>>> >
>>> > --
>>> > Zachary C. Miller - @= - http://zach.chambana.net/
>>> > IMSA 1995 - UIUC 2000 - Just Another Leftist Muppet - Ya Basta!
>>> >  Social Justice, Community, Nonviolence, Decentralization,  
>>> Feminism,
>>> >  Sustainability, Responsibility, Diversity, Democracy, Ecology
>>> >
>>>
>>>
>>> -- 
>>> Zachary C. Miller - @= - http://zach.chambana.net/
>>> IMSA 1995 - UIUC 2000 - Just Another Leftist Muppet - Ya Basta!
>>>  Social Justice, Community, Nonviolence, Decentralization, Feminism,
>>>  Sustainability, Responsibility, Diversity, Democracy, Ecology
>>> _______________________________________________
>>> IMC-Tech mailing list
>>> IMC-Tech at lists.ucimc.org
>>> http://lists.chambana.net/cgi-bin/listinfo/imc-tech
>>>
>>
>>
>
> _______________________________________________
> IMC-Tech mailing list
> IMC-Tech at lists.ucimc.org
> http://lists.chambana.net/cgi-bin/listinfo/imc-tech