[UCIMC-Tech] Sites Down

[Josh King]

So far so good. I hit the server with a thousand requests to ucimc.org,
both sequentially and in 100 simultaneous chunks. Load stayed below 1.0,
where before it spiked up above 15.0 and caused process failures.

Can probably still do more to optimize for speed, but assuming the
stability problems are tied to load, this should fix things.

On 01/19/2011 02:16 PM, Mike Lehman wrote:
> Josh,
> Thanks. Hope this gets to the problem.
> Mike
> 
> On 1/19/2011 12:40 PM, Josh King wrote:
>> Sorry, I brought it back up last night and this morning, and now I'm
>> running tests to see if I can see what the problem is.
>>
>> Okay, so let's try this. I re-tuned varnishd and apache. I actually
>> think the problem is with apache, not with varnish. Under load spikes
>> apache is ignoring its directives and spawning too many processes, and
>> once the system gets saturated from load varnishd is less able to
>> recover than apache because apache is composed of multiple, dynamically
>> restarting processes as opposed to varnishd's single process that is
>> more subject to error conditions. I believe the tuning done before to
>> try and maximize performance was wrongheaded with respect to how Linux
>> and FreeBSD respectively manage memory and processes. Hopefully with
>> these tweaks, the load will be brought down to something more manageable
>> given the server's capacity. I'm going to do some more benchmarking
>> (trying not to lock it up again) and see if this works better).
>>
>> On 01/19/2011 12:05 PM, Mike Lehman wrote:
>>> Back up about an hour ago and now back down again now.
>>> Mike Lehman
>>>
>>> On 1/19/2011 8:45 AM, Mike Lehman wrote:
>>>> Someone got things back on-line last night before I hit the sack. Now
>>>> they're down again this morning.
>>>> Mike Lehman
>>>>
>>>> On 1/18/2011 8:57 PM, Mike Lehman wrote:
>>>>> Both sites are in 503, I suspect on the way to being down as there
>>>>> are codes/messages about Varnish again.
>>>>> Mike
>>>>>
>>>>> On 1/18/2011 7:41 AM, Chris Ritzo wrote:
>>>>>> Both are now back online.
>>>>>>
>>>>>> It appears to be the caching server daemon filling up the tmp file.
>>>>>> I cleared /tmp, restarted varnishd and apache, but this is a
>>>>>> temporary fix.
>>>>>>
>>>>>> Anyone have suggestions on how to tune Varnish?
>>>>>>
>>>>>> -Chris
>>>>>>
>>>>>> On Tue, Jan 18, 2011 at 5:52 AM, Mike Lehman
>>>>>> <rebelmike at earthlink.net<mailto:rebelmike at earthlink.net>>  wrote:
>>>>>>
>>>>>>      UC IMC and WRFU are down.
>>>>>>      Mike Lehman
>>>>>>      _______________
>>>>>>
>>>
>>>
>>> _______________________________________________
>>> IMC-Tech mailing list
>>> IMC-Tech at lists.chambana.net
>>> http://lists.chambana.net/mailman/listinfo/imc-tech
> 
> _______________________________________________
> IMC-Tech mailing list
> IMC-Tech at lists.chambana.net
> http://lists.chambana.net/mailman/listinfo/imc-tech

-- 
Josh King

"I am an Anarchist not because I believe Anarchism is the final goal,
but because there is no such thing as a final goal." -Rudolf Rocker

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://lists.chambana.net/pipermail/imc-tech/attachments/20110119/d3f1b04d/attachment.bin>