Donated hosting and dedicated box from networkredux

[Luke Schierer]

On Sat, Jul 05, 2008 at 10:27:12AM -0400, Ethan Blanton wrote:
> Kevin Stange spake unto us the following wisdom:
> > Evan Schoenberg wrote:
> > >Thomas Brenneke of NetworkRedux, which provides Adium's hosting - both  
> > >a dedicated box they provided and co-lo for a second machine we  
> > >provided - has offerred to provide both hardware (a dedicated machine)  
> > >and hosting for free to Pidgin.
> > 
> > Not to start a "bidding war" here, but I should note that I talked it 
> > over with my boss at Steadfast Networks and we are also prepared to 
> > provide a dedicated server for free, depending on bandwidth 
> > arrangements, with the single requirement of a button indicating our 
> > hosting services are used at the bottom of the site (similar to what we 
> > already do for DV LABS).
> 
> I don't have a problem with us adding a second machine, or replacing
> homing, but just to be clear ...
> 
> As I pointed out yesterday (or the day before?) on devel at cpi, in each
> of the previous meltdowns to this ICQ meltdown, it has turned out that
> there was a serious configuration and/or content problem which was
> chewing up the machine.  Had we had a bigger or better machine, it
> would have simply chewed harder and still melted down.  Now, I don't
> know about this ICQ situation, maybe it was a simple overload.  If it
> was, the answer is *still* probably to streamline the services, and
> throwing more power at it will be (of course -- it always is) stopgap.
> 
> We can get a new machine, that's fine; however, I think we need to
> look hard at the costs of maintaining two (or switching over, if
> that's what we want to do) in light of the fact that it almost
> certainly won't *fix* any problems.  It may or may not push them back.
> 
> Note that I think one real benefit would be to move the monotone
> repository (Gary has also offered this in the past); we have a lot of
> problems with syncing from the distribution database to the databases
> for the various local services.  This is also a problem which is only
> worked around by distributing, but the workaround is effective and
> quantifiable -- those databases are huge, and (I think) we spend a lot
> of time with I/O contention between the various databases.  As to
> where it goes, I am ambivalent.
> 
> Ethan

This time I think the monotone & viewmtn loads were the biggest part of
of bringing nearly bringing us down.  

I do think it noteworthy though that this time, unlike past times, we
were not entirely floored.  We've improved a lot since the previous
meltdown, we never lost shell access (though there was some delays) and
I don't think the machine had to be rebooted.  It certainly didn't
require work at the console like last time.

Luke