trac.fcgi on homing

[Ethan Blanton]

Mark Doliner spake unto us the following wisdom:
> Alright, so what's the deal with trac.fcgi on homing?  We should
> really fix this.  The resident memory usage of the trac.fcgi process
> climbs up to a 1GB in a matter of hours.  It uses lots of CPU, but I'm
> not sure how out-of-the-ordinary that is.  If the process doesn't have
> a high nice value it makes the machine fairly unresponsive.  I've
> restarted it a few times this weekend because it developer.pidgin.im
> was very slow to respond.

[Unless things have changed significantly while I was out for the
weekend,] We don't know.  Both Daniel and I (and perhaps others) have
spent some time looking at it.

> Did anyone upgrade, downgrade, or otherwise change anything related to
> trac or viewmtn or lighttpd or fastcgi around January 14th?  The munin
> graphs[1] show an noticeable increase in committed memory around that
> time.  It looks like someone has been trying to run trac in a Python
> memory profiler.  Any luck with that?

Not that we can find.  The only thing which has been recently upgraded
(that we know of) which is related to trac is postgresql, and the
problem does not *seem* to be in postgres.  (It's hard to tell.)  I
also suspect that if a massive postgres problem had crept into Etch,
someone else would have noticed.

As best we can tell, the massive CPU munching started around January
17th, somewhat later than the memory problem.  It may be that the two
are related; I'm not sure.  Certainly the CPU time billed against trac
is mostly system time.

> /var/log/dpkg.log shows that the following packages have been installed/upgraded:
> 
> On January 7th: php5-common, libapache2-mod-php5, mysql-common,
> libmysqlclient15off
> On January 10th: dovecot, php5, php5-cli, php5-mysql, mediawiki1.7,
> mediawiki-extensions
> On January 13th: libpq4, libxml2, postgresql-client, postgresql-plpython
> On January 18th: php5-memcache, mysql-client-5.0, mysql-server-5.0
> On January 19th: mediawiki
> 
> Have we tried disabling parts of trac to see if we can narrow down the
> source of the problem?  Like the Doxygen plugin?  Or TracMonotone?

We have, at least some.  TracMonotone is not in use; we have changed
some [more] previously trac-served information to be lighttpd-served,
expanded robots.txt, and tried upgrading our account manager.  I'm not
sure what else.  We have *not* gone through the plugins one at a time,
or at least I have not.  That's on my list of things to do, but
obviously it would be great if someone else could try, because I
haven't had time.

> I noticed there are 4694 subdirectories under
> /home/var/tmp/viewmtn-graphs/ dating from May of last year.  Each
> directory is a revision number.  The total size of all directories is
> only 98MB.  It doesn't like like this would be causing a problem, but
> would it be safe to delete these directories?

It should, and I will.  This would not be related to trac, however,
unless it's generally just screwing up /home that badly.

So ... that's a long message saying that it would be awesome if
someone figured this out.  I tried profiling trac with the python
profiling junks, but (as I expected, since the massive time is system
time) I didn't see anything I knew enough to be able to work with.
Caveat lector, I don't even know python.  ;-)

Ethan

-- 
The laws that forbid the carrying of arms are laws [that have no remedy
for evils].  They disarm only those who are neither inclined nor
determined to commit crimes.
		-- Cesare Beccaria, "On Crimes and Punishments", 1764
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://pidgin.im/pipermail/devel/attachments/20080128/6ccc480e/attachment.sig>