4

1

My users are complaining that FogBugz is running slowly all of a sudden. When I run top on my FogBugz web server (which is on Linux), I see that the mod-mono-server2.exe process is consuming 99% CPU.

How can I prevent this from happening?

Fog Creek Case FC1863199

flag

8 Answers

3

We're still working on getting a reliable repro of this problem. This issue has been very tough to replicate and minimize into a small test case, but once we do, we'll report it to the Mono team for a fix.

As a workaround, your best bet is to setup up a nightly cronjob which restarts FogBugz. This has worked pretty well as a stopgap for most customers.

The command to restart FogBugz on *nix is:

/[path_to_fogbugz]/bin/fogbugzctl restart

The default path is /opt/fogbugz.

If this doesn't work try the steps in the Not Working? section below.

Deprecated as of FogBugz 8.3:

Contact Fog Creek Customer Service before trying this option. Feel free to use the simple script restart further below, however.

There was a more elegant way to reset FogBugz without having to use a cronjob. In your /opt/fogbugz/conf/httpd.conf file add one of the following settings:

MonoMaxRequests 500

MonoMaxUptime 86400

This should force Mono to restart after a certain number of requests or a certain number of seconds of uptime, and should keep the memory leak problem at bay. The best solution is to perform a simple restart cron job described below.

Not Working?

Sometimes mono doesn't restart when it should. If you find that fogbugzctl stop doesn't work (use ps -ef | grep mono to see if mono is still running), try the following:

Setup a cron job to do the following:

/opt/fogbugz/bin/fogbugzctl stop
pkill mono
/opt/fogbugz/bin/fogbugzctl start

If you don't have pkill, you should be able to get it through your package manager. You could also do an analog of the above in the fogbugzctl script itself. If you do so, we recommend having the script sleep for a couple of seconds between the pkill and the start. If you need more information about that please contact us!

link|flag
Just to add some weight, we've just seen this on our setup (7.3.0, Fedora 8), but only once. – james2115 Jun 9 2010 at 0:54
And just as I was reading this - it's happened again. The restart ended up with another (extra) copy of mono at 100% - had to manually kill both and re-start FogBugZ. We do use (and see errors from) the Backlog plugin - maybe that could be part of the problem? – james2115 Jul 6 2010 at 1:21
@james2115 - No, I don't think the backlog plugin has anything to do with the issue. – Rob Sobers Jul 6 2010 at 13:17
I have seen the same problem where running fogbugzctl restart causes there to be two mono processes, both using max cpu. Like james2115, I had to manually kill them and then restart fogbugz again. – Jason Oct 18 2010 at 16:56
Using the new settings in FogBugz 8.3, FogBugz fails to restart the mod_mono_server2 process so shutdown never completes and FogBugz gets stuck in a half started state where it just gives an error message. We had to disable them again due to the constant hangs. – harsman Mar 18 2011 at 9:30
1

FogBugz 8.1.1 running on Ubuntu 9.2:

After running for several days, ran ‘ps alx | grep mono’ and found two processes:

Two processes: 12502 and 12509, MEM 8.6% and 0.8% respectively

Notice that 12502 is using a significant amount of memory (8.6%) but so far the CPU utilization looks fine.

Restarted FogBugz (/etc/init.d/fogbugzctl restart) and checked for mono processes:

12502 CPU 97% and MEM 9.8%
24514     -0-         2.8%
24521     -0-         0.9%

Now we have three … one “old” process (12502) – and it is using nearly all of the CPU. The amount of free memory has been reduced significantly.

Stopped FogBugz, killed all mono processes, and started FogBugz: 

24715 CPU -0- MEM 0.9%

Notice that the available memory is of the order of 50MB more than it was after running FogBugz for several days.

My conclusions: 1. mono has a memory leak 2. Neither restarting nor stopping FogBugz terminates the mono process spawned by FogBugz.

Temporary fix: • Run a job daily that (a) stops FogBugz, (b) searches for and kills all mono processes, and (c) starts FogBugz again.

link|flag
1

I tried using the script with

/opt/fogbugz/bin/fogbugzctl stop
pkill mono
/opt/fogbugz/bin/fogbugzctl start

per the answer given above, but I found that I needed to insert a sleep statement before the start command in order for the server to come back up properly. Being the cautious type, my script now looks like this:

/opt/fogbugz/bin/fogbugzctl stop
sleep 2
pkill mono
sleep 2
/opt/fogbugz/bin/fogbugzctl start

She's set to fire every morning around 3:00!

link|flag
0

Seeing this also with Version 7.3.1 (DB 743, Build 1866) on CentOS 5.5

link|flag
0

I'm still seeing this with 8.1.4 (DB 770, Build 1). Any new answers from the fogcreek people?

link|flag
@Sean we're looking into packaging a new version of the Mono runtime with FogBugz. The new version has an updated garbage collector, which we're hoping will address this issue permanently. We don't have a timeline for this yet. – Rob Sobers Dec 28 2010 at 17:28
0

I see this as well. I'm going to put a "kill nightly" script on the Linux server because once mono goes "runaway" like this, the performance hit on the server is severe!

link|flag
0

I'm interested to see if there have been any updates on this as well! Going to try implementing the suggested fix to see what happens.

Any chance this is related to having a previous install of Mono on the server? We ran FB6 on the same machine and have not as of yet un-installed Mono that was installed to support that older configuration. Server is RHEL5 on Intel.

link|flag
0

Been seing this regurlarly too. Both in ver 7 and now in latest ver 8.x

link|flag

Your Answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.