Bad News About the Good News
We seem to have solved the memory leak problem by replacing a faulty module in the compiled version of... I forget? Apache or Linux. Then we solved another memory leak by downgrading PHP to 5.2x since some modules we are using have problems with 5.3 and above. Thanks Rackspace! for providing continued support for 5.2 when Zend and even Centos had abandoned it. Now we are trying to find the residue of the problem... and we are looking at possible IPSF - intermittent power supply fault.
Sufficient to explain symptoms but not necessary... it could instead be three other things, but this is the simplest explanation. Which means replacing the server since paying for datacenter techs to work on hardware is not cost effective.
IPSF was what killed ZsaZsa, our other server two months ago. Magda (the BC server) is a near-twin. ZZ's disease progressed from IPSF to RPSF (recurrent... which made it easier to detect and pretty much impossible to run the machine. ZsaZsa didn't have a leg to stand on, as it were. Actually, ZZ had a redundant power supply and a slight rewiring has restored her to usefulness at Piper's home, she's now our test bed. That name metaphor is getting a bit strained. :)
Anyway, it's going to cost $800 to $2000 to replace Magda and we are now debating on how long we should try to limp along.
Hugs,
Erin, Piper, and the gang