That's the problem.

Post #246,612 by static 3/2/06 8:36:17 PM Reply	That's the problem. I don't know how many files it has open. Rummaging in /proc/(pid)/fd shows that threads often have the same file open multiple times - does this count? There were just over 1500 items in the We're fairly sure something is capping connections before it hits max_connections because under sustained load, the thread count in MySQL caps below max_connections. So it's running out of some other resource before connection threads. And when that happens, we get 'cannot create thread' errors - but that doesn't say why the thread can't be created. :-/ 2 I'd love to move to one connection per Apache process. Unfortunately, that's a political fight, as I mentioned. The web server app is written in PHP so the equivalent is called pooled connections. The problem is that the CEO has access to MySQL's show processlist; pooled connections show up as idle threads in that and Apparantly That's Bad. However, MySQL is known for being fast at connection setup and teardown. So. The reverse-proxy arrangement sounds interesting and would be something to try for. Thanks for the thoughts. Wade. "Insert crowbar. Apply force."
Post #246,621 by drewk 3/2/06 9:37:51 PM Reply	You already know this, but I'll say it anyway The problem is that the CEO has access to MySQL's show processlist; pooled connections show up as idle threads in that and Apparantly That's Bad. First, regardless of what technical problems you're having the CEO shouldn't be involved at that level. If he is, what is he paying you guys for? Now, assuming you're dealing with someone who should be looking at the processlist -- and since he's The Boss™, he gets to decide that he is that someone -- it might be appropriate to contrast real production problems that you can demonstrate you are actually having vs. a potential perceived inefficiency. Is there a slow time or regular maintenance window during which you could try changing the configuration and benchmark it? If so, I'd put together a list of two or three changes you'd like to try -- like bumping up the limits you mentioned above, things you believe won't actually fix it -- and then while you're doing it also try the pooled connections. And just so I have some frame of reference, what kind of transaction volume (page views) are you doing? I worked on a system with three fairly beefy webservers hitting a MySQL master with five slaves. === Purveyor of Doc Hope's [link\|http://DocHope.com\|fresh-baked dog biscuits and pet treats]. [link\|http://DocHope.com\|http://DocHope.com]
Post #246,635 by ben_tilly 3/3/06 1:21:08 AM Reply	There is a risky political solution Go to the CEO and say, "I have a big scalability improvement that I could easily make which I've been told not to do because we're afraid that you'll misinterpret how efficiently our database is being used." He'll get upset, but it won't be at you. And the political barrier to the solution will melt away. However the person who he gets upset at is likely to become unhappy with you. OTOH the CEO is likely to be relatively pleased with you, but that might not help you very much. I suspect that very few people would choose that solution... Cheers, Ben I have come to believe that idealism without discipline is a quick road to disaster, while discipline without idealism is pointless. -- Aaron Ward (my brother)
Post #246,639 by static 3/3/06 6:34:00 AM Reply	Heh. It's tempting. It could even work. All we really need is for pages to be noticeably faster. And for sales to go noticeably up. But we've got another solution in the offing. We're going to remove the CEO's access from that application. I believe we've weened him off root access on the web servers. It will be part of the effort to stop the web developers having near full reign in the production environment, too (that gives me the horrors sometimes). Wade. "Insert crowbar. Apply force."
Post #246,816 by folkert 3/4/06 6:20:33 PM Reply	TCPOE? How about a TCP Offload Engine? There was a couple I found that were Linux only, but only one was available. Ping Barry about the hardware, I think he actually got one. It might improve you connection success ratio, also make your idle connections a might bit more wantable... by jerri-engineering about the TCPOE with regard to connections and all that. A small amount of money on hardware might just avoid the political firestorm. -- [link\|mailto:greg@gregfolkert.net\|greg], [link\|http://www.iwethey.org/ed_curry\|REMEMBER ED CURRY!] @ *i_wethey* Freedom is not FREE. Yeah, but 10s of Trillions of US Dollars? SELECT * FROM scog WHERE ethics > 0; 0 rows returned.

Welcome to IWETHEY!