Despite my best efforts, PHP seems to have issues on Lupinia’s server, and I don’t know what to do about it anymore.

When we first moved Lupinia to its new server last January, everything seemed to be working fine with PHP in an IIS environment. However, as time went on, we discovered a problem: After about 100,000 requests to a particular application pool (in IIS, a group of virtual hosts), PHP would suddenly stop responding, and start giving internal server errors, until the application pool was recycled (about a 15-minute process if left to do it by itself). And, every once in awhile, apparently at random, it would take a full restart of IIS to get it working again.

I was able to keep things under control for awhile, but as load increased, so did the frequency of these errors, to the point where it was happening every couple of days. I finally took the time to search for it, hoping that I was doing something wrong, only to find that it was a known bug in PHP’s ISAPI module, and they haven’t done anything about it since 2004. Lovely. The only other built-in option is to run PHP as a CGI application, where it launches an EXE every time a request is sent to it. Unfortunately, in a Windows environment, that equates to obscene loads on the system resources, and it’s not a viable option on a server that hosts the traffic we do.

After some searching, I found the answer in the Zend Core, a corporate-grade PHP installation for IIS. It has an interface called FastCGI, which is the best of both worlds in this case; it has the stability of a CGI application, but it only launches one EXE per virtual host (which is still a large number, but manageable). Everything has been working quite peachy in the few months since we upgraded, except for one problem; instead of PHP errors giving error messages in the page generating them, they result in a generic Internal Server Error.

Until recently, I didn’t really have the time to look into fixing it. But, this weekend, I’ve been working on a project that’s really pushing the limits of my abilities (in a good way, this is how I learn new things), which means little errors here and there. And, I’m getting really tired of blind debugging.

And, unlike the previous problem, Google has been no help whatsoever. I’ve spent the last two hours searching for possible causes, and tweaking config variables. All of the search results I’ve found have been irrelevant in some way.

I really don’t want to spend $500 for a support package from Zend (the software is free, but they charge out the nose for support).

(more…)

…It’s faster and more responsive running a full-screen remote desktop session with a computer a thousand miles away than it is running its own desktop. Note to self, needs more memory

It’s common knowledge to everyone in the IT industry that the goal of any server is to run as long as possible with no interruptions, keeping reboots to an absolute minimum. Why, then, does Windows 2003 require a reboot to install Windows Updates? I mean, this OS seems to know better in every regard, but to require a freaking reboot for patches that don’t affect the core of the OS (read: IIS patches)?

Perhaps it’s because Windows Update on 2003 is the same code as on XP, which requires a reboot for patches that don’t actually do anything (read: “Malicious Software Removal Tool” du jour)?
(more…)

In the process of this server project, I’ve encountered some rather bizarre problems with the various PHP applications hosted on Lupinia. In particular, moving things from the old environment to the new one has a tendency to break things, and none of the problems I encountered seemed to be in the documentation for the particular application in question.

Now, I expected problems when I started the migration process, since most of these systems are highly complex (Gallery2, in particular, boggles my mind in its complexity). What I didn’t anticipate was the utterly abysmal support response from every open-source application I had issues with. In the last couple of weeks, I’ve started at least a dozen new threads in various support forums. And in all of them, I’ve either had no responses at all, or the only response was someone questioning basic configuration/permissions problems that I had already checked, and I had stated as much in the initial post.

What’s even more frustrating is that a lot of these problems were fixed by a few extra lines of code to check for a particular contingency that needed to be addressed. Several more required a touch of database configuration/table adjustment that wasn’t covered in the documentation, and the Gallery2 problem mysteriously fixed itself after viewing the code of a couple of its files, without actually changing anything.

Now, I realize that these are volunteers providing support, and I also realize that PHP on IIS/Windows isn’t the most common server environment. But when every other thread on a support forum receives multiple posts that attempt to solve the problem, while mine receives none, there’s something very wrong.

The one good experience I’ve had so far with supporting web applications has come from the company that wrote the helpdesk software we’re using, Expinion. It’s an ASP-based application, and I had to pay for it, but that money was well spent. Right after installation, I noticed a very strange issue with the ticketing interface, and I submitted a support request on their site. Turns out, the problem was related to the fact that I was running it on MySQL instead of MS SQL, and the guy who responded to the ticket actually wrote a patch to fix it on the spot, and had it working by lunchtime. That, my friends, is excellent tech support, and I will gladly recommend their products to anyone because of that one experience.

Hello everyone! Apologies as usual for my lack of updates, you know the drill.
(more…)

Much to my dismay, the Lupinia server project has been stalled. I have everything sitting on my desk waiting, except for one thing: The second hard drive necessary to create a RAID 1 array.

RECAP: I ordered a pair of really nice Seagate 500gb drives from Newegg at the same time I ordered the server itself, and they even arrived first. Upon installing them and giving the server a test-boot, HD 1 gave the Click of Death. Pissed off, I RMAed the drive, and almost two weeks later, the replacement arrived. I installed the drive, fired up the server and…HD 1 Not Present Or Not Installed. Curious, I gave it power from my desktop computer (which has external SATA ports and is far, far quieter), and it wouldn’t spin up, it just sat there and gave a repeating buzzing noise, like the drive shaft on one of the motors wasn’t engaged or something.

I’ve submitted another RMA, a Refund this time, and I’ll be purchasing a companion drive from another vendor. I’ve heard great things about Tiger Direct, but they don’t have an exact match for the drive I already have. I found a place that has a bajillion positive reviews from Google Shopping or whatever it’s called, but does anyone have any recommendations? Here’s the drive I need, it has to be an exact match to set up RAID properly: http://www.newegg.com/product/product.asp?item=N82E16822148288

As I mentioned before, this project is finally underway, though I think I overstretched myself financially to do it. Considering that my current host appears to be outright ignoring my critical email issues (still affecting all Lupinia addresses intermittantly, btw, and ticket still with no response), this comes not a moment too soon. I’ve acquired everything necessary to start working on this, for the most part. One of my new HDs was DOA, so I had to send it back to Newegg, and I’m waiting for it to arrive. The guy on Ebay I bought my Windows CDs (8 FREAKING DISKS) from apparently did ship them, they arrived today. I had absolutely no contact with him, so I was kinda getting worried, despite his 100% positive feedback.

I expect to go live sometime in early January. I also have a question for you people who’ve dealt with co-lo before: Should I drive the server to the facility myself (it’s in Florida, I’m in Virginia), or ship it? Florida is a long way away, I realize, but if transporting it myself is that much more preferable to shipping it, I’ll do it. I saved the original packing material, in any case (one of many reasons I wanted a pre-built machine).

Also, could someone who’s wiki-savvy create a Wikifur page for Lupinia based on the info on this page? I’d like to have one, but I find the wiki tags/syntax highly confusing, to put it mildly. I’ve tried multiple times to start one, but I just can’t make sense of writing it. Whoever makes the initial page will be credited when I link to it from Lupinia (it doesn’t look like people usually do that on the site), I just want it done (doing it well is a plus!), and I don’t have the time/patience to figure it out. Thanks in advance to whoever does it

Oh, and many thanks to the people who’ve made edits/additions to my Wikifur page in the past, you guys rock! Like I said above, I have trouble understanding how to properly manipulate wikis, but I can make little edits here and there without trouble.

Hooray! In case you’re just joining our adventure, I announced in March that Lupinia would be moving to a co-located dedicated server. There were many reasons for this decision, including poor support from our current host, flaky email system, and a large number of limitations on the current site. Due mostly to my own screw-ups, this project fell grossly behind schedule several times, and I wouldn’t blame many of you if you considered this to be vapour-ware.

But, I’ve managed to pull through, and this project is finally on track! The server arrived today, and I installed the new hard drives. I’ve also acquired almost all of the software, I’m just waiting on an Ebay seller to ship the OS. I’m also not going about this by myself anymore, either; and I will be co-administrating the server, and will be applying his expertise as our network security administrator.

Here’s a picture (click on it for the full gallery):
(more…)

I posted about this in lupinia_status, but I’m posting about it here, too, so everyone will know what’s going on. My current web host is having major email issues, leaving me unable to receive or send messages from my lupinia.us email accounts. So, if you’ve sent me anything since the beginning of the week, please re-send it.

This comes right after I’ve acquired the last of the necessary funds to start the transition to the new server, so I’ll be rolling that out as fast as I possibly can.