sfManagedCachePlugin

The rough plugin for Symfony is done, albeit not published yet. Usage is very similar to the sfFunctionCache class, and it comes with the appropriate tasks to refresh/clear cached items.

Usage example:

$sfManagedCache = new sfManagedFunctionCache(
  new sfFileCache(array(
    'cache_dir' => sfConfig::get('sf_external_cache_dir').'/sfManagedCache',
    'lifetime' => 86400))
);

$xml = $sfManagedCache->call(
  array('Webserice', 'call''),
  array($parameter1, $parameter2),
  1800
);

The tasks are used like this and should be set up as cron jobs:

./symfony managed_cache:refresh --application=webservice --env=prod
./symfony managed_cache:refresh --env=prod

The first call is for a specific application while the second one is for application-agnostic cache items (not for all items!).

This is not a fire-and-forget plugin. It requires planning and maintenance. Since the cached items are designed to never expire, carelessness can easily lead to problems. I will cleanup the plugin some more, write documentation and then release it.

Cachtrocities

This week, I've had to incorparate a vendor's webservice into the rendering of an existing website. It was going to replace the programatic generation of URLs for a list of files that used to be hosted internally. In the process I had to deal with X things:

1) The data was raw and needed to be reprocessed before we could use it. That meant parsing the XML, running it through the processor and then writing it out again. Too much effort to do on every request, so again data would have to be cached.

2) More importantly, the webservice performance was atrocious. Request times for a 4 item rss feed hovered around 5 seconds. A completely rendered page could require several calls to this service. Whether these long request times were due to problems on our end, their end or somewhere in between was unimportant for the moment. The results would have to be cached to be useful.

Naturally, I could have left the caching to the application that used the webservice, ie. parse the information provided and store the results in memcache. The problem with that approach is that memcache is a very simple cache, and paired with the long request times for the webservice you're bound to run into problems. No matter how scalable your application and infrastructure is, if a reasonably common request is taking 5, 10, 15 seconds to complete, you're going to see a thread pileup sooner or later, which is what will happen when the items in memcache expire. And don't even get me started on the degraded user experience.

The proper solution for this is a managed cache.  One important concept is prefetch, ie. refresh the cached data before it expires, so it doesn't matter how long it takes to create it. WP-Supercache has this feature and Varnish recently introduced this feature as well along with grace. Grace is the other concept, which allows you to continue serving up stale or expired data if the data source becomes unavailable. In most cases, serving up old data is better than no data at all.

Symfony doesn't have any plugins or classes or other facilities to deal with these kinds of things as of right now, so I decided to just deploy Varnish for the purpose of caching the modified webservice data. No development time and very little effort to deploy and working perfectly. Unfortunately, I have to abandon that setup because of "complexity concerns". So in the next couple of weeks, I'll probably write a managed cache plugin for symfony.

Life

Life got in the way of a regular posting schedule. Right now, I'm preparing an article about a cache primer and passive cache refresher for symfony. Stay tuned.

Quick Thought

Here's a video of Dell showcasing their new tough laptop. I think it's hilarious that there's one guy whose job it is to gently push the laptop off the the table and then there's another guy for picking it back up.

The Internet isn't serious business after all

I hear tell that South Carolina's Department Of Labor accepts online applications for unemployment benefits only during business hours. I'm sure there's a funny story behind that. I'm equally sure the newly umemployed don't care.

In South Carolina, the Internet is still a 9-5 business.

Excel bug causes Microsoft to overpay severance

I mean, it could happen.

WTF

UPS sends a package that had already made it to Rockhill, SC back to Columbia, SC so it can go back up to Charlotte, NC. What is going on?

How to operate Symfony in a virtual folder

Problem:

I needed to run symfony in a virtual subfolder under an existing site, so that the asset URLs generate by helpers still originate from the root folder while URLs generated from routes contain the virtual subfolder. For example:

I have a site xyz.com that is run on some legacy platform that uses the same sort of layout for assets as symfony (stylesheets in /css, etc.). All the traffic for xyz.com goes through a load-balancer that splits off requests which URL starts with the prefix /symfonyApp and sends them to the server(s) running the symfony app. Symfony needs to generate asset URLs that look like /images/logo.gif while generating action URLs looking like /symfonyApp/signup/index.

Analysis:

The reason I was able to make this work at all is that the helper functions and the routing classes use different ways to insert the prefix folder name. Helpers get something called "RelativeUrlRoot" from the request object. This value can be set in factories.yml. Otherwise, it's calculated on demand by taking whatever is in front of the script file name in the SCRIPT_NAME environment variable.

Routes, on the other hand, store the prefix internally, though it's initialized to "RelativeUrlRoot + ScriptName", unless you override it in factories.yml.

Solution:

One solution would be to set the desired prefix values in factories.yml. But that requires duplicating the desired environment, which would be a pain to replicate. Instead, I opted to modify the values in the script file prior to dispatching the request. This way seems to work smoothly.

<?php

$tmp = dirname(__FILE__);
ini_set('error_log', $tmp . '/../../log/frontend_virtual_prod-error.log');

require_once($tmp.'/../../config/ProjectConfiguration.class.php');

$configuration = ProjectConfiguration::getApplicationConfiguration('frontend', 'prod', false);
$context = sfContext::createInstance($configuration);

// set separate prefixes for assets and links
$context->getRequest()->setRelativeUrlRoot(");
$routing = $context->getRouting();
$routingOptions = $routing->getOptions();
$routingOptions['context']['prefix'] = '/symfonyApp';
$routing->initialize($context->getEventDispatcher(), $routing->getCache(), $routingOptions);

$context->dispatch();

Vonage

I've recently switched home phone service to Vonage, away from my local cable provider which was providing basically the same service (VOIP) with less features for more money. A service I hardly used anyways. The most egregious bug/feature with the old company was that I HAD to dial "1" in front of every number with a different area code, while for numbers within my area code, I HAD to leave out the area code. Pair this with the fact that the caller ID service they provided didn't normalize numbers to account for their idiosyncratic and completely retarded system, and you have a major stroke-inducing situation at your hands.

Anyways, I made the switch and I couldn't be happier to have access to modern age phone service. I can manage my voice mail box from my browser and voice mails are emailed to me as a .wav attachment. I don't like voice mail in the first place so that makes it much more pleaseant to use this necessary evil. I am always delighted when competion and innovation create a benefit I actually care about.

New memcache client / updated old memcache client

It's a sad truth that many good and relatively popular open source projects die a slow death because they lose there maintainers and nobody steps up to replace them. One example would be PHPEclipse, a remarkebly stable and feature rich PHP plugin for Eclipse. For the longest time, it was leaps and bounds above PDT, the "official" PHP plugin. (PHPEclipse has recently been revived) The issue is only exacerbated if language the project is written in is unfamiliar to its user base. Eclipse plugin development (Java) is a huge leap even for a professional PHP developer.

This sort of problem strikes far less often when the project in question plays an important part in a successful company, since they have the resources to pay someone to contribute/maintain. It's not a guarantee though. Memcache is an excellent example. It was developed by Danga for internal use in Livejournal. It was later made public and quickly became intensely popular. The server part is in pretty good shape, although there are only about 2 releases a year. Facebook is contributing heavily now, though. Hopefully, we will see the benefits of those contributions soon.

The client library is another matter. Development there has been pretty anemic, probably coinciding with Dango's aquisition by six apart (just a guess). The C library libmemcache which serves as the underlying implementation for the higher-level language specific libraries is not being maintained anymore. The perl library has seen no update in almost 2 years. The other script languages seem to be doing better. Even PHP, which PEAR/PECL projects often suffer from abandonment issues, has been steadily maintaining their client library and even has a new major release in beta right now.

Even so, the underlying problem remains that libmemcache is dead. From digg's technology blog I now hear that not only is there a new C client library under heavy development (libmemcached), but they also hired a PHP developer to work on a new PECL extension that utilizes the new library. This is excellent news and should give memcache's client side a much needed boost to go along with the upcoming server side improvements.