So whilst at work this morning I noticed I had several alerts from LibDemBlogs telling me it had been unable to create the half hourly updates. I get these once in a while, but as long as it’s just one it will be fine on the next build. However these were pretty much every 30 minutes.
With a spare few minutes I could see in the logs that a request to read an RSS feed of a site had gone from taking a few seconds to over a minute, and LibDemBlogs reads the RSS feeds of over 400 sites.
However I had no time to take a further look, but managed to send an email to the hosting company to see if they were aware of any issues. A reply later in the day informed me that indeed there was an issue with packet loss in to the datacentre and they would let me know any more updates.
On getting home I’ve been able to activate a couple of settings, the first is to stop reading the blogs of anyone who hadn’t posted anything in the last 60 days. This reduces the number of blogs it needs to read to around 200, but means that if you haven’t blogged for a while, and just created a new post it won’t be spotted.
It’s helped and the site managed to build the 8pm update, however as it’s giving up on reading the RSS feeds of some blogs there is a chance your post just now won’t appear for a couple of hours.
My hosts have a plan of action late tonight, so there will be some downtime near midnight. Hopefully this will fix the issue, and I can role back some of the tweaks.
As a side affect this also impacts Liberal Tweets, LibDig and dab.argh where you might see random errors or timeout messages.