We're all done! Enjoy! (Updated below with open issues)
Due to some awesome shifts with JoomlaCode to get newer, more powerful servers (thanks again to Rochen). However as a part of this we're going to have some scheduled downtime for parts of JoomlaCode as we reconfigure the system into a new environment. I'm intending this post to be a sort of rolling update page as I work through all of the items and get the different bits and pieces finalised with Rochen.
As these are pretty significant changes, I'm hoping to keep the amount of disruption to a minimum. This is one of the first times I've personally tried to shift JoomlaCode as a whole beast so I'm hoping for the best and expecting some issues to fall out. I'm aiming to structure the moves in such a way that we're going to cause as little disruption, move in as small a chunk as possible and get everything tested.
The deadline for this project is the end of November, however I hope to be up and running at least a weekend before then. More updates as they come to light.
14th/15th Nov 2009 Downtime
We're planning some significant (8 hour chunk) down time for JoomlaCode over the coming weekend (Saturday 14th November Midnight GMT to 8am). This will impact on all web based services - downloads, SVN, trackers and project services. During this time we are hoping to migrate the database from its current environment into a new server. This is the first of a few moves to migrate services from the current hardware onto newer hardware.
During this process we'll be upgrading from PostgreSQL 8.3 to PostgreSQL 8.4 for the database backend of JoomlaCode. The last PostgreSQL upgrade, from PostgreSQL 7.4, brought JoomlaCode to a new level of stability. Recently we've been having issues with the stability of the database layer with JoomlaCode so we're hoping a PostgreSQL upgrade and some beefier hardware will help us chew through even more queries as we grow.
At this point in time there is no major maintenance planned for the Sunday however if there is an issue shifting the database server, I will be redoing this again on the Sunday. For testing purposes the system may appear to be online during the scheduled downtime and hopefully I will be able to restore service earlier.
15th Nov 2009 Downtime
Due to incredibly success and speed on the 14th, the plans for next weekend have been advanced to the 15th. All services are offline as off Sunday 15th November at midnight for eight hours or until the down time is completed.
At the moment I'm working to resolve a few issues:
- On the 14th and 44th minute (or there abouts), JoomlaCode may issue a Forbidden error message. This is due to a bug in the way that Subversion 1.4+ (including 1.6.6) appears to parse the authorisation file that wasn't an issue for Subversion 1.2 that we were previously on. I've filed a bug on their tracker but I am doubtful that it will be resolved any time soon.
- ViewVC appears to be only returning the first revision for some reason. Not sure why but I also tried ViewVC 1.1.2 and it did the same so I'm still trying to work out what is going wrong. It appears like it is only returning the first revision of the repository. ViewVC is what powers the "scmsvn" pages for a project.
- There appears to be an issue with some of the mailing lists. Some systems are having handler issues (sendmail isn't liking the PHP backends in some places) and the system is also refusing some inbound emails because of identity problems. As a result items like commit mailing lists and some bug notifications may not work properly. Some emails however are working and being sent so as to the exact problem I'm presently uncertain.
Again, I'd like to reiterate the thanks to Rochen for helping to make this happen. Without the great, professional services provided by Rochen, the web infrastructure for Joomla! would not be the same.