As of sometime this morning, musicalmousemat.com is offline. I didn’t take it down, we exceeded our monthly bandwidth for that hosting account, which isn’t something I ever expected to happen. To be honest, in the decade or more that I’ve been building and running various websites I have never seen one exceed the monthly bandwidth limit.
It turns out that our store is using over 100GB per month of data. That’s far more than even simplysyndicated.com and this site runs all our RSS feeds! That uses a lot of data. After checking the logs it shows that the 100GB was used by under 1000 people. That means that our store system, Magento, uses an obscene amount of data to work.
There isn’t much I can do about the existing store hosting account. It really is over its limit and won’t come back on line until the end of this month. Obviously that’s not very good so I’ve set up a new hosting account with a different provider and am now in the process of moving things over. This can take time, as I write this I’m waiting for the domain name to change over to the new account before I can even start trying to transfer things.
With any luck I’ll be able to get things moved and back up and running as they were yesterday. If things go well all user accounts and purchases will be available.
The worst case senario is that I can’t transfer anything and I have to start again from scratch. If that’s the case please don’t worry about any files that you’ve bought but haven’t downloaded yet. I have complete records on who has bought what, so if you email me I’ll be able to link you to the files you’re missing. Hopefully we won’t have to do that.
Keep watching this space for updates on the situation. Sorry for the problems, it’s driving me nuts too.
UPDATE 1: Our existing host suggest that I buy a dedicated server for our store at a cost of $449 per month. To put things in perspective simplysyndicated.com costs $4.99 per month. I have already signed up for an account that will cost us £16 per month. I don’t think we’ll be getting the dedicated server.
UPDATE 2: I worked out that according to our stats, each visitor to our store is using over 100MB of data just by browsing the site. Obviously users can’t be using that much data, the site would be so slow to load that you just wouldn’t bother. The hosts are looking in to it.
UPDATE 3: It looks like an update that was performed in June is to blame for the problems. I hate updates. You get something working great and then somebody comes along and tells you they can make it work better. Then like an idiot I go and do the upgrade just to get rid of that “there’s an update available” thing that pops up every time I log in.
UPDATE 4: Couple of bits here, It’s looking more and more like the recent update is what caused the problems.When all is well, you click on a link to download a file. The store system then looks up that file in the database and connects you with it on AmazonS3. I think that the store was downloading the file, then delivering it to you instead, which would account for the sudden spurt in our bandwidth use. Our hosts have been good enough to increase our bandwidth so I can get in and have a poke around. Unfortunately I had already started rebuilding the site on another hosting account so the domain name had already been redirected. I’m now waiting for it to change back so I can get in to the old system and at least make a backup of the database so I can maybe restore it to a new install if necessary. I’m tired, cold and hungry. Please send Jaffa Cakes and Henderson’s.
UPDATE 5: I’m back in the old system and the domain name has propagated. As far as we know the store is back up and running for now. If you’ve bought files that you haven’t downloaded yet I very seriously recommend that you go download them right away. I don’t know how long things will be up and running for. Really, go do it NOW. If you only ever follow one piece of advice I ever give you, please let it be now when I tell you to go and download all the files that you’ve bought and haven’t downloaded yet. Is that clear enough? I want to make very sure that you’ve understood what I’m saying to you. If you have bought a downloadable show from musicalmousemat.com but you haven’t downloaded it yet, then you must go there now and download it. Right now, not in ten minutes, the site might not be there in ten minutes. Also if you want to buy something and download it right away you can do.
UPDATE 6: It’s definitely doing that thing where the server downloads the file then delivers it to you. I got a chart off our stats that shows a list of different file sizes and what percentage of total downloads those files made up. Seeing as the store is just a normal website you should expect to see file sizes around the 1MB and less getting most of the requests. Things like text and small images. However on our stats a whopping 63% of files requested are over 100MB. Strangely most of the files we sell are around 110MB in size. Funny that isn’t it. I’m still not sure what has caused this problem, but it looks like the easiest thing to do would be to reinstall the store system and restore a backup of the database. In my mind it makes sense, it’s just that it’s a bit like when they try to reboot Jurassic Park and I’m the daft idiot that has to go out with the Raptors and turn the power back on. Still at least that’s better than being stuck with the kid who thinks she can work the computer because it’s running UNIX. Yeah, all the UNIX machines are the same.
UPDATE 7: Things just aren’t working properly, I need to do a complete reinstall of the whole store system. With any luck I’ll be able to restore user accounts and purchases from a backup. If it turns out that this doesn’t work then everything will have to start from scratch again. That means you’ll lose your account on the store and any purchases you’ve made. Please don’t worry. If you’ve bought things that you haven’t downloaded yet I’ll be able to hook you up with the files. I still have records of who has bought what, I just don’t have the records of what you’ve actually downloaded. If you find yourself unable to download anything you’ve payed for, please get in touch with me. In the mean time the store will be off line, but it will be coming back asap.
UPDATE 8 21:22 4th Sep – I’ve managed to move everything over to the new server. How well things are working will come over time. All the products have been ported over, but all of the downloadable ones have had their URLs removed, so I need to go through each and every episode available for sale and retype the URL of the file. That will take a little while and it will hurt. I’ll maybe watch a Bond movie while I do it. I also managed to save all the user accounts. Right now as far as I can tell you should be able to log on and see your account. Of course trying to download things that you’ve bought before I put all the URLs back will only result in you using up downloads. The look and theme of the store will be the last thing to be fixed. It’s important but not as important as basic store functionality. Things should be back to normal before too long.
UPDATE 9 03:35 8th Sep – The last four days have been quite an experience I can tell you. Now I look back over update 8 I see that it sounds so hopeful and optimistic. If only I’d known then what I know now. When I got everything installed on the new server I thought it was all working perfectly, then I tried buying something and downloading it. Downloading the file failed and I was left with a 0kb file. That has pretty much been the state of things for the last four days. I have been blessed with a couple of things. First there was hosting company support that truly helped, a bit. Then there’s Simon whom you’ll know from the World 1 Stage 1 podcast. I can’t praise him enough. He spent the last couple of days working on the problem and has managed to get things up and running tonight. I would like to thank Simon publicly for that. Anyway, the upshot of that is that we now have a working store. All that remains if for me to put things in it for people to buy and make it look nice. Things will happen in that order. In test restores of data from the old store I have managed to restore everybody’s account on the store. Unfortunately past purchases have not carried forward. That means we might be stuck with a few of you who have bought things and not downloaded them. If that has happened to you please watch this post for more information soon. I have a record of what people have bought but not what has been downloaded. I’ll be able to sort you out by way of coupon codes and the like. Panic over for now, normal service should resume by the end of the week. Then I can concentrate on making shows rather than inputting URLs.
UPDATE 10 02:45 9th Sep – About 24 hours on and things are still working. User accounts have been restored so you can sign on and buy things. As I’ve said before there isn’t a whole lot for people to buy yet but it’s all coming online slowly. Remember that your past purchases have not been saved. If you’re missing any downloads please email and I’ll send you a coupon for free downloads once everything is up in the store. I’ll also be adding download bundles of Paddy’s TV Surprise and the 24 Hour Podcast.
