We had to do a 12 hour rollback. It's horrible and we hate having to do it. Unfortunately it was the only option we had.

The problems were caused from a newly installed backup system that was advertised as being able to handle the job but for whatever reason it couldn't. We ran it on the test server for almost a week and we had no problems with it. Its a very small server with a very small map though. The backup system wasn't able to keep up with the large server that we have. It caused memory leaks, it corrupted files and it was allowing items to be duped in game.

Unfortunately the newest backup that wasn't affected was from 12 hours before the server was turned off. Unfortunately we had to rollback until then.

We will be putting measures in place so that if something like this happens again, it won't be such an impact. We understand that doing a 1 hour rollback is horrible enough and that 12 hours is ridiculous.