I deal with his company network but have nothing to do with his web server or associated web development. He has always had an outside company deal with it and I was always out of the loop on it and never been involved.

He calls me in a panic yesterday about the website being down and he needed my help. He had spoken to IBM Cloud support and they said the cause was due to the fact he was out of space on his server and that he needed to free up space on the partition and explained what he needed to do. I asked him why didn't he call his web developer and he said he had an argument with them about the site being down. And now he was looking for another developer.

So in the short term I was his only option I guess. The steps the IBM Cloud tech told him he needed to do of course are over his head so he asked me to call IBM support back since he figured I would understand it better than him. While I certainly can understand it better than him I do not know Linux well and certainly don't know IBM Cloud or anything about web server administration.

So I contacted IBM support went thru the steps with them which involved running Rescue Mode from the IBM Cloud dashboard and then going in thru Putty to delete unneeded files or folders from directories on the mounted partition.

We found a directory in the root of the partition called Backup which inside of it contained a folder called OST_BACKUP. Thought that subfolder was odd but it contained a couple of dated backups. One from May and one from June . Decided to delete the older one called 2022-5-15 that was 82 gigs in size. Neither I nor IBM Cloud support know what exactly was contained in those backups but certainly weren't worried that deleting one would cause a problem.

So from inside the OST_BACKUP directory he had me run the RM command on 2022-5-15. Upon doing so it started to the process although it doing it it was failing to delete quite a few files that it was claiming were busy or lacked permissions. The files seemed more like system files to me which I thought were odd. Once it finished it seemed Putty then lost the connection and I could not reconnect

Now I was getting nervous. He said he wasn't sure why but suggested we go back to the IBM Cloud Dashboard and run Rescue Mode.

Well we did that and I was now able to get back into via Putty but now the partition was empty except for sone basic default directories.

So of course I'm freaked out and asked him if we can restore the partition from a backup for which he told me that is only if backup services were set up. I would have assumed my clients web developer would have had backups of the server set up. But the IBM Cloud support guy said he did not see any indication of that in the backend.

Makes no sense how running an RM command inside the Backup directory on a specific subdirectly could wipe out the entire partition. So as of right now my client is screwed. He left a message wit his web developer but they have not gotten back to him. Can't believe they never bothered to setup any sort of cloud based backup service for the server.m on the event of catastrophic failure. Who runs an unmanaged server with no backup routine in place. He never should have been running a private unmanaged server if he did not have someone to properly manage it.

So not sure what he is going to do. Unless the web developer somehow has a backup copy of the site and is willing to cooperate with him at this point. He is fucked.

UPDATE: He did finally hear back from his web developer he had the argument with involved and they are working on it but although I am not having high hopes unless they indeed somehow had a backup that me nor the IBM tech were aware of. He also told me that the development company also has a department that handles the server management administration so I assume they are the ones involved now.

I asked my client why he even originally spoke to IBM support rather than just calling the development company and letting their admin department handle it and his excuse was he thought it might be easier and faster to reach out directly to IBM and subsequently have me work with them to get it back up and running since "I know computers" and the "technical jargon". Sadly I caved and gave in even when I told him am unfamiliar with any of it.