As it often happens among my colleges, I thought I should share what I already know.
Reproducible way of getting large notebooks corrupted is to open the file from a removable storage or network drive which is not always connected. If you remove and remount the device, the file is still there. Now, one might save a file, and see error message saying "hard drive is full or no write access".
Everything seems fine thereafter. One might save again and this time there is no longer an error message. One might keep on working and does not close Mathematica for days. Later, the file is saved and opened again, and only know one learns, that the file is corrupt, and important part of the work is gone if not all of it.
Why does it happen so often with my colleges? In our network infrastructures the network drives are backed up overnight. Probably for that reason the servers drop the connections, which the operation system recovers shortly. However, the file references get broken. I guess, Mathematica tries to optimize and does not always load the full notebook to the memory. If one saves a file, which is not fully in memory, part of the file needs to be read before it can be written. As the file link is broken, the read process fails corrupting the internal state. This does not affect the opened notebook it seems, but it does affect the saved file.
If you see the "hard drive full or no write access error", it might already be impossible to save the opened notebook without it being corrupted. In the worst case one even over wrights the old file.
If such a thing happens, one should copy the whole notebook content to a new one and save it as different file. It seems, that the new file produced in this way has is not corrupt, while the original saved notebook is. If the notebook is huge (embedded data, images large graphics), it is less painful to copy all the content after navigating
Cell->Delete all output.
Good practice to avoid it happening in the first place:
- do not work with files on removable drives
- always save files with a new name, e.g. add an incremental number in the end