OK, after a two weeks of upset customers I'm starting to realize that there are some inherent flaws in how I've configured DMX to store files.  Lets assume the following.

DMX is installed, WebDav is enabled and there are 4,000+ folders containing 50,000+ individual files, all are supposed to be protected by DNN authentiation (no public access).  The site is hosted on an internet accessable server. Activity on this site is very high with folders/files changing almost by the minute.

I've setup the Storage provider to use the default Disk File Storage Provider with the repository location in the default location. I've also set it to change the extentions. Hard Deletes are off.

Problem #1 - WebDav User inadvertantly drags and drops folders into the wrong place - moving hundreds of subfiles and folders.

Problem #2 - WebDav user deletes a folder which contains - gets around the recyclebin (apparently a bug in DMX) and files are in the graveyard.

Keep in mind that activity on this site is very high and restoring the portal in it's entirty is not a solution.

How do I get this data back?  I could do a point in time restore of the DB but then I'm missing the files.   Lets say I have a backup from last night with all the missing files... how do I re-import those files into a folder that doesn't exist any longer.

 

What is the best way to setup a backup routine and restore routine in this scenario?  Both of these problems have already occured and I have 3 other clients coming online in the next 2 months that will face this same problem.  There has to be some way to reliably backup and restore this data.

 

If I wanted to convert from one Datastore (disk File) to another (SQL or Amazon S3) what are the procedures for doing so? How do I export all the data (and retain permissions and folder structure)?

Hi Robb,

First let me say I'm sorry about your woes. To migrate to a new storage provider set it as the default and run the script "move old content to new storage provider". Please backup everything before you do this. And use DMX 5.2.3 or higher.

WebDAV potentially hard deletes content. This is due to the nature of WebDAV which does not know about soft deletes. You can switch off hard deletes for WebDAV (switch is on the Main Options screen) but the consequence is that soft deleted documents will be "revived" when a user drops a file with the same name in the folder. It will revive the old one and create a new version.

Problem #1 is tied to the power of WebDAV. Yes, drag and drop works and users will be able to do massive amounts of damage with this. There is no technical way to interfere here. I can't pop up something in the user's Windows to warn. And switching off drag n drop seems rather draconic and would result in error message popping up for the users as WebDAV assumes this is functional.

We're looking at including a "restore" function to bring back content from the graveyard in the next major release.

Peter

Ahhh... got it :)

I found the change storage providers script yesterday after posting, along with an export script (I gotta read more of the docs when poking around)

I know that getting data from the graveyard is on the to-do list and you and I have discussed it before.

What I was really looking for (and would appreciate any insight) is what is the best way to to setup DMX to faciliate backup/restore on a highly active portal?

If I use Disk the based provider, I can do physical backups every hour to get the files, then use the SQL transaction logs to do a point in time restore. The problem is that it would wipe out any changes after that poitn in time and in this case there could be hundreds of changes in a normal hour.

If I use the SQL based provider, I don't have to worry with the hourly backups and can just use the SQL trans logs to do the restore, but I run into the same problem with the point in time restore. An alternate would be to restore to a diff db and then use SQL compare by redgate to move over the SQL records that I needed to restore.

A hybrid option would be to hack your export script and set it to export all files and fodlers changed in "x date/time" to a secondary location for backup. then if a file/folder was moved/deleted I could just re-import that from the exported backup. The problem here is the time and load on the server to maintain two sets of files/data.


Right now I'm looking at the SQL datastore with RedGate SQL compare to see if that scenario would work out the best.

thanks
robb

Hi Robb,

I can't say I see an easy solution. Rolling back is incredibly hard as the data model is quite extensive. It's not very complex, but there are lots of bits and pieces that you'd need to puzzle back again. It's probably easier to program a Graveyard restore yourself than set up this kind of thing. No easy way I'm afraid.

BTW I use Red Gate myself and it has saved the day on many an occasion, but I've never attempted this.

Let me know if you have questions about the data model.

Peter