...you could stick a tail on it and call it a weasel.

I have had a cunning plan.

Whenever I get around to doing a backup, I am always frustrated at how crap NTBackup's UI is. It doesn't let you specify which files to backup in a sane manner.

Basically, you can select files or folders to backup manually (by clicking on the checkboxes), but that's about it.

I want it to do clever stuff, but it can't. But I think I've thought of a way to give it a brain transplant, so it will do my bidding, and be quick about it.

The Final Frontier

When it comes to backing up my files, there's one big problem - space. In these days of disks sizes in the 10s of Gb range, not many tape drives can actually back them up onto a single tape. I have a DLT4000 drive, and even that can only store 20Gb per tape - and the tapes cost about £30-40! There is, of course, reams of stuff on my hard drive that I don't need or want to backup - most of the contents of the Windows folder, for a start.

What I really want to be able to do is to say things like this:

• Backup everything in C:\SourceCode except *.ilk, *.pdb, *.ncb, *.obj, *.sbr etc. files. These files are all large, and can be regenerated as needed. As opposed to source code/resources.
• Backup everything in C:\Games, unless it's over 4Mb, in which case it's very unlikely to be one of my saved games or preferences, and much more likely to be a 300Mb WAD file that I really don't need to be backing up as I'll just reinstall the game if disaster strikes.
• And so on...

So what's the problem?

Now then. I'll tell you what the problem is. It's this. That is, the thing I'm about to tell you about. Next. You know. Thing.

The main problem with writing a backup program is actually writing a robust backup engine, and especially one that can use my high capacity DLT tape drive. There's (seemingly) nice APIs under NT that (seemingly) let you store files on tape fairly easily, but:

1. I bet they're a bitch to use, and
2. Goddamn, writing a decent backup engine is a lot of work.

It struck me this morning that if you can actually tell NTBackup what you want it to do, then it usually works pretty well¹.

Unfortunately, NTBackup doesn't let you do clever stuff with its .BKS files - the .BKS files are the 'backup selection sets'. These are Unicode text files that just have lines naming folders or files in them that should be backed up. If the line ends with /Exclude, then it specifies something that should not be backed up.

Think about that. It's just a simple text file. Ok, so it's Unicode, but that's not exactly Röckét Sçíeñcè. The only thing wrong with the system is the poor UI that won't let you select files in an intelligent or flexible way.

I think you know what I'm going to say next.

Well, duh...

For those who haven't guessed, here's the thing: we've now reduced a large problem (writing a full backup program, that copes with errors gracefully, handles tape drives, etc) to a much smaller problem, which is to write a GUI app that lets you choose the files you want, and then spits out the relevant .BKS file which you then use with NTBackup.

If we're feeling really keen, that GUI app could also automatically schedule the backups for you, but slow down, Captain - where's the fire? It's certainly not a necessity - we can do the scheduling in the standard Windows Scheduled Tasks applet in the Control Panel for now.

What could possibly go wrong?

However, there's a possible problem with this solution. Due to the very simple way that .BKS files are specified, coupled with the fact that wildcards (e.g. "*.cpp") are not allowed², the .BKS files could get very large.

For example, in the scenario I gave of backing up my source, the .BKS file would probably end up having to name every single .cpp and .h file on my hard drive. If I'm erring on the side of caution when backing up my data, this proposed .BKS file could get very large indeed. I have no idea how well NTBackup copes with large .BKS files - it may start to slow down and/or crash³.

Nil Desperandum!

There is a solution, and it is (in my humble opinion) beautiful. Think about it. We need to get NTBackup to only backup certain files on the hard drive, but we don't want to have to name them all in a text file - ideally, we really just want to name the folders we want to backup, and then somehow tell NTBackup which files to skip.

Go on, have a think about what the solution is.

I can wait.

Let's restate the problem: to put it simply, if only we had a flag we could apply to an individual file that would tell NTBackup to backup or ignore the file.

A Thing of Beauty

You know the answer now, of course - it's our old friend the 'archive bit'. Thank heavens for MS-DOS, eh? Files on Windows disks (FAT and NTFS formats) have a number of attribute bits. One of these is the archive bit. Whenever Windows (or DOS, come to that) writes to a file, or changes it in any way (e.g. renames it) then the archive bit is set. This might seem pointless, but then your friendly backup program comes along and backs up a set of files, as specified by you⁴. It then clears the archive bit on all the flags it backed up (and only those files).

This lets you do the 'differential' backup, which usually means "Backup all the files I've asked you to, but only if they're been changed since the last backup'. The backup program simply looks at the archive bit on a file, and if it is set, the file has changed, and should be backed up.

So there you go - that makes it easy - all our GUI app has to do is produce a minimal .BKS file, and then set the archive bits on the files we do want to backup, and clear them on the ones we don't want to backup (if they're in folders that we've asked to backup), and then tell NTBackup to do a differential backup.

Essentially, we're fooling NTBackup into doing what we want.

And I like that.

It occurs to me that maybe someone else has had such an idea, but a quick Google reveals nothing obvious. If you have heard of such a program, let me know. Otherwise I'll be thinking about how to start implementing this damn useful utility.

I love it when a plan comes together.

1. As long as you remember to use the /UM option, otherwise it'll just sit there asking you if you really want to back stuff up onto that tape, which makes it useless for unattended backups.

2. Yes, I know, crazy, isn't it, but wildcards are not supported in NTBackup .BKS files - read the NTBackup help file, and/or try it out yourself - I know I have.

3. Bear in mind that this is a backup program that doesn't accept wildcards.

4. Well, almost - that's why we're here :-).

« Sweet Georgia Blog | They're all morons from the planet Moron »