Personal Backup Scripting Example

I’m using Personal Backup by Dr. Rathlev as my backup tool. It’s name is a bit misleading, because you could think that it is only allowed to use it as a private person. But no – its current license says it may be used also by any club, organization or even private companys for free.

It is easy to create backup tasks with Personal Backup. But a backup task does not run automatically. And I have to say, it is not easy to create backups that run automatically with this program. The UI of the part of the program which deals with creating automatically running backup tasks is near incomprehensible – even when you read the docs.

What compensates this deficit is that Personal Backup has a perfect command line interface with a good docmentation. So it is very well suited for being scripted.

I control my backups with some tclkit scripts. As you might know, I do use tclkit for many tasks.

The script below creates a full backup in every uneven month (Jan, Mar, May, …) and in between it creates an incremental backup daily. It needs a fitting Personal Backup task file Backup-Task-All.buj.

To trigger the script daily, I use the Windows Task Planner.

The directory structure of the created backups looks like this:

G:/Backup/2014/BD01F   # A full backup created on March 1st in 2014
G:/Backup/2014/BD22I   # An incremental backup created on March 22nd
.....
G:/Backup/2014/BD21I   # An incremental backup from September 21st 
#
#  This script steers the backups that I do with Pesonal Backup.
#
# To adapt the script, just set the following variables taskName,
# rootdir and personalBackupExe to your liking. 

# The name of the backup task in PB
set taskName Backup-Task-All

# The name of the target root directory.  This must be the same as 
# in the Backup-Task mentioned above.
set rootdir G:/Backup

# Full path to the Personal Backup executable
set personalBackupExe "C:/Program Files/Personal Backup 5/Persbackup.exe"


#console show

puts "BackupSteer.tcl  (c) A.J.W 2014"


set rootdrive [string tolower [string range $rootdir 0 0]]
set seconds [clock seconds]
set yearMonth [clock format $seconds -format {%Y/%m}]
# the name of the basedir of the backup of today. E.g  G:/Backup/2014/09
set basedir [file join $rootdir $yearMonth]


puts "    basedir=$basedir"
puts "    taskName=$taskName"

# Return 1 if it's an even month, 0 if not.  
proc isEvenMonth { seconds } {
    set month [clock format $seconds -format {%m}]
    if { [string range $month 0 0] == 0 } {
        set month [string range $month 1 1]
    }
    set even [expr $month % 2 == 0] 
    return $even 
}

# Return a list of all connected drives. 
proc drives {} {
    foreach drive [list a b c d e f g h i j k l m n o p q r s t u v x y z] {
        if {[catch {file stat ${drive}: dummy}] == 0} {
            lappend drives $drive
        }
    }
    return $drives
}

proc doBackup {} {
    global  taskName  seconds yearMonth basedir  personalBackupExe 

    set relativeDirForFullBackup BD01F
    set pathForFullBackup [file join $basedir $relativeDirForFullBackup]
    puts "    pathForFullBackup=$pathForFullBackup"

    # if full backup for this month already exists or we have an even month
    if { [file isdirectory $pathForFullBackup] || [isEvenMonth $seconds] } {
        # only do an incremental backup
        set mode incr
        set day [clock format $seconds -format {%d}]
        set relDir BD$day
        append relDir I
        set pathForIncrBackup [file join $basedir $relDir]
        puts "    pathForIncrBackup=$pathForIncrBackup"

        if [file isdirectory $pathForIncrBackup] {
            set mode none
        }

    } else {
        # do a full  backup
        set mode full
        set relDir $relativeDirForFullBackup
    }

    puts "    mode=$mode"
    puts "    relDir=$relDir"



    if { $mode != "none" } {
        puts "    Running backup like this:"
        puts "    $personalBackupExe $taskName /force /hide /mode:$mode /prompt:delay  /backupdir:$relDir & "
        # If I do not add the & at the end, Persbackup hangs. 
        exec $personalBackupExe $taskName /force /hide /mode:$mode /prompt:delay  /backupdir:$relDir &
        puts "     Backup running." 
    } else {
        puts "    Nothing to do."
    }
}

if { [lsearch [drives] $rootdrive] == -1 } {
    bell; bell; bell
    set a "The daily backup is about to be done. "
    append a "Please connect the backup drive, make it "
    append a  "available as network drive $rootdrive: and click Ok."

    message .m  -textvariable a -width 250
    # strange, the width of the button is in another dimension than that of the message. 
    button .hello  -text "Ok" -command { 
            if { [lsearch [drives] $rootdrive] != -1 } { doBackup; exit } else { bell }
        } -default active -width 15

    bind . <Return> {.hello invoke}

    pack  .m .hello -padx 5 -pady 5

    wm deiconify .
} else {
    doBackup
    exit
}

Unit Tests, Unicode and Special URI Characters

Unicode

Under NTFS, nearly all unicode characters can be used in file names. Characters like ÄÖÜ äöü ß, 新資料夾, Видео, 新增文字文件 may appear in file and directory names.

If you are dealing with software that handles user created file names you should always test your program with names containing several different unicode characters from different character sets.

Special URI Characters

Also, there are some characters that are special characters in the URI specification. For me, these special characters are the reserved characters :/?#[]@!$&'()*+,;= and the unreserved characters which are not alphanumerical or digits -._~.

As URIs and filenames are often used interchangeably, you need also to test with file names containing the special URI characters.

I hope, that you are seeing all the unicode and special characters on your screen like I’ve intended them 😉

Filenames Which Look Like Encoded Uri Characters

And, further, as in some cases file names get URI-encoded and -decoded it is also necessary to test with file names that look like some encoded special URI characters. I mean file names like %25%20.png or %23.png. Especially important are the encoded space %20 and the encoded percent %25.

Zip Archive and Tcl Script

To make your life easier, I have created a zip archive that contains various files with file names and path names with a set of the mentioned special characters that is usually enough for you to test. All the files are the same small plain text file, just named differently.

You can download the set of files here: TestFiles.zip

It may well be that for your testcase you cannot use the tiny txt files in my zip. Maybe you need pngs or jpegs or whatever. For that case, I’ve created a Tcl script with which you can copy a file of yours to all the differently named files.

Download the script here: createTestFiles.zip

To run the script, you need tclkit, which you can download here.
You can find some more info about tclkit in another post.

Happy testing!

Tclkit: A Tiny Full Featured Scripting Language

Tcl/Tk

Tcl is a full featured script programming language with a small footprint. It is available on many platforms. Tcl’s syntax looks a bit odd if you are coming from a C-like language. But in reality, it is pretty simple … Tcl is a small language and you’ll learn it fast.

Tk is a cross-platform GUI system that has been built for Tcl. Together, they form Tcl/Tk. Tk is used with many other languages (Perl, Python, Ruby, …) too.

Tclkit and Tclkitsh

Tclkit is Tcl and Tk and several libraries altogether put into one single executable. No installation is needed.

For Windows, there is also available tclkitsh.exe which contains only the command line version of Tcl and libraries as one single executable. It is among the first ten things I put onto a new computer.

Why I Use Tclkit

  • Deployment nearly can’t be easier. You just have to copy the single tclkit or tclkitsh executable and your script file. I’ve used this method with my zip speed test program. You can download tclkitsh.exe from there.

  • It is tiny. Tclkitsh.exe 8.5.9 is only 740 kB. Tclkit.exe,
    which contains the full Tk GUI is only 1.3 MB

  • Development with tclkit is fast. No compile-run cycles.
  • The GUI system is easy to handle. You don’t even need any visual
    tool to create your GUI.
  • The windows cmd shell language is just terrible. That beast
    cannot even be called a language. If you don’t want to dive
    into Pwershell, Tclkit is a simple and perfect
    replacement for the windows cmd shell.
  • Compared to Perl and Python, it is easier to learn,
    read and understand.
  • It is perfect for doing file operations in your build process.
  • It is perfect for controlling other executables,
    e.g. in your build process. In fact, Tcl has been built
    with the main target to be a language to control other executables.

Effektiv Tcl/Tk programmieren . (Programmer’s Choice)

Example 1: Check If Drive Is Available

I do my backups onto an external USB drive. The backup program is controlled via the Windows Task Scheduler. For security reasons, this drive is usually not connected. This means, before the backup program can do its work, I have to manually connect the drive.

I’ve written a small Tcl/Tk program to check if the drive is available and to inform me that I should connect it if it’s not.

This is the Tcl/Tk Script. With graphical interface and bells.


proc drives {} { foreach drive [list a b c d e f g h i j k l m n o p q r s t u v x y z] { if {[catch {file stat ${drive}: dummy}] == 0} { lappend drives $drive } } return $drives } proc checkDrive {} { if { [lsearch [drives] z] != -1 } exit else { bell; bell } } set a "The daily backup is about to be done. " append a "Please connect the backup drive, make it " append a "available as network drive Z: and click Ok." checkDrive # Create a label or message called .m message .m -textvariable a -width 250 # Create a button called .hello # strange: the width of the button is in another dimension # than that of the message. button .hello -text "Ok" -command { checkDrive } -default active -width 15 # This binds the return key to the pressing of the .hello button bind . <Return> {.hello invoke} # Pack the message and the button together. pack .m .hello -padx 5 -pady 5 # Show the main window wm deiconify .

When this tiny script is run, it checks if drive Z: is connected and if not, it shows the following message box. When the Ok button is hit, it checks again.

0806-223826-BackupPre

Example 2: With SVN, Show All Changed Files

I am working on a system with svn as version control since some days. I have not yet found a fine graphical svn client, aside of TkCVS. (Yes, TkCVS handles svn quite well, despite its name.)

One of my first problems was to find out which files are changed locally in several unconnected checked out directories. A task for which tclkitsh is suited very well.


proc isEmptyDirectory { path } { set isED false if { [file isdirectory $path] } { set content [glob -nocomplain $path/*] if { [llength $content] == 0 } { set isED true } } return $isED } proc handleEntry { dir entry } { set path [file join $dir [string range $entry 8 end]] if { ! [regexp -nocase {( bin|/bin| obj|/obj|.user|.suo)$} $entry] && ! [isEmptyDirectory $path] } { puts $entry } } proc statDir { dir } { cd $dir puts "" puts $dir if [catch { set l [exec svn status . | findstr /V Pics ] } ] { # above, I'm piping the output of svn status . through # findstr. Just as an example. puts "" } else { set l [string map { /} $l] set l [split $l "n"] foreach s $l { handleEntry $dir $s } } } while { 1 } { statDir C:/code/Client/WebClient statDir C:/code/Client/ServiceLayer statDir D:/src/TraceOfDeathApp statDir E:/src/AspServer puts "Press Return key to repeat." gets stdin }

Notepad++: How to Make the Function List work with Tcl and Bash

In another post about Notepad++ I critisized that its function list feature does not work for Tcl and Bash scripts. Now I’ve got a solution. Here is it.

How to Make the Function List Work for Tcl

Open functionList.xml in an editor.  FunctionList.xml is in %APPDATA%\notepad++ or in the installation directory of Notepad++.
Add this line to the section with all the other association-entries.

 <association langID="29" id="tcl_procedure"/>

Add this to the section <parsers>

<parser id="tcl_procedure" displayName="Tcl source" commentExpr="(#)">
    <function
            mainExpr="^[\t ]*((proc)[\s]+)[^\n]+\{"
            displayMode="$functionName">
        <functionName>
            <nameExpr expr="[\w: ]+ \{.*\}"/>
        </functionName>
    </function>
</parser>

Restart Notepad++. Credits for the solution go to Detlef of compgroups.net/comp.lang.tcl. I’ve copied his’ and shortened it a bit. And added support for bash.

How to Make the Function List Work for Bash

In functionList.xml add this line to the section with all the other association-entries.

 <association langID="26" id="bash_function"/>

Add this to the section <parsers>

<parser id="bash_function" displayName="Bash" commentExpr="(#)">
    <function
            mainExpr="^[\t ]*((function)[\s]+)[^\n]+"
            displayMode="$functionName">
        <functionName>
            <nameExpr expr="[\w: ]+"/>
        </functionName>
    </function>
</parser>

Restart Notepad++.

Download

Or just download my functionList.xml with Tcl and Bash support and replace yours.

Effektiv Tcl/Tk programmieren . (Programmer’s Choice)

Update: As Olivier from comp.lang.tcl pointed out, my version does not work with Npp 6.4.3. I’ve rechecked this, he is right. My version does work at least with versions 6.4.5, 6.5 and 6.5.5, though.

Update 2: Thanks to Rasha Matt Blais for his improved version of the mainExpr for bash functions. Though that one still misses some types of function definition. Here my latest mainExpr string:

mainExpr="^[\t ]*((function)[\s]+)[^\n]+|[\w_]+[\s]*(\([\s]*\))"

I’ve also adapted the downloadable functionList.xml. And I’ve added the testfunc.bsh file which contains some bash functions with which I’ve tested the my mainExpr string.

Why Zip Is a Better Archive Format than 7z

Scott Hanselmann declared 7z to be a much better format than zip and zip to be dying.
Or at least, this is how I understand his writing:

The 7z format is fast becoming the compression format that choosey hardcore users choose.

Though I’ve got a lot of respect for Scott, I have to add some facts.

Zip is Much Faster Than 7z

zip-7z-comparison.1 In my humble opinion, Scott has overseen that zip is much faster than 7z… I’ve done  tests with both formats and with both formats I’ve used compression methods 1 and 5. Some of the results are staggering.

 

Discussion of the Results

compressFullzip-7z-comparison.3

In the compressFull tests, a data set is compressed and added to a new archive.  As you can see above, compressing with zip-1 is around 4 times faster than compressing with 7z-1 and seven times faster than compressing with 7z-5. The resulting archive is only 6% bigger than the 7z-1 archive and 40% bigger than the 7z-5 archive. In my opinion, the much greater speed speaks clearly for zip.

extractFull

In the extractFull tests, all the files in the archive created by fullCompress are extracted locally. Here, the speeds are not too different, e.g. extracting from a zip-1 is only around 30% faster than from a 7z-5 archive. As the 7z archives are smaller, I rate this as a draw.

extractSome

zip-7z-comparison.2

In the extractSome tests, only a small number of all the files are extracted from the archive. Extracting from a zip-1 archive is 50% faster than from a 7z-1 archive and an astonishing twenty times faster than from a 7z-5 archive. As this is unbelievable, I have repeated the tests a lot of times. But it stays true. Victory by knockout for zip.

Test Details

  • I’ve done the tests on my oldish Fujitsu Siemens Lifebook S with a Dual Core Processor, running Win 7.
  • For both archive formats I’ve used the commandline version of 7-zip 9.20, controlled by tclkitsh.exe and the tcl script given below. It is the current and stable release of 7-zip. Unchanged since 2010.
  • The compression method 5 is in 7-zip the default value for both formats. Compression method 1 is the fastest compression method for both formats.
  • I’ve repeated each test several times. The standard deviation of the results have mostly been very small.
  • My data set has been a set of 2800 files with 192 MB uncompressed.
  • The files which are extracted in the extractSome test are 72 files with 68 kB uncompressed.

Reproduction of the Tests

You can reproduce the tests easily:

  1. Download my script zip-test.zip and tclkitsh.zip and extract both into the same directory.
  2. Open the file zip-test.tcl with a text editor and adapt the first three lines. In the first line, adapt the path to the 7z.exe file on your PC. In the second line, adapt the path to the directory you want to compress.
  3. Open a command window, cd to the directory where you’ve put zip-test.tcl and type in     tclkitsh zip-test.tcl
  4. Wait and don’t use the computer until the tests are finished.
  5. The results will be written on screen and at the same time appended to the file zip-test-results.txt.
  6. You should discard the first test, because for the first one, the speed is highly determined by the time you need to read data from the hard drive. In later runs, much of the data is in the OS’s drive buffer. So the first test is not comparable to the following ones. It does not measure compression speed, but hard drive speed. You’ll see that the first run (with zip-1) takes much longer than the following ones, even those with zip-5 or 7z-5.

Summary

  1. All operations on zip archives are much faster or as fast as the same operation on 7z archives.
  2. 7z archives are somewhat smaller than zip archives – but not much.
  3. My recommendation is:
    Use zip as archive type. If you are using the software 7-zip, do not use the default settings. Always use compressing method 1.

What is your opinion? I’d love to hear from you.