Delete older files with Unix find and xargs

find . -type f -mtime +5  -print0 | xargs -0 rm
find . -type d -depth -print0  | xargs -0 rmdir  2>/dev/null

With these two bash commands, you can recursively delete all files older than 5 days and after that all now empty directories. I use these commands to clean up temp directories.

In the first line -mtime +5 finds all files that are older than 5 days. The + sign is important. Without it, only those being exactly 5 days (5 * 86400 seconds) old would be found.

In the second line -depth makes that child directories are treated before parents. That is important here. Otherwise a directory a which contained only an empty directory b would not be deleted.

rmdir only deletes empty directories. For that you don’t see rmdir's griping about non-empty ones, the 2>/dev/null is used.

Bash, Awk on Windows: Git-Bash and ConEmu

Every developer who has worked for some time on Linux or another Un*x system will miss the bash, some of the other unix tools like find, cat, grep and awk and a decent console bitterly.

There are several ways how to get some of the unix power back to windows.

  • Cygwin. It is sometimes a bit complicated to install and handle and there are often problems with line endings. I have used this but never was really totally content.
  • MinGW, MSYS and MSYS2. I have never used them and cannot tell much about them.
  • Git-Bash and ConEmu. Lately I have detetcted these tools and it seems they are working quite well. In this post, I’ll go a bit deeper on them. Git-Bash is a version of mingw32, as far as I know.

Git Bash

Git is a well known source control system. And its installation kit for windows contains bash, awk, grep, find and several other unix tools. So even if you don’t want to use git on your windows machine, you can install this package. It is free software.

Installation (with Git 1.9.5 from 2015/03/19) goes like this:

  1. Download the git package for windows.
  2. Run the installer.
    • Install it to tools/Git. Next.
    • Leave away Windows Explorer integration. Do not associate file endings. Next.
    • Do not create a start menu folder. Next.
    • Use Git from Git Bash only. (Other options didn’t work as intended anyway.)
    • Next. Check out Windows-style, commit Unix-style line endings.
    • Next. Wait. Finish.
  3. Create a home directory for your unix tools where you like it. Create an environment variable called HOME pointing to this directory.
  4. Add to your PATH environment variable the directory C:/tools/Git/bin.
  5. Into the directory pointed to by HOME, put a .bashrc with some aliases.
  6. Make sure that your .bashrc contains this line:
    PATH=/bin:$PATH
    This makes sure that the bash tools find and sort are in the PATH before the windows tools with the same names.
  7. Set the Double Commander terminal command to C:/tools/Git/bin/bash.exe.
  8. Done and works. Now you’ve got a usable bash with pipes and filters, awk, find, grep etc available.

ConEmu

ConEmu is a very fine terminal emulator. Download the ConEmu portable package. It’s got dozens of settings and it can be used for any consoles on windows, not only the bash. E.g. it can be used with the ordinary cmd, too.

Here is how to install it:

  1. Download the portable version of ConEmu.
  2. Extract it beneath the Git directory, to C:/tools/ConEmu.
  3. Run ConEmu.exe or ConEmu64.exe, depending on your system. Delete the other one.
  4. Select as settings location C:/tools/ConEmu/ConEmu.xml
  5. As startup task select {Bash::Git bash}.
  6. Done. Now you’ve got a nice looking console window with your git-bash inside. And for the occasional case you still need a cmd console, you can start this also inside ConEmu.
  7. And the best, you have all these things in the console you’ve missed your whole life:
    • You can copy text to the clipboard by just marking it.
    • You can insert text by typing Ctrl-v. Yes, really. No need to Alt-Space-Edit-Paste.
    • You can resize even the width of the window by dragging at the corner. Shocking.
    • ConEmu remembers its position and size and reopens at the previous place
    • There are dozens of other settings and possibilities for configuration with which you can play if you’d like to.
  8. Set the Double Commander terminal command to:
    C:/tools/ConEmu64.exe -here

000005

I’ve got the idea of combining ConEmu with Git on Windows from this post.

Things I like to have in my .bashrc:

PATH=/bin:$PATH

alias cd='pushd'
alias -- -='popd'

alias ..='cd ..'
alias ...='cd ../..'
alias ....='cd ../../..'

alias ls='ls -ACF'
alias ll='ls -l'

alias no='notepad++'

How to Change the Encoding of Filenames, Directory Names and File Content From ISO 8859-1 to UTF-8

8859-1 Encoded Filenames, Directory Names And Content

I have got a lot of Perl-scripts, and text data files on an older Linux system. All this stuff together makes a management system which I still need. Now I want to bring this management system to a current Linux system, because I suspect that the old one maybe will stop to work some day.

The oldish Linux system uses 8859-1 as encoding for filenames, directory names and content of text files.

The current Linux uses UTF-8 as encoding for filenames, directory names and content of text files.

Of course, as the system is in German, many files, filenames and directory names contain Umlauts like ÄÖÜäöü.

To work with the system on the current Linux, all must be converted from ISO 8859-1 encoding into UTF-8 encoding, including file- and directory names.

Side Note: Bash Glob Problem

Otherwise, such simple things as a bash glob do not work any more. Example: I have two files Maus and Möhre on the old 8859-1 system in a directory.

Now I bring these files to the new system and try to list them with ls.

> ls
Maus M?hre
>
> ls M*
Maus

Unbelievable. But true.

Use Iconv to Change Encoding

The iconv tool is available on most Linux systems, including my new one. With iconv, you can convert text from any encoding known in the world to any other.

As I have to change the encoding not only of the file contents but also of the filenames and directories, I have written a bash script to do all the transformation.

The script creates a backup .bak.8859-1 of every file it transcodes and it creates an empty .eonc file (eonc for encoding of name changed) for every file or directory as a marker that its name has been changed. We need to set markers for the already transcoded names, otherwise a second run of the script would damage things.

The script transcodes all filenames, directory names, file content in and below the current directory from ISO 8859-1 to UTF-8. Don’t run it in / 😉

Shell-Programmierung: Das umfassende Handbuch (Galileo Computing)

#! /bin/bash

eoncExtension=.eonc


function iconvContent {
  local tmpf=tmp.`basename $1`
  if [[ -s $1 && ! -e $1.bak.8859-1 ]] ; then 
    echo "iconvContent $PWD $1"
    cp -p $1 $1.bak.8859-1
    iconv -f ISO_8859-1 -t UTF8 $1 -o $tmpf

    # Iconv does no preserve file attributes. 
    # chmod --reference  ... does not work on my Linux version. 
    cat $tmpf > $1      # this preserves the file attributes of $1
    rm $tmpf
  fi 
}


# Change encoding of a file- or directory name.
# But only if the encoding has not already been changed.     
function iconvName {
  if [[ -e $1 && ! -f $1$eoncExtension ]] ; then 
    local target=$(echo $1 |  iconv -f ISO_8859-1 -t UTF8)
    if [[ $1 != $target ]] ; then
      echo "iconvName $PWD $1  --> $target"
      mv $1 $target

      # Create a .eonc file as a marker that the encoding is 
      # already changed.
      touch $target$eoncExtension      
    fi
  fi
}              

# Check if the file is not one of those that I've created myself 
# as backup or marker or temp file.
function isSelfCreatedFile {
  local bak='.*\.bak.*'
  local eonc='.*\.eonc'
  local tmpre=".*tmp\..*"
  local f=$1
  local self=1
  if [[ $f =~ $bak || $f =~ $eonc || $f =~ $tmpre ]] ; then   
    self=0
  fi
  return $self
}

function iconvNamesInDir {
  for f in `find . -maxdepth 1`  ; do
    if [ -e $f ] ; then 
      if ! isSelfCreatedFile $f ; then   
        iconvName $f
      fi
    fi
  done
}

function iconvContentInDir {
  for f in `find . -maxdepth 1 -type f`  ; do
    if [  -s $f ] ; then   
      if ! isSelfCreatedFile $f ; then 
        iconvContent $f
      fi
    fi
  done
}

function iconvCurrentDir {
  echo "iconvCurrentDir  $PWD"

  iconvNamesInDir
  iconvContentInDir  


  for f in `find . -maxdepth 1 -type d`  ; do
    if [[ ! $f == . ]] ; then 
      local dir=$PWD
      cd $f
      iconvCurrentDir
      cd $dir
    fi
  done
}


if [ "$1" == "run" ] ;  then 
  iconvCurrentDir
else 
  echo "USAGE: iconv-multi run"
  echo "  This script changes the encoding of all filenames, "
  echo "  directory names and all file content in and below the "
  echo "  current directory from ISO 8859-1 to UTF-8."
  echo "  Running it twice does not hurt because the script "
  echo "  creates backup and marker files to detect which "
  echo "  files already have been encoded."
  echo "  (c) 2014 by Andreas Wicker."
fi

Notepad++: How to Make the Function List work with Tcl and Bash

In another post about Notepad++ I critisized that its function list feature does not work for Tcl and Bash scripts. Now I’ve got a solution. Here is it.

How to Make the Function List Work for Tcl

Open functionList.xml in an editor.  FunctionList.xml is in %APPDATA%\notepad++ or in the installation directory of Notepad++.
Add this line to the section with all the other association-entries.

 <association langID="29" id="tcl_procedure"/>

Add this to the section <parsers>

<parser id="tcl_procedure" displayName="Tcl source" commentExpr="(#)">
    <function
            mainExpr="^[\t ]*((proc)[\s]+)[^\n]+\{"
            displayMode="$functionName">
        <functionName>
            <nameExpr expr="[\w: ]+ \{.*\}"/>
        </functionName>
    </function>
</parser>

Restart Notepad++. Credits for the solution go to Detlef of compgroups.net/comp.lang.tcl. I’ve copied his’ and shortened it a bit. And added support for bash.

How to Make the Function List Work for Bash

In functionList.xml add this line to the section with all the other association-entries.

 <association langID="26" id="bash_function"/>

Add this to the section <parsers>

<parser id="bash_function" displayName="Bash" commentExpr="(#)">
    <function
            mainExpr="^[\t ]*((function)[\s]+)[^\n]+"
            displayMode="$functionName">
        <functionName>
            <nameExpr expr="[\w: ]+"/>
        </functionName>
    </function>
</parser>

Restart Notepad++.

Download

Or just download my functionList.xml with Tcl and Bash support and replace yours.

Effektiv Tcl/Tk programmieren . (Programmer’s Choice)

Update: As Olivier from comp.lang.tcl pointed out, my version does not work with Npp 6.4.3. I’ve rechecked this, he is right. My version does work at least with versions 6.4.5, 6.5 and 6.5.5, though.

Update 2: Thanks to Rasha Matt Blais for his improved version of the mainExpr for bash functions. Though that one still misses some types of function definition. Here my latest mainExpr string:

mainExpr="^[\t ]*((function)[\s]+)[^\n]+|[\w_]+[\s]*(\([\s]*\))"

I’ve also adapted the downloadable functionList.xml. And I’ve added the testfunc.bsh file which contains some bash functions with which I’ve tested the my mainExpr string.

Cygwin: Linux Commands on Windows

Maybe you – like me – have learned to know and love the linux bash with its pipes and filters and its standard commands like sed, grep, find, awk, alias, xargs and so on. Maybe you have to do anything with the Windows batch or Windows command shell now. Then you will not only feel like in the stone age, but you’ll feel personally insulted by how ugly, impractical and unusable the Windows tools are.

02-13 15_44_44-Cygwin Setup

You’ll long for your precious bash and your beloved grep. You are not alone…. That’s why since 1995 people are working on Cygwin. Cygwin is a bash with a huge lot of commands, usable on Windows.   I have been using it now for more than ten years, and there have not been too many working days when I haven’t used it for this or that. I only can recommend it warmly for bash addicts coming to Windows.

To start using it, just download the latest Cygwin installer and run it. You’ll be firing your first find | xargs | grep in a quarter of an hour. It’s that easy.

Though it has some quirks, it can be used quite well as your personal shell on Windows PCs. (I would not recommend to use it on a build server or as part of the software build process. Because of the quirks and because it would be an administration nightmare.)

When it works, it works well. I have not updated one single package of the Cygwin installation on my working PC since 2009…. This week, I had to move the Cygwin to my new working PC. But this is another story.

I’d love to hear from you: Which shell do you use on Windows? Why?

 

What are the quirks, you are asking?

In short: line endings.

Unix line endings are \n, Windows line endings are \r\n. This leads to a whole lot of problems which have never been solved well by the Cygwin developers. And I assume these problems cannot and will not ever been solved well.
Because of the different line endings you have such strange things as being able to mount a directory in textmode or in binmode.

Some Cygwin commands do different stuff, depending on the mount mode of the stuff. When the stuff comes from a binmode mounted directory, it is treated differently than when it comes from a textmode mounted directory.  But not all Cygwin commands do the same. Some treat stuff like this and some like that. And of course, it also depends on the version of the tool. And, as some commands create temp files, your result may also depend on the mount mode of your temp directory.

It may be that cat foo | grep bar  results in different line-endings than grep bar foo.  And that grep bar foo | awk // gives a third type of line-ending. There even have been versions of Cygwin where the commands also differentiated between different path notations.

All of this binmode/textmode/line-endings stuff is just !$%*!$%*!$.

And you’ll end up with files having \n as line-endings or even \r\n\n or probably anything that matches [\r\n]+ where they should just always have \r\n. This will happen, when you are using Cygwin in a regular way. Mostly, it is just a call of u2d (unix2dos) or some similar command and the file is correct again.