Tag Archives: Linux

Bash, Awk on Windows: Git-Bash and ConEmu

Posted on by

Every developer who has worked for some time on Linux or another Un*x system will miss the bash, some of the other unix tools like find, cat, grep and awk and a decent console bitterly.

There are several ways how to get some of the unix power back to windows.

  • Cygwin. It is sometimes a bit complicated to install and handle and there are often problems with line endings. I have used this but never was really totally content.
  • MinGW, MSYS and MSYS2. I have never used them and cannot tell much about them.
  • Git-Bash and ConEmu. Lately I have detetcted these tools and it seems they are working quite well. In this post, I’ll go a bit deeper on them. Git-Bash is a version of mingw32, as far as I know.

Git Bash

Git is a well known source control system. And its installation kit for windows contains bash, awk, grep, find and several other unix tools. So even if you don’t want to use git on your windows machine, you can install this package. It is free software.

Installation (with Git 1.9.5 from 2015/03/19) goes like this:

  1. Download the git package for windows.
  2. Run the installer.
    • Install it to tools/Git. Next.
    • Leave away Windows Explorer integration. Do not associate file endings. Next.
    • Do not create a start menu folder. Next.
    • Use Git from Git Bash only. (Other options didn’t work as intended anyway.)
    • Next. Check out Windows-style, commit Unix-style line endings.
    • Next. Wait. Finish.
  3. Create a home directory for your unix tools where you like it. Create an environment variable called HOME pointing to this directory.
  4. Add to your PATH environment variable the directory C:/tools/Git/bin.
  5. Into the directory pointed to by HOME, put a .bashrc with some aliases.
  6. Make sure that your .bashrc contains this line:
    PATH=/bin:$PATH
    This makes sure that the bash tools find and sort are in the PATH before the windows tools with the same names.
  7. Set the Double Commander terminal command to C:/tools/Git/bin/bash.exe.
  8. Done and works. Now you’ve got a usable bash with pipes and filters, awk, find, grep etc available.

ConEmu

ConEmu is a very fine terminal emulator. Download the ConEmu portable package. It’s got dozens of settings and it can be used for any consoles on windows, not only the bash. E.g. it can be used with the ordinary cmd, too.

Here is how to install it:

  1. Download the portable version of ConEmu.
  2. Extract it beneath the Git directory, to C:/tools/ConEmu.
  3. Run ConEmu.exe or ConEmu64.exe, depending on your system. Delete the other one.
  4. Select as settings location C:/tools/ConEmu/ConEmu.xml
  5. As startup task select {Bash::Git bash}.
  6. Done. Now you’ve got a nice looking console window with your git-bash inside. And for the occasional case you still need a cmd console, you can start this also inside ConEmu.
  7. And the best, you have all these things in the console you’ve missed your whole life:
    • You can copy text to the clipboard by just marking it.
    • You can insert text by typing Ctrl-v. Yes, really. No need to Alt-Space-Edit-Paste.
    • You can resize even the width of the window by dragging at the corner. Shocking.
    • ConEmu remembers its position and size and reopens at the previous place
    • There are dozens of other settings and possibilities for configuration with which you can play if you’d like to.
  8. Set the Double Commander terminal command to:
    C:/tools/ConEmu64.exe -here
>000005

I’ve got the idea of combining ConEmu with Git on Windows from this post.

Things I like to have in my .bashrc:

PATH=/bin:$PATH

alias cd='pushd'
alias -- -='popd'

alias ..='cd ..'
alias ...='cd ../..'
alias ....='cd ../../..'

alias ls='ls -ACF'
alias ll='ls -l'

alias no='notepad++'

Linux Live USB Creator

Posted on by

Sometimes it is useful to be able to start a computer not with its installed OS, but with another OS.

  • to clean up and rescue data after virus infestation.
  • to rescue data after crash of the system hard disk.
  • to test an OS.
  • to repartition hard drives.

So, people have developed so-called Linux Live Systems. Formerly, these have been usable from CD – and they still are – if your computer has an optical drive.

>LinuxLive USB Creator

But many newer PCs and Laptops do not have an optical drive. If you want to start such a computer with a live system, you need to put it onto an USB stick.

For this task, the Linux Live USB Creator (Lili) has been created.

Though it has got a terrible transparent/half-transparent UI, it is really easy to use, its documentation is decent and you can easily create a bootable Linux Live USB from e.g. the Avira Rescue System or one of dozens of other Linux distributions.

For the Avira Rescue System you cannot have persistence, so skip step 3. Also you don’t want to start your Avira Rescue System from Windows, and normally you don’t want to hide your files: Unselect all three checkboxes in step 4.

If you want to start a live system directly from Windows, you can select the corresponding option in Lili’s step 4. Then a special version of VirtualBox will be put onto the stick and you can run your live system inside VirtualBox.

Troubles With Booting the PC From USB Stick

To make your PC willing to boot from the created USB stick, you probably have to change some BIOS settings. You can reach the BIOS settings by pressing one of the keys Del, Esc, F1, F2, F8, F12 during boot. It is different from PC to PC. Many PCs write a short notice onto the screen during boot which key to press. But also some don’t and with some the text disappears too fast to read it so you have to guess.

If you’ve got a PC with UEFI BIOS, you probably have to enable the Compatibility Support Module (CSM) and/or disable Secure Boot.
You still might get problems: PCs with UEFI BIOS often use the GPT partitioning scheme for their hard drive and many live systems cannot cope with the GPT partitioning scheme.

How to Change the Encoding of Filenames, Directory Names and File Content From ISO 8859-1 to UTF-8

Posted on by

8859-1 Encoded Filenames, Directory Names And Content

I have got a lot of Perl-scripts, and text data files on an older Linux system. All this stuff together makes a management system which I still need. Now I want to bring this management system to a current Linux system, because I suspect that the old one maybe will stop to work some day.

The oldish Linux system uses 8859-1 as encoding for filenames, directory names and content of text files.

The current Linux uses UTF-8 as encoding for filenames, directory names and content of text files.

Of course, as the system is in German, many files, filenames and directory names contain Umlauts like ÄÖÜäöü.

To work with the system on the current Linux, all must be converted from ISO 8859-1 encoding into UTF-8 encoding, including file- and directory names.

Side Note: Bash Glob Problem

Otherwise, such simple things as a bash glob do not work any more. Example: I have two files Maus and Möhre on the old 8859-1 system in a directory.

Now I bring these files to the new system and try to list them with ls.

> ls
Maus M?hre
>
> ls M*
Maus

Unbelievable. But true.

Use Iconv to Change Encoding

The iconv tool is available on most Linux systems, including my new one. With iconv, you can convert text from any encoding known in the world to any other.

As I have to change the encoding not only of the file contents but also of the filenames and directories, I have written a bash script to do all the transformation.

The script creates a backup .bak.8859-1 of every file it transcodes and it creates an empty .eonc file (eonc for encoding of name changed) for every file or directory as a marker that its name has been changed. We need to set markers for the already transcoded names, otherwise a second run of the script would damage things.

The script transcodes all filenames, directory names, file content in and below the current directory from ISO 8859-1 to UTF-8. Don’t run it in / 😉

#! /bin/bash

eoncExtension=.eonc

function iconvContent {
  local tmpf=tmp.`basename $1`
  if [[ -s $1 && ! -e $1.bak.8859-1 ]] ; then 
    echo "iconvContent $PWD $1"
    cp -p $1 $1.bak.8859-1
    iconv -f ISO_8859-1 -t UTF8 $1 -o $tmpf

    # Iconv does no preserve file attributes. 
    # chmod --reference  ... does not work on my Linux version. 
    cat $tmpf > $1      # this preserves the file attributes of $1
    rm $tmpf
  fi 
}

# Change encoding of a file- or directory name.
# But only if the encoding has not already been changed.     
function iconvName {
  if [[ -e $1 && ! -f $1$eoncExtension ]] ; then 
    local target=$(echo $1 |  iconv -f ISO_8859-1 -t UTF8)
    if [[ $1 != $target ]] ; then
      echo "iconvName $PWD $1  --> $target"
      mv $1 $target

      # Create a .eonc file as a marker that the encoding is 
      # already changed.
      touch $target$eoncExtension      
    fi
  fi
}              

# Check if the file is not one of those that I've created myself 
# as backup or marker or temp file.
function isSelfCreatedFile {
  local bak='.*\.bak.*'
  local eonc='.*\.eonc'
  local tmpre=".*tmp\..*"
  local f=$1
  local self=1
  if [[ $f =~ $bak || $f =~ $eonc || $f =~ $tmpre ]] ; then   
    self=0
  fi
  return $self
}

function iconvNamesInDir {
  for f in `find . -maxdepth 1`  ; do
    if [ -e $f ] ; then 
      if ! isSelfCreatedFile $f ; then   
        iconvName $f
      fi
    fi
  done
}

function iconvContentInDir {
  for f in `find . -maxdepth 1 -type f`  ; do
    if [  -s $f ] ; then   
      if ! isSelfCreatedFile $f ; then 
        iconvContent $f
      fi
    fi
  done
}

function iconvCurrentDir {
  echo "iconvCurrentDir  $PWD"

  iconvNamesInDir
  iconvContentInDir  

  for f in `find . -maxdepth 1 -type d`  ; do
    if [[ ! $f == . ]] ; then 
      local dir=$PWD
      cd $f
      iconvCurrentDir
      cd $dir
    fi
  done
}

if [ "$1" == "run" ] ;  then 
  iconvCurrentDir
else 
  echo "USAGE: iconv-multi run"
  echo "  This script changes the encoding of all filenames, "
  echo "  directory names and all file content in and below the "
  echo "  current directory from ISO 8859-1 to UTF-8."
  echo "  Running it twice does not hurt because the script "
  echo "  creates backup and marker files to detect which "
  echo "  files already have been encoded."
  echo "  (c) 2014 by Andreas Wicker."
fi

How do You Move Cygwin?

Posted on by

Last week, I had to move Cygwin from my old to my new working PC.

I can tell you, starting new with Cygwin is a completely different thing than copying or moving a working installation to another PC after four years of not updating it. I spent one and a half days on the task and it still does not work exactly like it did before … but I can use it productively. And it was worth the trouble. 😉

What follows is my personal moving cygwin leaflet.

How do I copy or move Cygwin from PC 1 to PC 2?

As said, it was not easy and I had to follw many blind alleys and got lost in the mazes of the Cygwin forums a lot ….. I’m leaving away some of the blind alleys and all of the system reboots in the following description.

  1. Zip the cygwin directory on PC 1. On my PC 1, it is located under C:\cygwin. It contains cygwin’s setup.exe and the packages directory and the bin, usr, etc, lib … directories.  I zipped it to c-cygwin.7z.
  2. Zip the user’s home directory on PC 1. My cygwin username on PC 1 was ‘wic’, and the home was located under D:\wic. I zipped it to d-wic.7z.
  3. Unzip the c-cygwin.7z on PC 2 to C:\cygwin.
  4. Unzip the d-wic.7z, ideally to D:\wic. But my PC 2 has no D:\ drive and I don’t want to repartition it now. So I extract it to C:\wic.
  5. Run c:\cygwin\setup.exe. Select “Install from local directory”. Select “c:\cygwin” as root directory. Select C:\cygwin\packages as local package directory. Click Next, Next, … until setup.exe ends.
  6. Set the Windows environment variable HOME for your user to /cygdrive/c/wic. Literally, as cygwin like path.
  7. Now I can doubleclick my C:\cygwin\cygwin.bat file and it tries to run my olde .profile and so on…..But, I get some strange errors like

    bash: $’\r’: command not found

  8. I’m assuming my .profile has Windows- line-endings but the drive is mounted in binmode. I call ./mount and get

    c: on /cygdrive/c type system(binmode,noumount)
    d: on /cygdrive/d type system (binmode,noumount)

    I call ./od -c ~/.profile  and see a lot of \r\n. So my assumption seems plausible.

  9. I’m trying to remount in textmode: mount -t c:\cygwin /cygdrive/c but I’m getting this error: mount: /cygdrive/c: Device or resource busy.
  10. Somewhere in the net they say if this error occurs, you should unmount first. I’m trying umount -A, and there is no error message. But still, the binmode-mounts stay. I cannot not remount c: as textmode.
    During all the reading in the forums, I find out that the cygwin maintainers changed the treatment of line-endings totally around 2011. I think there’s no way to resist these changes….. After four years, finally I have to update my cygwin.
  11. I download and install the newest version of cygwin. At least the PATH variable is now set correctly, I can call cygwin commands now from every folder as long as I’m inside the bash.
  12. I’m mounting mkdir /c; mount C: /c   Seems to work. Still the mounting of my home directory is wrong.
  13. I’m moving c:\wic to c:\cygwin\home\wic and setting the HOME environment variable to /home/wic. Still, it cannot interpret the windows line endings correctly.
  14. I’m calling d2u (dos2unix) for the most important files:
    cd /c/cygwin/home/wic
    d2u .*   # this works, d2u ignores binary files and directories
    d2u *
    cd bin
    d2u .*
    d2u *
  15. cp /bin/bash.exe /usr/local/bin/bash.exe  # because some of my bash scripts use #!/usr/local/bin/bash
  16. Copy scripts from c-cygwin.7z/usr/local/bin to /usr/local/bin, running d2u * in /usr/local/bin.
  17. When I close the cygwin window and start it again, the mount of c: -> /c is away. Maybe because of lacking rights? I try to start cygwin with admin rights and do the mount again. No, it’s not a rights problem, but mount does only create temporary mounts. If you want to have persistent mounts, you have to change the /etc/fstab file.
  18. Also, I want to replace the /cygwin prefix by /, so that cd c:/ can be written as cd /c.  In the /etc/fstab, replace this line
    none /cygdrive cygdrive binary,posix=0,user 0 0
    this:
    # getting rid of /cygdrive and make all new mounts text mounts.
    none / cygdrive text 0 0
  19. Hooray hooray. Now, I can use all of my scripts, my aliases and all of the un*x commands that come with cygwin. Only some fine tuning is lacking.
  20. Create a shortcut on the screen to Cygwin.bat.
  21. Select the little tree as icon for the shortcut. The tree is my bash symbol for ages. (pic)
  22. Double click it. Right-click on the window header. Select Properties. Choose these properties:
    Quick-Edit-Mode: on
    Insert-Mode: on
    Font:7×12
    Window buffer size: w x h 120 x 3000, window size: 120 x 43
    Window background color: 244,244,188
    Window text color: 0,10,10

cygwin

DONE.

WinDirStat: Find Harddisk Memory Monsters

Posted on by

Wicker’s Lemma to Moore’s law:
Every hard disk gets full, independent of its size.

Astonishing, but true. If When your hard disk gets full, you can do two things:

  • Clean it up
  • Buy a new one

Of course, when your harddisk gets full late saturday night and you need some more space now, buying a new one is not an option. Also, if the harddisk in your laptop gets full, buying a new one will come with a lot of work: Transferring all the data, programs and the OS to the new harddisk. This is not something you really want to do! So often cleaning up is the only option.

How do you clean it up? First, empty the trashcan. This may already be enough. If it isn’t, download WinDirStat for Windows (or KDirStat for Linux or Disc Inventory X for Mac), install and start it. It will scan your harddrives and create a nice looking view where you can see exactly what consumes the most space on your drive. Every file is represented by a rectangle and the size of the rectangle is proportional to the size of the file.

It is indeed enjoying to use this tool for hunting and killing those evil memory monsters. It may be more enjoying than killing pigs with birds 😉 Happy hunting!

02-15 13-WinDirStat

WinDirStat is free software, GPL’ed.