In the first part, we setup the project structure, wrote the java wrapper class, generated the c header, wrote the c library and write the Android.mk file.

Building shared library

With traditional JNI (not targetting Android devices), you just compile the library with gcc using the -shared option. However, for Android devices, we use the ndk-build script.

To compile, we first switch to the project directory (the directory that contains the jni and src directories then run the ndk-build script provided in the Android NDK:

cd Android-JNI-Demo
/path/to/ndk-build # for example: ~/build/android-ndk/android-ndk-r7c/ndk-build

If when trying to compile on linux you get the following error:

Invalid attribute name:
    package

You’ll want to check the line endings of your AndroidManifest.xml. I used the dos2unix command to correct them.

If the build script found your Android.mk and the library compiled without issue, you’ll see the following:

Compile thumb  : squared <= squared.c
SharedLibrary  : libsquared.so
Install        : libsquared.so => libs/armeabi/libsquared.so

So we’re happy with the library compilation and now we’ll move on the developing a simple UI to test whether our functions perform as we expect.

Develop Simple UI

By default, when we created a new Android project in Eclipse an activity was generated with the following:

package org.edwards_research.demo.jni;

import android.app.Activity;
import android.os.Bundle;

public class Android_JNI_DemoActivity extends Activity {
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
    }
}

If we didn’t care about looking pretty, we could change this to something like:

package org.edwards_research.demo.jni;

import android.app.Activity;
import android.os.Bundle;
import android.util.Log;

public class Android_JNI_DemoActivity extends Activity {
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        int b = 3;
        int a = SquaredWrapper.to4(b);
        Log.i("JNIDemo", String.format("%d->%d", b,a));
    }
}

and run it, either in the emulator or on a device (with USB debugging enabled), we would see in LogCat an entry tagged with JNIDemo. In this case, we’d expect something like 3->81, since 3^4 = 81. But we’ll do a little bit more to see the performance of the library directly on the UI.

Instead of walking through the specific steps of creating the UI, I’ll simply post the pertinent files:

Android_JNI_DemoActivity.java

package org.edwards_research.demo.jni;

import android.app.Activity;
import android.os.Bundle;
import android.view.View;
import android.widget.EditText;
import android.widget.TextView;

public class Android_JNI_DemoActivity extends Activity {
    private EditText etInput;
    private TextView txtTo2;
    private TextView txtTo4;

    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        // Define Input EditText, TextViews
        etInput = (EditText) findViewById(R.id.etInput);
        txtTo2 =  (TextView) findViewById(R.id.resTo2);
        txtTo4 =  (TextView) findViewById(R.id.resTo4);
    }

    public void cbCalculate(View view)
    {
        int in = 0;
        try{
            in = Integer.valueOf( etInput.getText().toString() );
        } catch(NumberFormatException e) { return ; }

        txtTo2.setText(String.format("%d", SquaredWrapper.squared(in)));
        txtTo4.setText(String.format("%d", SquaredWrapper.to4(in)));
    }
}

res/layout/main.xml

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:orientation="vertical" >

    <TableLayout
        android:id="@+id/tableLayout1"
        android:layout_width="match_parent"
        android:layout_height="wrap_content" >

        <TableRow
            android:id="@+id/tableRow1"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content" >

            <EditText
                android:id="@+id/etInput"
                android:layout_width="150dp"
                android:layout_height="wrap_content"
                android:inputType="number" />

            <Button
                android:id="@+id/button1"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="Calculate"
                android:onClick="cbCalculate"/>

        </TableRow>

        <TableRow
            android:id="@+id/tableRow2"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content" >

            <TextView
                android:id="@+id/lblTo2"
                android:layout_width="match_parent"
                android:layout_height="wrap_content"
                android:text="@string/squared" />

            <TextView
                android:id="@+id/resTo2"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="" />

        </TableRow>

        <TableRow
            android:id="@+id/tableRow3"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content" >

            <TextView
                android:id="@+id/lblTo4"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="@string/to4" />

            <TextView
                android:id="@+id/resTo4"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="" />

        </TableRow>

        <TableRow
            android:id="@+id/tableRow4"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content" >
        </TableRow>
    </TableLayout>

</LinearLayout>

res/values/strings.xml

<?xml version="1.0" encoding="utf-8"?>
<resources>
    <string name="app_name">Android-JNI-Demo</string>
    <string name="squared">Squared:</string>
    <string name="to4">To 4:</string>
</resources>

When run in the emulator the app looks as follows:

You can enter a number in the text field and press the calculate button which will result in the squared and ^4 calculations:

Of course this was a pretty silly demo library because a squared function could have been trivially implemented in java without the need for c code, cross compiling or dealing at all with the Java Native Interface, however it still illustrated the steps necessary to compile a native library against the Android NDK and how to import and use it in an Android Project.

Prerequisites

As a prerequisite for this tutorial, you’ll need:

1. Eclipse installed and configured to create Android projects. There are a number of tutorials out there about how to do this if you need help.
2. A JDK installed, as I don’t believe the standard JRE contains the javah command that will be needed.
3. The Android NDK (available here) downloaded and extracted somewhere.

Create the Android Project

For this tutorial we’re going to create a new project. However there is no reason you couldn’t integrate this into an already-created project.

To create the project, right click in Eclipse’s Package Explorer → New → Android Project. Give your project a name and select an API. For this tutorial I choose the latest Gingerbread API, 2.3.3.

Add jni folder, Android.mk makefile

Once your project has been created, you’ll need to create a new folder inside the top level of the project. To do this right click on your project name → New → Folder. Name this folder jni.

Inside this folder, create a new blank text file. To do this right click on your newly-created jni folder → New → File. Name this file Android.mk. Leave this file blank for now, we’ll come back to it later.

Your project should look something like this:

Create java source

For this tutorial, we’re going to have a simple c program — squared — that accepts an int and returns the square (e.g. 2 → 4, 3 → 9, etc.)

In order to accomodate that, we first create a java source wrapper. The wrapper’s job is to load the library, expose any native functions we wish to use directly, and provide any functions that we want to be able to utilize private native functions.

For this tutorial, we’re going to expose directly the native squared function as well as provide a to4 “derivative” function.

To expose the native function directly, we just declare it public. Alternatively, we could declare it private and limit it’s availability to other functions of the class.

Our full java source is

package org.edwards_research.demo.jni;

public class SquaredWrapper {
    // Declare native method (and make it public to expose it directly)
    public static native int squared(int base);

    // Provide additional functionality, that "extends" the native method
    public static int to4(int base)
    {
        int sq = squared(base);
        return squared(sq);
    }

    // Load library
    static {
        System.loadLibrary("squared");
    }
}

Create C header

After we outline the native methods we’ll be using, we can use this java source to create a c header file with the function prototypes for the native methods we used. To do this, we first have to compile the java source into a class file. You can do this manually via the javac command, e.g.:

cd src # change into the source directory
javac -d /tmp/ org/edwards_research/demo/jni/SquaredWrapper.java

Note that the -d switch specifies the output directory for the class file — in this case, I’m just throwing it into /tmp.

Now that we have the class, we can create the c header file., e.g.:

cd /tmp
javah -jni org.edwards_research.demo.jni.SquaredWrapper

Note the need to specify the fullly-qualified class name (including package) and not the .class file extension.

The resulting header file in our case is /tmp/org_edwards_research_demo_jni_SquaredWrapper.h, but we can rename it to whatever we want. In this case, we’ll rename it to squared.h and place it in the jni folder in our project directory.

The resulting squared.h file looks like:

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class org_edwards_research_demo_jni_SquaredWrapper */

#ifndef _Included_org_edwards_research_demo_jni_SquaredWrapper
#define _Included_org_edwards_research_demo_jni_SquaredWrapper
#ifdef __cplusplus
extern "C" {
#endif
/*
 * Class:     org_edwards_research_demo_jni_SquaredWrapper
 * Method:    squared
 * Signature: (I)I
 */
JNIEXPORT jint JNICALL Java_org_edwards_1research_demo_jni_SquaredWrapper_squared
  (JNIEnv *, jclass, jint);

#ifdef __cplusplus
}
#endif
#endif

The function name is annoying long, I agree. We could eliminate a lot of that if we didn’t use a package in our java source, but I’m not really that concerned.

Create C source

Using the prototype generated by javah, we can implement our c source as follows:

#include "squared.h"

JNIEXPORT jint JNICALL Java_org_edwards_1research_demo_jni_SquaredWrapper_squared
  (JNIEnv * je, jclass jc, jint base)
{
        return (base*base);
}

Note we have to give the parameters names and I arbitrarily chose je andjc, and chose base to replicate our java source parameter name.

In this case, the c source is very simple, but this tutorial is meant to illustrate how to include native code into your Android app and more complex c functions could be substituted with few modifications.

Create Android.mk

After we create our c source file, we have to create our Android.mk file. This file serves the as a sort of makefile for the Android build tools. There are a number of sample Android.mk files in the samples/ directory of the NDK and we’ll actually be using almost the exact lines from the hello-jni sample project.

Our Android.mk file looks like:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE    := squared
LOCAL_SRC_FILES := squared.c

include $(BUILD_SHARED_LIBRARY)

At this point, we have laid most of our groundwork for setting up and compiling the library. The next steps are actually creating the shared library, and implementing some simple UI code to show that our native function (squared) and derivative function (to4) work as expected.

This is continued in Part 2.

To some, this might be obvious, they might suggest holding down the left arrow until you reach the beginning of the line and “commenting it out” by inserting a # in front of the text.

But there are actually two quicker solutions, one somewhat obvious, and the other not so much. The reason I’m writing this is just to point out the second one.

The first quick solution would be just to press the “Home” key and then insert the # symbol and press enter. This is the pretty obvious method.

The second method, which is actually a keystroke shorter, is to press Escape then the # symbol. These two keys have the same result as the three keys in the first method.

Saving a single keystroke is really not the motivation for posting this tip, but rather the fact that some terminals, especially when first configuring your client / initialization scripts don’t accept the home key correctly (or quite often your client isn’t sending it correctly).

Anyway, in that situation, its helpful to have a backup option and the “Escape #” method comes in very handy.

The Adobe plugin creates a secure temporary file — the typical way to do this is to (a) create a file, (b) open a file descriptor to it, (c) unlink (delete) the file. As long as the file descriptor remains open, the file will be fine. But once the file descriptor is closed, the bytes are free to be overwritten by the filesystem. The last step of unlinking is in an attempt to limit other programs from accessing the file.

In our case, flash’s cache file is located in /tmp and named something like “FlashXXXXXXXXX”, for example: /tmp/FlashXX9Vj3jC

However, if you go into /tmp and run ls, it won’t be there — because it’s been deleted.

You can verify this by opening firefox, going to a site with a flash video, and running something like the following:

ls -l /proc/$(pidof plugin-container)/fd/ | grep Flash

For example, I did this and got:

l-wx------. 1 user user 64 Mar 20 22:41 17 -> /tmp/FlashXX9Vj3jC (deleted)
lrwx------. 1 user user 64 Mar 20 22:41 23 -> /tmp/FlashXXSUb8Wv (deleted)

Here we’re just looking at the plugin-container’s open file descriptors with “Flash” in them.

While you can’t access the cached files via /tmp/FlashXXXXXXXX, you can access them via the process file descriptor. For example, in this case the file descriptors 17 and 23 correspond to the two cached files.

To copy the first one, you do something like:

cp /proc/$(pidof plugin-container)/fd/17 flashsave.flv

To make all of this a bit easier, I wrote the following python script:

#!/bin/env python

import subprocess
import sys
import os
import shutil
from datetime import datetime

debug = False
outdir = '/tmp/flashsave'

# Great size_fmt function
# Source: http://stackoverflow.com/questions/1094841/reusable-library-to-get-human-readable-version-of-file-size
def sizeof_fmt(num):
    for x in ['bytes','KB','MB','GB']:
        if num < 1024.0:
            return "%3.1f%s" % (num, x)
        num /= 1024.0
    return "%3.1f%s" % (num, 'TB')

# Find the pid of libflashplayer via plugin-container
ph = subprocess.Popen(['pidof', 'plugin-container'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
[out,err] = ph.communicate()

# Ensure out is an int
if len(out) == 0:
    print('Process not found.')
    sys.exit(1)

pid = int(out)
print("Process found.  PID: %d" % pid)

# Find open file descriptors
flash_fds = list()
filenames = os.listdir('/proc/%d/fd/' % (pid))
for f in filenames:
    fd = int(f)
    link_name = os.readlink('/proc/%d/fd/%d' % (pid, fd))

    if(debug): print("  %3d %-20s" % (fd, link_name))

    if(link_name[0:10] == "/tmp/Flash"): flash_fds.append(fd)

# Display flash file descriptors
i = 0
for fd in flash_fds:
    path = '/proc/%d/fd/%d' % (pid, fd)
    size = os.stat(path).st_size

    print("Flash temp file found: %-20s [%s]" % (path, sizeof_fmt(size)))

    dpath = "%s/%s-%d.flv" % (outdir, datetime.now().strftime("%Y-%m-%d-%H%I%S"), i)
    shutil.copy(path, dpath)
    print("Copeid to: %s" % (dpath))
    # Increment counter to try to keep filenames unique
    i = i+1

It’s really informal and could use some clean up, but it does the trick. A few things to note: (1) you will want to wait until the video has finished loading before running this script. You’ll only save what has been loaded — if you run it earlier, you’ll copy a half-complete file, and (2) you’ll need to either create the folder

/tmp/flashsave

or update the outdir variable. To run, simply save this as flash-save.py and type

./python flash-save.py

or make it executable with chmod and type

./flash-save.py

The dreaded Blue Screen of Death

For a while, I was getting random BSODs and some searching led me to decide to enable a feature called the driver verifier. [More info about Driver Verifier here] My understanding of the driver verifier is that, when a driver acts out of line, even a little bit, even if to an extent that otherwise would be tolerated, Windows throws up a BSOD and lists the offending driver.

In my case, I was getting the Special Pool Detected Memory Corruption BSOD with no specific driver being listed. The number of solutions I found online were almost unbelievable: everything from bad memory to corrupt MBR to corrupt CMOS settings. In fact, none of these would have solved my issue. But a few sites seemed to indicate that this meant that some driver was failing the verifier, but to determine which specific one, I had to examine the dump file that is generated once a BSOD is displayed.

So I loaded into a recovery session, and examined the dump file (C:\Windows\MEMORY.DMP — actually, it was D:\, but thats another story). After analyzing it, WinDbg guessed that my mouse driver was causing the issue (RzSynapse.sys, presumably for the Razer mouse I had connected).

So I disconnect the mouse and restart hoping all would be well, but of course nothing is that easy and I immediately got another blue screen. This time, however, it identified a driverm and a different one. The problem here, however, is that the driver was for a remote management application I use, and not something I could easily disconnect to prevent it from being loaded.

So I know I need to update this driver, but I can’t do so easily because windows BSODs out before it even lets me log in.

So I decide to disable the verifier first, and then update the drivers.

In a command prompt in the recovery session I type

verifier /query

but it says something to the effect of “no drivers are being verified”. But that was simply not true, since the newest BSOD clearly says a driver failed verification.

I guess that the

verify

command doesn’t work properly inside a recovery session, so I go to check the registry. From pages like this I find the registry path, so I fire up regedit, navigate to the where the keys should be and… they’re not there.

Well this complicates the matter. On one hand, the BSOD indicates that driver verification is enabled, but neither the command line “verify” command nor the registry seems to think so.

After thinking about it for a few minutes, I recalled that another page I had read previously discussed loading a registry hive into regedit. I didn’t think much of it at the time, but now it got me thinking that, maybe what regedit was displaying was a registry specific to the recovery session, and the registry that was used to boot windows was somewhere else.

I followed the steps outlined in this second article to load my system registry hive and voila, the driver verifier keys were, of course, present. I promptly delete them, unload the registry hive, and restart the computer. Hoping for the best, but fearing for the worst I walked out of the room to do some other things and when I came back, I was greeted by the glorious login screen — something a little bit earlier I hadn’t thought I’d see again without reformatting.

So, recap — lessons learned:

• In a recovery session, the registry regedit displays isn’t the registry you might expect. You need to load external hives to edit them. See the link above for how.
• Driver Verifier is a decent trick to pin down random, unexplained BSODs, but the hassle it caused me makes me wish I simply made an effort to save more often and never even worried about curing them. Simply throwing users to a BSOD seems harsh, especially when they may not cause BSODs had the verifier not been running. I mean, it would be nice if I could “ignore” it, so I could then continue to boot windows and update the offending drivers.
• There must be some recovery session information stored on non-primary disks, because when I disconnected my non-primary disks in an attempt to rule out hardware issues, the recovery session option was gone and in its place was a message asking me to boot from an install or recovery disk. When I replaced the disk, the recovery session option returned. Also interesting was the fact that, when I rearranged by drives in my BIOS correctly (primary on SATA order 1, secondary on SATA order 2) the recovery session again disappeared. I’m sure this would be easy to look up and verify, but I’ve wasted enough time on this issue — maybe later.

In order to adjust the color scheme, you can go into Edit -> Profile Preferences -> Colors Tab.

However, its kind of hard to tell how the colors will look until they’re displayed on the screen with the black background. The following simple script will generate a text in every font color, in every background color, and in all the styles:

#!/bin/bash

# The syntax for escaped color sequences looks like:
#
#    BLUE="\[\033[0;34m\]"
#
# Lets see what we can find...

NONE="\033[0m"

for COLOR in $(seq 30 40) ; do
    for STYLE in $(seq 0 1) 4 5 7; do
        TAG="\033[${STYLE};${COLOR}m"
        STR="Color:${COLOR}--Style:${STYLE}"

        echo -ne "${TAG}${STR}${NONE}  "
    done
    echo
done

The output looks like:

Tango (default) theme

Linux Console theme

Xterm theme

• Removal of minimize and maximize button in title bar, leaving only the “x” close window button.
• Removal of a task switching bottom panel
• Removal of “Shutdown” / “Restart” from … Shutdown? menu — leaving only suspend
• Forced grouping of windows by application when you’re alt tabbing. Now 3 gnome-terminal sessions + firefox + gedit will show as 3 windows instead of 5 and require you to press down to descend into the individual windows of each group.
• Remove the notification area available for applications and force all the applications that used it (sometimes, very cleverly) to rework their apps. See this long list of “suggested changes” proposed by the Gnome devs.

The list really goes on, but what really bothered me was that I found that the delete key in the file manager no longer worked. To be clear, this isn’t exactly a Gnome3 issue but rather a Nautilus issue, the filemanager for the gnome desktop.

At some point, the developers (at least those that accepted the patch) decided that they would diverge from decades of desktop environment tradition and change the shortcut from “Delete” to “Ctrl + Delete”. The rationale being that “Delete” was too easy to press by accident, and since there was no notification that the files were moved to the trash, it would be transparent to the user that accidentally pressed delete.

The thing is, I think they correctly identified an issue: accidentally pressing delete could result in user confusion when they can’t find their files. But what is amazing, is how they chose to address it — not by popping up a notification but by simply making it harder to accidentally (and intentionally) delete.

Its amazing because every OS I can think of already pops up a notification when you’re “soft-deleting” files. By soft-deleting, I mean “moving to trash”, etc. — something that is completely reversible — and that’s the functionality we’re talking about with the (formerly) Del / (now) Ctrl+Del command.

But instead of popping up a simple notification, they chose the almost inexplicable route of confusing every desktop user who is even relatively comfortable with the limited subset of keyboard commands that “just work”.

Huge changes like this, I argue, require sufficient motivation. In this case, the desire not to have a notification pop up is no where near sufficient. It surprised me that this bug fell through the cracks, but it baffles me that its possible the Nautilus devs might think this is “working as intended” and not a bug. This issue is not settled, as the bug is still technically open, but the way the devs seem to be leaning is that this change is acceptable.

For example:

Proposing WONTFIX. As written before this is not considered a bug but
intentional behavior.

–André Klapper [developer] 2012-02-18 09:22:13 UTC

But, to be honest, its not surprising. The arrogant / selfish way that the gnome desktop has “evolved” in recent history should have prepared me for this. From the outside looking in, it seems its just a bunch of people who think they know better about how the majority of the userbase should want to work — even despite an inordinate number of complaints and an immeasurable amount of time spent by third party developers in attempts to “bring back gnome2″. More on that in the links below.

So in closing, I upgraded to Gnome3, but found myself unhappy as I expected would be the case. In most of my workstations I’ve noticed a general decrease in performance clearly attributable to the gnome-shell. I spent a considerable amount of time getting it to feel better, and it does, but I would really prefer my Gnome2 back.

That being said, there are two main options for Gnome3 users who hate the new UI.

The first is using “fallback mode”. This gets the user extremely close to the Gnome2 look and feel. Its not as polished as Gnome3 is, but it gets the job done.

The second is adding “shell extensions” to gnome to reproduce the functionality that Gnome longer ships with. Specifically, this include a “Places” menu drop down, removing some of the annoying icons and menus that are there by default, restoring minimize and maximize buttons, restoring “shutdown” and “restart”, and (most importantly, imo) restoring the bottom panel task bar.

I’ve taken the latter approach and the best set of extensions I’ve found so far are the Mint Gnome Shell Extensions. For a while, it seemed like Mint was going to back a Gnome2 fork (which was encouraging), but now it looks like they’ve decided to go with Gnome3, but reproduce the missing functionality through extensions. These extensions are available on their github site, and can be easily installed on a fresh F16 install.

I only use a few of these, as well as a few from Gnome’s own https://extensions.gnome.org/.

While I have managed to get back 75-80% of the look and feel, the increased resource issue still a problem and I only expect it to get worse. I use linux on my workstations so that I can get quck calculations at the expense of visual flair. I’d be okay working (and do in fact often work) primarily from the command line. These workstations had all of the visual flair disabled (Compiz, 3d rendering, transparency, etc) and ran like a charm. Unfortunately, it looks as if this will not be the case with Gnome3.

For example, lets consider a simple script, globme.sh

#!/bin/bash

echo "1[${1}]"
echo "@[${@}]"
echo

which is in a directory containing:

globme.sh
notes.txt
phonebook.csv
todo.txt

Let’s see how the script reacts to different inputs. First, lets look at what it does without any wildcards:

$ ./globme.sh Not Wildcards
1[Not]
2[Wildcards]
@[Not Wildcards]

It works as one might expect, the 2nd space-separated argument passed to the script is placed into $2. If you expect a certain value to always be at $2, no matter what, you might be surprised when you call the script with something like:

$ ./globme.sh *.txt red

and find out that $2 is not “red”, but in fact the name of a file in your current directory:

$ ./globme.sh *.txt socket
1[notes.txt]
2[todo.txt]
@[notes.txt todo.txt socket]

But note the fact that, if your wildcard does not match set of files in your current directory, it will be passed to the script as-is:

$ ./globme.sh *.py socket
1[*.py]
2[socket]
@[*.py socket]

Well this is not good. The same file and same input acts two different ways based what directory you’re in. Of course, one solution would be to always remember to enclose your arguments in quotes (single or double):

$ ./globme.sh '*.txt' socket
1[*.txt]
2[socket]
@[*.txt socket]

$ ./globme.sh "*.txt" socket
1[*.txt]
2[socket]
@[*.txt socket]

But what if you didn’t want to require this? What if there were no circumstances under which you wanted the shell to expand wildcards for you? I’ll bring you slowly through the steps you might have taken had you not found this tutorial, but if you’re in a rush, you can skip to the end to my favorite solution.

After some searching, you may have found out that there is a shell option (f) that you can set that will disable this behavior. To do this, simply call set -f, as in:

$ ./globme.sh * *
1[globme.sh]
2[notes.txt]
@[globme.sh notes.txt phonebook.csv todo.txt globme.sh notes.txt phonebook.csv todo.txt]

$ set -f

$ ./globme.sh * *
1[*]
2[*]
@[* *]

So you could always call set -f before every globme.sh, but that gets tedious.

Okay, so then you could alias it by adding the following to your .bashrc / .bash_profile file:

alias globme='set -f; /path/to/globme.sh'

Which is better, since you don’t have to remember to call another program before globme, and even if you remember to call another program, you don’t have to remember the program or syntax.

But now you run into the problem where your shell option persists after your command. So while you might expect that cat * prints the contents of each file in your current directory, you’ll get an error:

$ cat *
cat: *: No such file or directory

This problem is exacerbated by the fact that subshells that spawn from your current shell inherit those options. So shell scripts, etc. that expect and rely a certain behavior will not function correctly.

Okay, so one solution would be to remember to clear the shell option after each time using set +f. This would work, but it’s terribly annoying to have to remember.

If you’re wondering whether you can modify your alias to also clear the option, perhaps like:

alias globme='set -f; /path/to/globme.sh; set+f;'

You’ll be disappointed to learn that you cannot. Your arguments are added after the alias, so you’d get something like

$ set -f
$ /path/to/globme.sh
$ set+f
$ YourFirstArgument YourSecondArgument ...

So maybe you’ll drop the alias and define a function in your .bashrc / .bash_profile to handle this for you. Maybe something like

globme()
{
    set -f
    /path/to/globme.sh
    set+f
}

You’d be further disappointed when you found out that the shell expanded your wildcards before they were passed to the function, so you’d be setting the -f option too late!

Working (but not optimal solutions)

My first solution was to edit my script to respawn the shell upon exiting. Simply put, I would use the shell’s exec command to exit into a new instance of the shell. Specifically, I would add

exec /bin/bash

or to generalize it (perhaps unnecessarily),

exec ${SHELL}

at the end of the program.

The problem with this solution is that you would lose any environment variables you had set. For example:

$ TEST="Testing"
$ echo $TEST
Testing
$ exec ${SHELL}
$ echo $TEST
(blank line)

This is not expected behavior unless you explicitly respawn your shell instance. Also, consider that your script may have multiple exit points (perhaps erroring out at various points, etc) — in that case you’d need to add the line at every exit point. Further complicating the problem is that by doing this, you’re losing the exit-code of your program. Whereas before you could have used return 1 to indicate an error (which can be viewed by echo $? as the next command after your program), now your exec will have to be in it’s place.

So next, my almost-done solution integrated both aliases and functions:

alias globme='set -f; g'
g(){ /path/to/globme.sh "$@"; set +f; }

Here, the alias allows us to set the shell option first, so the wildcards are not expanded. Then, the function g() is called, which is simply globme, with “$@” as the argument (will get to that in a second), followed by the command to clear the shell option.

This works. Perfectly. To check, you could type

echo $-

and if you see an ‘f’, the option is set (remember, it’s set with -f and cleared with +f) — if you don’t, it’s cleared.

The “$@” expands to the arguments that were passed to the function g — which in this case, are the arguments you passed at the prompt!

Cut to the chase!

Lastly, I realized that it might be a little annoying to have a function and an alias for each script that you wished behave like this. To generalize our lines, I changed them to the following:

reset_expansion(){ CMD="$1"; shift; $CMD "$@"; set +f; }
alias globme='set -f; reset_expansion /path/to/globme.sh'

which basically does the same thing, but it allows you to reuse the reset_expansion() function. For example if you had a second script that you wanted to behave the same way named newscript, you would only need to add the following to your .bashrc / .bash_profile:

alias newscript='set -f; reset_expansion /path/to/newscript'

And that’s all she wrote. Now, the bash philosophers out there will will say something along the lines of “Just enclose wildcard arguments in quotes”, which has a lot of merit, but I’d rather add these lines and not worry about it. Enjoy!

(Continued from Part 2)

cprintf.h

////////////////////////////////////////////////////////////////////////////////////////////////////
//
// (C)2011 Edwards Research Group
// You are licensed to use this work under a CC-BY-SA License.
// See: http://blog.edwards-research.com/about/
//      http://creativecommons.org/licenses/by-sa/3.0/us/
//
////////////////////////////////////////////////////////////////////////////////////////////////////

#include <stdio.h>
#include <stdarg.h>

enum CP_ColorType {BLACK   = 0,
                   RED     = 1,
                   GREEN   = 2,
                   YELLOW  = 3,
                   BLUE    = 4,
                   MAGENTA = 5,
                   CYAN    = 6,
                   WHITE   = 7,
                   UNDEF   = -1};

enum CP_AttrType  {NONE    = 0,
                   BOLD    = 1,
                   DIM     = 2,
                   UNDERLINE = 4,
                   BLINK   = 5,
                   REVERSE = 7};

class ColorSet
{
public:
    CP_ColorType _fg;
    CP_ColorType _bg;
    CP_AttrType  _attr;

    // Default Constructor
    ColorSet(){ set(UNDEF,UNDEF,NONE); }

    // 1 param constructor
    ColorSet(CP_ColorType fg){ set(fg,UNDEF,NONE); }

    // 2 param constructors
    ColorSet(CP_ColorType fg, CP_AttrType attr){ set(fg,UNDEF,attr); }
    ColorSet(CP_ColorType fg, CP_ColorType bg) { set(fg,bg,NONE); }

    // 3 param constructor
    ColorSet(CP_ColorType fg, CP_ColorType bg, CP_AttrType attr){ set(fg,bg,attr); }

    void set(CP_ColorType fg, CP_ColorType bg, CP_AttrType attr)
    {
        _fg   = fg;
        _bg   = bg;
        _attr = attr;
    }

    // Cool Wrapper
    void cprintf(const char * fmt, ...)
    {
        va_list args;
        va_start(args, fmt);

        cp_init();
        vprintf(fmt, args);
        cp_rst();

        va_end(args);
    }

private:
    void cp_init(void)
    {
        if(_bg != -1){
            printf("%c[%d;%d;%dm",27,_attr,(30+_fg),(40+_bg));
        }
        else{
            printf("%c[%d;%dm",27,_attr,(30+_fg));
        }
    }

    void cp_rst(void)
    {
        printf("%c[%dm", 27, 0);
    }
};

cprintf_test.cpp

////////////////////////////////////////////////////////////////////////////////////////////////////
//
// (C)2011 Edwards Research Group
// You are licensed to use this work under a CC-BY-SA License.
// See: http://blog.edwards-research.com/about/
//      http://creativecommons.org/licenses/by-sa/3.0/us/
//
////////////////////////////////////////////////////////////////////////////////////////////////////

#include <stdio.h>

#include "cprintf.h"

int main(void)
{
    ColorSet alert(RED, BOLD);
    ColorSet warn(YELLOW, UNDERLINE);

    printf("\n");
    printf("Unformatted...\n");
    alert.cprintf("THIS IS AN ALERT!\n");
    warn.cprintf("THIS IS A WARNING.\n");
    printf("\n");

    int i,j;
    for(i=0; i<8; i++)
    {
        for(j=0; j<8; j++)
        {
            if(j == 3 | j == 6){ continue; }
            ColorSet((CP_ColorType)i, (CP_AttrType)j).cprintf("FG=%d,A=%d", i, j);
            printf("    ");
        }
        printf("\n");
    }
    printf("\n");

    warn.cprintf("Much later, I can simply use warn without having to lookup the style I last used.\n");

    // Example Showing real-time definition and overloaded constructors
    ColorSet(YELLOW, RED).cprintf("Yellow on Red...");
    printf("\n");
    ColorSet(BLUE, REVERSE).cprintf("Reversed Blue...");
    printf("\n");

    return 0;
}

Which looks like this in gnome-terminal:

Leveraging some of the benefits of c++, I was able to streamline some of the ColorSet construction that took up an annoying amount of lines with the struct style.

I also changed the #defines to an enum to make the compile-time checking more intelligent.

Lastly, the overloaded constructors allow me to use a 2-parameter constructor for both (FOREGROUND, BACKGROUND) and (FOREGROUND, ATTRIBUTE) style.

Now, if I were to make this production code, I’d hide my member variables, expose getter and setter interfaces (if necessary), re-prefix the enum’s to avoid conflicts, and separate out the implementation into a .cpp file — but this was really just a proof-of-concept thing.

(Continued from Part 1)

In order to be able to specify a “color set”, and avoid having to define parameters I didn’t want, my next iteration was to move to a structure-based approach:

cprintf.h

////////////////////////////////////////////////////////////////////////////////////////////////////
//
// (C)2011 Edwards Research Group
// You are licensed to use this work under a CC-BY-SA License.
// See: http://blog.edwards-research.com/about/
//      http://creativecommons.org/licenses/by-sa/3.0/us/
//
////////////////////////////////////////////////////////////////////////////////////////////////////

#include <stdio.h>
#include <stdarg.h>

#define CLR_BLACK   0
#define CLR_RED     1
#define CLR_GREEN   2
#define CLR_YELLOW  3
#define CLR_BLUE    4
#define CLR_MAGENTA 5
#define CLR_CYAN    6
#define CLR_WHITE   7

#define ATTR_NONE       0
#define ATTR_BOLD       1
#define ATTR_DIM        2
#define ATTR_UNDERLINE  4
#define ATTR_BLINK      5
#define ATTR_REVERSE    7

typedef struct{
    int fg;
    int bg;
    int attr;
} cset_t;

cset_t * cset_init(cset_t * cs)
{
    cs->fg = -1;
    cs->bg = -1;
    cs->attr = -1;
    return cs;
}

cset_t * cset_setfg(cset_t * cs, int fg)
{
    cs->fg = fg;
    return cs;
}

cset_t * cset_setbg(cset_t * cs, int bg)
{
    cs->bg = bg;
    return cs;
}

cset_t * cset_setattr(cset_t * cs, int attr){
    cs->attr = attr;
    return cs;
}

void cprint_init(cset_t * cs)
{
    if(cs->bg != -1){
        printf("%c[%d;%d;%dm",27,cs->attr,(30+cs->fg),(40+cs->bg));
    }
    else{
        printf("%c[%d;%dm",27,cs->attr,(30+cs->fg));
    }
}

void cprint_rst(void)
{
    printf("%c[%dm", 27, 0);
}

int cprintf(cset_t * cs, char * fmt, ...)
{
    va_list args;
    va_start(args, fmt);

    cprint_init(cs);
    vprintf(fmt, args);
    cprint_rst();

    va_end(args);
}

cprintf_test.c

////////////////////////////////////////////////////////////////////////////////////////////////////
//
// (C)2011 Edwards Research Group
// You are licensed to use this work under a CC-BY-SA License.
// See: http://blog.edwards-research.com/about/
//      http://creativecommons.org/licenses/by-sa/3.0/us/
//
////////////////////////////////////////////////////////////////////////////////////////////////////

#include <stdio.h>

#include "cprintf.h"

void main(void)
{
    // Example Showing Individual Configuration
    cset_t alert;
    cset_init(&alert);
    cset_setfg(&alert, CLR_RED);
    cset_setattr(&alert, ATTR_BOLD);

    // Example Showing Cascaded Configuration
    cset_t warn;
    cset_setattr(cset_setfg(cset_init(&warn), CLR_YELLOW), ATTR_UNDERLINE);

    printf("Unformatted...\n");
    cprintf(&alert, "THIS IS AN ALERT!\n");
    cprintf(&warn,  "THIS IS A WARNING.\n");
    printf("\n");

    cset_t loop;
    cset_init(&loop);
    int i,j;
    for(i=0; i<8; i++)
    {
        for(j=0; j<8; j++)
        {
            if(j == 3 | j == 6){ continue; }
            cset_setfg(&loop, i);
            cset_setattr(&loop, j);
            cprintf(&loop, "FG=%d,A=%d", i, j);
            printf("    ");
        }
        printf("\n");
    }
    printf("\n");

    cprintf(&warn, "Much later, I can simply use &warn without having to lookup the style I last used.\n");

    return;
}

Which turned out like this (in PuTTY):

I liked this solution better than my first. Here I was able to define program-wide color schemes (e.g. alert, warn) and simply reference them on future calls to cprintf(). I could have also added a one-time-use “constructor” that would return a static pointer to a structure of type cset_t based off 3 integers for inline coding.

Something like:

cset_t onu;

cset_t * cset(int fg, int bg, int attr)
{
    onu.fg = fg;
    onu.bg = bg;
    onu.attr = attr;
    return &onu;
}

So that I could also do something like the following in my main program:

cprintf(cset(CLR_YELLOW, CLR_RED, ATTR_NONE), "Yellow on Red...");

As for my C version, this is where I left it — my next iteration was in C++.

Continue to Part 3.