Jobs Fast Jobs Fast

Cross-Domain AJAX – A simple workaround

First off, this isn’t new. It’s not actually breaking any security protection, its just a way to perform cross-domain AJAX requests, something that is usually blocked by your browser.

Consider the following javascript code[1]:

var requestData = {
        op: "fetchRecords",
        id: 124,
        key: "Q1GxWcxWOrKY"
};
var request = $.ajax({
    url: "http://some-other-domain.com/json",
    data: requestData,
    dataType: 'json'
});
request.done(function(msg) {
    callback(msg);
});

Unless the other domain is explicitly configured to allow for cross-domain AJAX requests, your request will fail.

Quick sidenote here: This failure may be very confusing, especially if you’re not aware of the same-origin policy restriction. The AJAX query will fail (so the fail and error methods will be called), but it will not indicate why it failed. More puzzling is that Firebug shows that the response header as 200 OK, but no data is passed back. Just a heads up in case this happens.

One option, if the server is setup to handle it, is JSONP. You can read more about it on the Wikipedia article, but unless the server is already configured to handle JSONP requests, or you can modify it to do so, this will not be an option for you.

When using third-party data providers, like in my case, these are not options.

What I needed was some way to asynchronously fetch the contents of a URL, but since I couldn’t do it directly, I did it indirectly.

To workaround this issue, I first created an extremely simple PHP script[2] that I called proxy.php:

<?php
$file = file_get_contents($_GET['requrl']);
echo $file;
?>

All this script does is fetch the URL that is specified in the requrl GET parameter and return its contents.

For example,

http://www.mysite.com/proxy.php?requrl=http://google.com

would just display the contents of google’s main page.[3]

And because this file, proxy.php, was hosted locally on the same domain, I’m able to send AJAX requests to it without issue.

If we take the javascript example above, it could be re-written to use the proxy as follows:

var requestData = {
        op: "fetchRecords",
        id: 124,
        key: "Q1GxWcxWOrKY"
};
var request = $.ajax({
    url: "proxy.php",
    data: {requrl: "http://some-other-domain.com/json" + $.param(requestData) },
    dataType: 'json'
});
request.done(function(msg) {
    callback(msg);
});

Now, the client will fetch proxy.php via AJAX, the PHP server will fetch the specified URL and return it in response to the client’s AJAX query.

If you notice, we’re even able to pass GET parameters to the proxy and those parameters will be used by the webserver when fetching the page. This is because jQuery’s param function will URL encode the query string after creating it from the map, eliminating any ambiguity about where the query string parameters should be sent to.

The reason this doesn’t break the same-origin policy is because the same-origin policy is meant to prevent the client from making unintended/unexpected requests (and usually, exploiting the fact that the client’s cookies are sent and can be used to imitate a user’s intentional actions). In the case of this proposed workaround, it is the server that is making the request and your cookies will be safe.

With any code samples you find on the internet, this one included, you should read up on the functions you’re considering using before putting them in any (especially production) code. :-)

Notes:

  1. This snippet, and the other javascript snippets require jQuery. Also, I specify the response dataType as json so that jQuery will parse the returned json and return an object. If you don’t want this, you can remove or edit this line.
  2. This is just a proof-of-concept proxy script and shouldn’t be used in practice. Without first checking things like referrer or requested URL against a whitelist, anyone could use the proxy for whatever they wanted. It’s just not a good idea to leave production code like this.
  3. Because google uses relative paths for things like CSS and images, the page you fetch via the proxy may look a bit different, specifically it may be missing styling, images or scripts. This is almost never an issue if you’re using a JSON API or even just scraping pages for the text data.
Leave the first comment

X Forwarding with sudo

With PuTTY and Xming I use X Forwarding to do a lot of work on my local linux boxes on my Windows box.

After a reformat of both, I ran into the issue where a command like wireshark would work fine but running it via sudo (sudo wireshark) failed with the error:

X11 proxy: wrong authorisation protocol attempted

For me, the only thing that was required was to properly define the $DISPLAY (which was fine) and $XAUTHORITY (which wasn’t set) environment variables and make sure they’re passed through to root via sudo.

The first can be done via:

export DISPLAY="localhost:10"                    # For example, this might be different for you -- this is probably already set
export XAUTHORITY="/home/<username>/.Xauthority" # For me, this wasn't set.  Also, make sure to use the full path and not ~

Once that is done, ensure that these two environment variables are “kept” via sudo by checking /etc/sudoers and finding the lines that look like:

Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"

These lines define what environment variables persist for the root user when sudo is invoked. By default in my installation (Fedora 16, x64) both DISPLAY and XAUTHORITY persist (first and last lines). If your file doesn’t have this, the environment variables won’t persist and you won’t be able to use X Forwarding as easily/transparently, so you should consider adding them.

Lastly, since $DISPLAY is already set correctly, I just added the $XAUTHORITY definition (shown above) in by ~/.bashrc file and I was good to go.

One comment so far, add another

Installing Eclipse/Subclipse on Windows 7 x64

Subclipse is a great subversion plugin for Eclipse allowing very tight integration into the IDE. Installing it on a 64-bit JVM is a bit tricky, however, and not as straightforward as it probably should be. Today’s post will walk through setting up Subclipse on a fresh Eclipse install.

We’ll be doing this the “right way”, using public key authentication (“passwordless”), but I’ll also point out the steps that can be skipped if you want to enter a password each time with an asterisk*. Note: entering a password for every transaction gets very tedious and I would recommend using public key authentication as I outline.

First, I’ll list the versions I’m working with:

  • Java: 1.7.0_03(x64)
  • Eclipse: 3.7.2 (Indigo) (x64)
  • SilkSVN: 1.7.5.1 (x64)
  • Subclipse: 1.8

You’re free to use a newer version if this post becomes outdated, but things may change which may or may not invalidate this walkthrough.

Prerequisites

A 64-bit JVM installed. Installing subclipse is presumably a lot easier if you have a 32-bit JVM installed.

Install

Update: I originally outlined the steps necessary to install a version of Subclipse that included packages that were shortly thereafter outdated. Mark Phippard, the project manager for Subclipse and developer for Subversion stopped by and updated us with the current status of the project, including fact that the updated packages make this install a lot easier. As a result, I’ve struck out the steps (4,5) that were made unnecessary in the latest update. Thanks Mark!
  1. Install Eclipse (Indigo x64)
  2. Install Subclipse via Eclipse (Help – Intstall New Software – Add)
  3. Exit Eclipse
  4. Download and Install SlikSVN — This is needed for the 64-bit JavaHL libraries. (What is JavaHL and why do I need it?) Make sure you select the 64-bit edition.
  5. Ensure you have MSVC++ 2010 (x64) Redistributable Package Installed. If you think you have this installed, you can skip this step. If you encounter an Eclipse error dialog stating: no msvcp100 in java.library.path, you’re missing this.
  6. Install TortoiseSVN (or just download TortoisePlink.exe). Note that TortoisePlink.exe is different than the PLINK.EXE that is provided by the PuTTY package and using that version will not work.
  7. Download Puttygen.exe* (available here, either as part of the entire PuTTY package or alone). Note that this step will be necessary if (a) you want to use passwordless login and (b) you don’t already have a private/public keypair (if you’re unsure about this, you probably don’t have one and generating another one is no big deal).
  8. Create Public / Private Keypair using Puttygen.exe* — save these somewhere you’ll remember (I use %HOMEPATH%\Keys which is (on Win7) C:\Users\<username>\
  9. Install public key on remote computer.* Note: this might depend on the OS of your subversion server, but generally the steps are the same: (1) copy public key to subversion server, (2) add key to ~/.ssh/authorized_keys. Something like:
    cat id_rsa.pub >> ~/.ssh/authorized_keys
  10. Configure Subclipse to use plink. Open %appdata%\Subversion\config and in the [tunnels] section, add the line
    ssh = C:/Progra~2/Putty/TortoisePlink.exe -l username -i C:/path/to/private_key

    Notes: (1) change paths and username as appropriate, (2) The path separator should either be a single forward slash or a double (escaped) backslash. If you decide not to use public key authentication, drop the last (i) option.

  11. Launch Eclipse
  12. Open SVN Configuration (Window – Preferences – Team – SVN) and ensure JavaHL (JNI) 1.7.5 … is selected under (SVN Interface – Client) and an error dialog isn’t displayed. If you get an error dialog and the dropdown contains JavaHL (JNI) Not available, you may have forgot to exit and restart Eclipse or something went wrong while installing SlikSVN.
  13. Add repository (Window – Open Perspective – Other… – SVN Repository Explorer – Right click in “SVN Repositories” – New – Repository Location) and enter repo location (svn+ssh://subversion.server/path/to/repo)

Results

At this point, one of four things should happen:

  • Your repository opens up in the Repo Browser. Congrats! You set up Subclipse successfully. Code on!
  • A password dialog opens up. Typically this is because either (a) you erred when specifying your private key on the ssh=… line, or (b) you didn’t, or didn’t correctly, add the public key to the authorized_keys file.
  • You get a red console error:
    Network connection closed unexpectedly
    svn: Unable to connect to a repository at URL ‘svn+ssh://path/to/repo’
    svn: To better debug SSH connection problems, remove the -q option from ‘ssh’ in the [tunnels] section of your Subversion configuration file.

    This means that you were able to connect to the server, but for some reason the connection later failed. This could be because you specified an incorrect username. Ensure your TortoisePlink.exe options are correct.
  • You get a red console error:
    The system cannot find the file specified.
    svn: Unable to connect to a repository at URL ‘svn+ssh://path/to/repo/’
    svn: Can’t create tunnel: The system cannot find the file specified.

    This means that Subclipse couldn’t locate TortoisePlink.exe. Ensure your path is correct and your path separators are appropriate.
2 comments so far, add yours

Tutorial: Android JNI (Part 2)

This is continued from Part 1.

In the first part, we setup the project structure, wrote the java wrapper class, generated the c header, wrote the c library and write the Android.mk file.

Building shared library

With traditional JNI (not targetting Android devices), you just compile the library with gcc using the -shared option. However, for Android devices, we use the ndk-build script.

To compile, we first switch to the project directory (the directory that contains the jni and src directories then run the ndk-build script provided in the Android NDK:

cd Android-JNI-Demo
/path/to/ndk-build # for example: ~/build/android-ndk/android-ndk-r7c/ndk-build

If when trying to compile on linux you get the following error:

Invalid attribute name:
    package

You’ll want to check the line endings of your AndroidManifest.xml. I used the dos2unix command to correct them.

If the build script found your Android.mk and the library compiled without issue, you’ll see the following:

Compile thumb  : squared <= squared.c
SharedLibrary  : libsquared.so
Install        : libsquared.so => libs/armeabi/libsquared.so

So we’re happy with the library compilation and now we’ll move on the developing a simple UI to test whether our functions perform as we expect.

Develop Simple UI

By default, when we created a new Android project in Eclipse an activity was generated with the following:

package org.edwards_research.demo.jni;

import android.app.Activity;
import android.os.Bundle;

public class Android_JNI_DemoActivity extends Activity {
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
    }
}

If we didn’t care about looking pretty, we could change this to something like:

package org.edwards_research.demo.jni;

import android.app.Activity;
import android.os.Bundle;
import android.util.Log;

public class Android_JNI_DemoActivity extends Activity {
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        int b = 3;
        int a = SquaredWrapper.to4(b);
        Log.i("JNIDemo", String.format("%d->%d", b,a));
    }
}

and run it, either in the emulator or on a device (with USB debugging enabled), we would see in LogCat an entry tagged with JNIDemo. In this case, we’d expect something like 3->81, since 3^4 = 81. But we’ll do a little bit more to see the performance of the library directly on the UI.

Instead of walking through the specific steps of creating the UI, I’ll simply post the pertinent files:

Android_JNI_DemoActivity.java

package org.edwards_research.demo.jni;

import android.app.Activity;
import android.os.Bundle;
import android.view.View;
import android.widget.EditText;
import android.widget.TextView;

public class Android_JNI_DemoActivity extends Activity {
    private EditText etInput;
    private TextView txtTo2;
    private TextView txtTo4;

    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);

        // Define Input EditText, TextViews
        etInput = (EditText) findViewById(R.id.etInput);
        txtTo2 =  (TextView) findViewById(R.id.resTo2);
        txtTo4 =  (TextView) findViewById(R.id.resTo4);
    }

    public void cbCalculate(View view)
    {
        int in = 0;
        try{
            in = Integer.valueOf( etInput.getText().toString() );
        } catch(NumberFormatException e) { return ; }

        txtTo2.setText(String.format("%d", SquaredWrapper.squared(in)));
        txtTo4.setText(String.format("%d", SquaredWrapper.to4(in)));
    }
}

res/layout/main.xml

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:orientation="vertical" >

    <TableLayout
        android:id="@+id/tableLayout1"
        android:layout_width="match_parent"
        android:layout_height="wrap_content" >

        <TableRow
            android:id="@+id/tableRow1"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content" >

            <EditText
                android:id="@+id/etInput"
                android:layout_width="150dp"
                android:layout_height="wrap_content"
                android:inputType="number" />

            <Button
                android:id="@+id/button1"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="Calculate"
                android:onClick="cbCalculate"/>

        </TableRow>

        <TableRow
            android:id="@+id/tableRow2"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content" >

            <TextView
                android:id="@+id/lblTo2"
                android:layout_width="match_parent"
                android:layout_height="wrap_content"
                android:text="@string/squared" />

            <TextView
                android:id="@+id/resTo2"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="" />

        </TableRow>

        <TableRow
            android:id="@+id/tableRow3"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content" >

            <TextView
                android:id="@+id/lblTo4"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="@string/to4" />

            <TextView
                android:id="@+id/resTo4"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:text="" />

        </TableRow>

        <TableRow
            android:id="@+id/tableRow4"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content" >
        </TableRow>
    </TableLayout>

</LinearLayout>

res/values/strings.xml

<?xml version="1.0" encoding="utf-8"?>
<resources>
    <string name="app_name">Android-JNI-Demo</string>
    <string name="squared">Squared:</string>
    <string name="to4">To 4:</string>
</resources>

When run in the emulator the app looks as follows:

You can enter a number in the text field and press the calculate button which will result in the squared and ^4 calculations:

Of course this was a pretty silly demo library because a squared function could have been trivially implemented in java without the need for c code, cross compiling or dealing at all with the Java Native Interface, however it still illustrated the steps necessary to compile a native library against the Android NDK and how to import and use it in an Android Project.

Leave the first comment

Tutorial: Android JNI

Today I’ll be posting a quick walkthrough of how to create and build a simple android project that includes native code using the Java Native Interface (JNI). As a note, there are sample projects included in the Android NDK, but this will walk you through building your own. After going through this, it’s suggested you review these sample projects.

Prerequisites

As a prerequisite for this tutorial, you’ll need:

  1. Eclipse installed and configured to create Android projects. There are a number of tutorials out there about how to do this if you need help.
  2. A JDK installed, as I don’t believe the standard JRE contains the javah command that will be needed.
  3. The Android NDK (available here) downloaded and extracted somewhere.

Create the Android Project

For this tutorial we’re going to create a new project. However there is no reason you couldn’t integrate this into an already-created project.

To create the project, right click in Eclipse’s Package Explorer → New → Android Project. Give your project a name and select an API. For this tutorial I choose the latest Gingerbread API, 2.3.3.

Add jni folder, Android.mk makefile

Once your project has been created, you’ll need to create a new folder inside the top level of the project. To do this right click on your project name → New → Folder. Name this folder jni.

Inside this folder, create a new blank text file. To do this right click on your newly-created jni folder → New → File. Name this file Android.mk. Leave this file blank for now, we’ll come back to it later.

Your project should look something like this:

Create java source

For this tutorial, we’re going to have a simple c program — squared — that accepts an int and returns the square (e.g. 2 → 4, 3 → 9, etc.)

In order to accomodate that, we first create a java source wrapper. The wrapper’s job is to load the library, expose any native functions we wish to use directly, and provide any functions that we want to be able to utilize private native functions.

For this tutorial, we’re going to expose directly the native squared function as well as provide a to4 “derivative” function.

To expose the native function directly, we just declare it public. Alternatively, we could declare it private and limit it’s availability to other functions of the class.

Our full java source is

package org.edwards_research.demo.jni;

public class SquaredWrapper {
    // Declare native method (and make it public to expose it directly)
    public static native int squared(int base);

    // Provide additional functionality, that &quot;extends&quot; the native method
    public static int to4(int base)
    {
        int sq = squared(base);
        return squared(sq);
    }

    // Load library
    static {
        System.loadLibrary(&quot;squared&quot;);
    }
}

Create C header

After we outline the native methods we’ll be using, we can use this java source to create a c header file with the function prototypes for the native methods we used. To do this, we first have to compile the java source into a class file. You can do this manually via the javac command, e.g.:

cd src # change into the source directory
javac -d /tmp/ org/edwards_research/demo/jni/SquaredWrapper.java

Note that the -d switch specifies the output directory for the class file — in this case, I’m just throwing it into /tmp.

Now that we have the class, we can create the c header file., e.g.:

cd /tmp
javah -jni org.edwards_research.demo.jni.SquaredWrapper

Note the need to specify the fullly-qualified class name (including package) and not the .class file extension.

The resulting header file in our case is /tmp/org_edwards_research_demo_jni_SquaredWrapper.h, but we can rename it to whatever we want. In this case, we’ll rename it to squared.h and place it in the jni folder in our project directory.

The resulting squared.h file looks like:

/* DO NOT EDIT THIS FILE - it is machine generated */
#include &lt;jni.h&gt;
/* Header for class org_edwards_research_demo_jni_SquaredWrapper */

#ifndef _Included_org_edwards_research_demo_jni_SquaredWrapper
#define _Included_org_edwards_research_demo_jni_SquaredWrapper
#ifdef __cplusplus
extern &quot;C&quot; {
#endif
/*
 * Class:     org_edwards_research_demo_jni_SquaredWrapper
 * Method:    squared
 * Signature: (I)I
 */
JNIEXPORT jint JNICALL Java_org_edwards_1research_demo_jni_SquaredWrapper_squared
  (JNIEnv *, jclass, jint);

#ifdef __cplusplus
}
#endif
#endif

The function name is annoying long, I agree. We could eliminate a lot of that if we didn’t use a package in our java source, but I’m not really that concerned.

Create C source

Using the prototype generated by javah, we can implement our c source as follows:

#include &quot;squared.h&quot;

JNIEXPORT jint JNICALL Java_org_edwards_1research_demo_jni_SquaredWrapper_squared
  (JNIEnv * je, jclass jc, jint base)
{
        return (base*base);
}

Note we have to give the parameters names and I arbitrarily chose je andjc, and chose base to replicate our java source parameter name.

In this case, the c source is very simple, but this tutorial is meant to illustrate how to include native code into your Android app and more complex c functions could be substituted with few modifications.

Create Android.mk

After we create our c source file, we have to create our Android.mk file. This file serves the as a sort of makefile for the Android build tools. There are a number of sample Android.mk files in the samples/ directory of the NDK and we’ll actually be using almost the exact lines from the hello-jni sample project.

Our Android.mk file looks like:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE    := squared
LOCAL_SRC_FILES := squared.c

include $(BUILD_SHARED_LIBRARY)

At this point, we have laid most of our groundwork for setting up and compiling the library. The next steps are actually creating the shared library, and implementing some simple UI code to show that our native function (squared) and derivative function (to4) work as expected.

This is continued in Part 2.

Leave the first comment

Quick tip: “Commenting out” current command in terminal

So if you’re ever writing a command in a terminal, only to realize you forgot to do something first (e.g. switch to the right directory, change permissions etc.), you don’t have to delete the entire row, do what you forgot to do and retype.

To some, this might be obvious, they might suggest holding down the left arrow until you reach the beginning of the line and “commenting it out” by inserting a # in front of the text.

But there are actually two quicker solutions, one somewhat obvious, and the other not so much. The reason I’m writing this is just to point out the second one.

The first quick solution would be just to press the “Home” key and then insert the # symbol and press enter. This is the pretty obvious method.

The second method, which is actually a keystroke shorter, is to press Escape then the # symbol. These two keys have the same result as the three keys in the first method.

Saving a single keystroke is really not the motivation for posting this tip, but rather the fact that some terminals, especially when first configuring your client / initialization scripts don’t accept the home key correctly (or quite often your client isn’t sending it correctly).

Anyway, in that situation, its helpful to have a backup option and the “Escape #” method comes in very handy.

Leave the first comment

Grabbing Flash files from cache

Adobe has been making it increasingly difficult to grab flash files that are downloaded and cached, most likely for anti-piracy reasons. While I do not in any way support piracy, I have to say that my experience is that watching flash videos in something like VLC is such a better experience than watching it in the browser plugin, especially in linux.

The Adobe plugin creates a secure temporary file — the typical way to do this is to (a) create a file, (b) open a file descriptor to it, (c) unlink (delete) the file. As long as the file descriptor remains open, the file will be fine. But once the file descriptor is closed, the bytes are free to be overwritten by the filesystem. The last step of unlinking is in an attempt to limit other programs from accessing the file.

In our case, flash’s cache file is located in /tmp and named something like “FlashXXXXXXXXX”, for example: /tmp/FlashXX9Vj3jC

However, if you go into /tmp and run ls, it won’t be there — because it’s been deleted.

You can verify this by opening firefox, going to a site with a flash video, and running something like the following:

ls -l /proc/$(pidof plugin-container)/fd/ | grep Flash

For example, I did this and got:

l-wx------. 1 user user 64 Mar 20 22:41 17 -> /tmp/FlashXX9Vj3jC (deleted)
lrwx------. 1 user user 64 Mar 20 22:41 23 -> /tmp/FlashXXSUb8Wv (deleted)

Here we’re just looking at the plugin-container’s open file descriptors with “Flash” in them.

While you can’t access the cached files via /tmp/FlashXXXXXXXX, you can access them via the process file descriptor. For example, in this case the file descriptors 17 and 23 correspond to the two cached files.

To copy the first one, you do something like:

cp /proc/$(pidof plugin-container)/fd/17 flashsave.flv

To make all of this a bit easier, I wrote the following python script:

#!/bin/env python

import subprocess
import sys
import os
import shutil
from datetime import datetime

debug = False
outdir = '/tmp/flashsave'

# Great size_fmt function
# Source: http://stackoverflow.com/questions/1094841/reusable-library-to-get-human-readable-version-of-file-size
def sizeof_fmt(num):
    for x in ['bytes','KB','MB','GB']:
        if num < 1024.0:
            return "%3.1f%s" % (num, x)
        num /= 1024.0
    return "%3.1f%s" % (num, 'TB')

# Find the pid of libflashplayer via plugin-container
ph = subprocess.Popen(['pidof', 'plugin-container'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
[out,err] = ph.communicate()

# Ensure out is an int
if len(out) == 0:
    print('Process not found.')
    sys.exit(1)

pid = int(out)
print("Process found.  PID: %d" % pid)

# Find open file descriptors
flash_fds = list()
filenames = os.listdir('/proc/%d/fd/' % (pid))
for f in filenames:
    fd = int(f)
    link_name = os.readlink('/proc/%d/fd/%d' % (pid, fd))

    if(debug): print("  %3d %-20s" % (fd, link_name))

    if(link_name[0:10] == "/tmp/Flash"): flash_fds.append(fd)

# Display flash file descriptors
i = 0
for fd in flash_fds:
    path = '/proc/%d/fd/%d' % (pid, fd)
    size = os.stat(path).st_size

    print("Flash temp file found: %-20s [%s]" % (path, sizeof_fmt(size)))

    dpath = "%s/%s-%d.flv" % (outdir, datetime.now().strftime("%Y-%m-%d-%H%I%S"), i)
    shutil.copy(path, dpath)
    print("Copeid to: %s" % (dpath))
    # Increment counter to try to keep filenames unique
    i = i+1

It’s really informal and could use some clean up, but it does the trick. A few things to note: (1) you will want to wait until the video has finished loading before running this script. You’ll only save what has been loaded — if you run it earlier, you’ll copy a half-complete file, and (2) you’ll need to either create the folder

/tmp/flashsave

or update the outdir variable. To run, simply save this as flash-save.py and type

./python flash-save.py

or make it executable with chmod and type

./flash-save.py
Leave the first comment

Windows BSODs and the Driver Verifier

The dreaded Blue Screen of Death

For a while, I was getting random BSODs and some searching led me to decide to enable a feature called the driver verifier. [More info about Driver Verifier here] My understanding of the driver verifier is that, when a driver acts out of line, even a little bit, even if to an extent that otherwise would be tolerated, Windows throws up a BSOD and lists the offending driver.

In my case, I was getting the Special Pool Detected Memory Corruption BSOD with no specific driver being listed. The number of solutions I found online were almost unbelievable: everything from bad memory to corrupt MBR to corrupt CMOS settings. In fact, none of these would have solved my issue. But a few sites seemed to indicate that this meant that some driver was failing the verifier, but to determine which specific one, I had to examine the dump file that is generated once a BSOD is displayed.

So I loaded into a recovery session, and examined the dump file (C:\Windows\MEMORY.DMP — actually, it was D:\, but thats another story). After analyzing it, WinDbg guessed that my mouse driver was causing the issue (RzSynapse.sys, presumably for the Razer mouse I had connected).

So I disconnect the mouse and restart hoping all would be well, but of course nothing is that easy and I immediately got another blue screen. This time, however, it identified a driverm and a different one. The problem here, however, is that the driver was for a remote management application I use, and not something I could easily disconnect to prevent it from being loaded.

So I know I need to update this driver, but I can’t do so easily because windows BSODs out before it even lets me log in.

So I decide to disable the verifier first, and then update the drivers.

In a command prompt in the recovery session I type

verifier /query

but it says something to the effect of “no drivers are being verified”. But that was simply not true, since the newest BSOD clearly says a driver failed verification.

I guess that the

verify

command doesn’t work properly inside a recovery session, so I go to check the registry. From pages like this I find the registry path, so I fire up regedit, navigate to the where the keys should be and… they’re not there.

Well this complicates the matter. On one hand, the BSOD indicates that driver verification is enabled, but neither the command line “verify” command nor the registry seems to think so.

After thinking about it for a few minutes, I recalled that another page I had read previously discussed loading a registry hive into regedit. I didn’t think much of it at the time, but now it got me thinking that, maybe what regedit was displaying was a registry specific to the recovery session, and the registry that was used to boot windows was somewhere else.

I followed the steps outlined in this second article to load my system registry hive and voila, the driver verifier keys were, of course, present. I promptly delete them, unload the registry hive, and restart the computer. Hoping for the best, but fearing for the worst I walked out of the room to do some other things and when I came back, I was greeted by the glorious login screen — something a little bit earlier I hadn’t thought I’d see again without reformatting.

So, recap — lessons learned:

  • In a recovery session, the registry regedit displays isn’t the registry you might expect. You need to load external hives to edit them. See the link above for how.
  • Driver Verifier is a decent trick to pin down random, unexplained BSODs, but the hassle it caused me makes me wish I simply made an effort to save more often and never even worried about curing them. Simply throwing users to a BSOD seems harsh, especially when they may not cause BSODs had the verifier not been running. I mean, it would be nice if I could “ignore” it, so I could then continue to boot windows and update the offending drivers.
  • There must be some recovery session information stored on non-primary disks, because when I disconnected my non-primary disks in an attempt to rule out hardware issues, the recovery session option was gone and in its place was a message asking me to boot from an install or recovery disk. When I replaced the disk, the recovery session option returned. Also interesting was the fact that, when I rearranged by drives in my BIOS correctly (primary on SATA order 1, secondary on SATA order 2) the recovery session again disappeared. I’m sure this would be easy to look up and verify, but I’ve wasted enough time on this issue — maybe later.
Leave the first comment

Script to Show All Font/Background Colors in Bash

gnome-terminal makes up about 50% of my windows open on my linux boxes — but it defaults to a color scheme that is pretty dark if you choose to change the background to black (like I do).

In order to adjust the color scheme, you can go into Edit -> Profile Preferences -> Colors Tab.

However, its kind of hard to tell how the colors will look until they’re displayed on the screen with the black background. The following simple script will generate a text in every font color, in every background color, and in all the styles:

#!/bin/bash

# The syntax for escaped color sequences looks like:
#
#    BLUE="\[\033[0;34m\]"
#
# Lets see what we can find...

NONE="\033[0m"

for COLOR in $(seq 30 40) ; do
    for STYLE in $(seq 0 1) 4 5 7; do
        TAG="\033[${STYLE};${COLOR}m"
        STR="Color:${COLOR}--Style:${STYLE}"

        echo -ne "${TAG}${STR}${NONE}  "
    done
    echo
done

The output looks like:

Tango (default) theme

Linux Console theme

Xterm theme

Leave the first comment

Gnome 3 Delete Key — what happened!?

So I updated a few of my workstations to Fedora 16. I had been hesitating to upgrade from F14 because with F15 came Gnome 3, a pretty drastic departure from the traditional desktop environment we’re all used to. Some of the notable changes (and some of the first ones I found work arounds for) were:

  • Removal of minimize and maximize button in title bar, leaving only the “x” close window button.
  • Removal of a task switching bottom panel
  • Removal of “Shutdown” / “Restart” from … Shutdown? menu — leaving only suspend
  • Forced grouping of windows by application when you’re alt tabbing. Now 3 gnome-terminal sessions + firefox + gedit will show as 3 windows instead of 5 and require you to press down to descend into the individual windows of each group.
  • Remove the notification area available for applications and force all the applications that used it (sometimes, very cleverly) to rework their apps. See this long list of “suggested changes” proposed by the Gnome devs.

The list really goes on, but what really bothered me was that I found that the delete key in the file manager no longer worked. To be clear, this isn’t exactly a Gnome3 issue but rather a Nautilus issue, the filemanager for the gnome desktop.

At some point, the developers (at least those that accepted the patch) decided that they would diverge from decades of desktop environment tradition and change the shortcut from “Delete” to “Ctrl + Delete”. The rationale being that “Delete” was too easy to press by accident, and since there was no notification that the files were moved to the trash, it would be transparent to the user that accidentally pressed delete.

The thing is, I think they correctly identified an issue: accidentally pressing delete could result in user confusion when they can’t find their files. But what is amazing, is how they chose to address it — not by popping up a notification but by simply making it harder to accidentally (and intentionally) delete.

Its amazing because every OS I can think of already pops up a notification when you’re “soft-deleting” files. By soft-deleting, I mean “moving to trash”, etc. — something that is completely reversible — and that’s the functionality we’re talking about with the (formerly) Del / (now) Ctrl+Del command.

But instead of popping up a simple notification, they chose the almost inexplicable route of confusing every desktop user who is even relatively comfortable with the limited subset of keyboard commands that “just work”.

Huge changes like this, I argue, require sufficient motivation. In this case, the desire not to have a notification pop up is no where near sufficient. It surprised me that this bug fell through the cracks, but it baffles me that its possible the Nautilus devs might think this is “working as intended” and not a bug. This issue is not settled, as the bug is still technically open, but the way the devs seem to be leaning is that this change is acceptable.

For example:

Proposing WONTFIX. As written before this is not considered a bug but
intentional behavior.

–André Klapper [developer] 2012-02-18 09:22:13 UTC

But, to be honest, its not surprising. The arrogant / selfish way that the gnome desktop has “evolved” in recent history should have prepared me for this. From the outside looking in, it seems its just a bunch of people who think they know better about how the majority of the userbase should want to work — even despite an inordinate number of complaints and an immeasurable amount of time spent by third party developers in attempts to “bring back gnome2″. More on that in the links below.

So in closing, I upgraded to Gnome3, but found myself unhappy as I expected would be the case. In most of my workstations I’ve noticed a general decrease in performance clearly attributable to the gnome-shell. I spent a considerable amount of time getting it to feel better, and it does, but I would really prefer my Gnome2 back.

That being said, there are two main options for Gnome3 users who hate the new UI.

The first is using “fallback mode”. This gets the user extremely close to the Gnome2 look and feel. Its not as polished as Gnome3 is, but it gets the job done.

The second is adding “shell extensions” to gnome to reproduce the functionality that Gnome longer ships with. Specifically, this include a “Places” menu drop down, removing some of the annoying icons and menus that are there by default, restoring minimize and maximize buttons, restoring “shutdown” and “restart”, and (most importantly, imo) restoring the bottom panel task bar.

I’ve taken the latter approach and the best set of extensions I’ve found so far are the Mint Gnome Shell Extensions. For a while, it seemed like Mint was going to back a Gnome2 fork (which was encouraging), but now it looks like they’ve decided to go with Gnome3, but reproduce the missing functionality through extensions. These extensions are available on their github site, and can be easily installed on a fresh F16 install.

I only use a few of these, as well as a few from Gnome’s own https://extensions.gnome.org/.

While I have managed to get back 75-80% of the look and feel, the increased resource issue still a problem and I only expect it to get worse. I use linux on my workstations so that I can get quck calculations at the expense of visual flair. I’d be okay working (and do in fact often work) primarily from the command line. These workstations had all of the visual flair disabled (Compiz, 3d rendering, transparency, etc) and ran like a charm. Unfortunately, it looks as if this will not be the case with Gnome3.

Leave the first comment