This post is a rework of my response to this thread in the LiveScribe forums.

When developing an app, I have a frequent (maybe a little *too* frequent in some cases) edit/compile/deploy/start-app/test cycle.  The nav cross can be used to start an app.  You double tap on the cross, then tap the down and right arrows as required to find your app in the menu, then tap right to start it.  If you have a few apps you are working on, this quickly becomes EXCRUCIATING!

There is a scrub-n-flick feature available to more quickly navigate through the menu structure. You can draw an up/down line and use it to scroll through menu items instead of using nav up/down arrows. It does work, but I never use it.  I cannot even remember how to set it up (its in the manual if you want to find out how to use it).  It is however faster than tapping ‘next menu item’ over and over again.  But there is a better way.

The approach I prefer is if you double tap the cross of the nav cross, you can then write the application name. (Look for a little “*” to appear on the right side of the display when “main menu” is displayed – this means you can write the app name.) It will show the match the pen will use as you write. You can then later tap on the word (or the first few characters if that is all you wrote) to restart the application again.
For example, double tap on the nav cross, then write “calc” on the page. The calculator app starts. Later you can tap “calc” again to restart the calculator application.

Well, this works for your applications as well.

There was concern expressed in the forums (I was one of them) about collisions between app as the names could not be changed by end users. Developers pick the names.  What if two apps used the same (or nearly same) name?  I am starting to suspect this is not a big deal.  I now just use a longer name resulting in a low propability of identical name collisions.  It is important to note you don’t have to write the whole name, and, importantly, after writing a character or two, the user can use the Nav cross up/down buttons to scroll through the matches if there is a collision on what the user has written so far. (You must then tap the right nav arrow to confirm the current selection is what is wanted.) So its not hard to pick the variant you want (without writing the full application name down).

For example, I have applications called “testa” and “testb”.  I wrote “t” (display showed “testa”) then tapped nav-down (display showed “testb”) and tapped nav-right to confirm.  The “t” became bound to “testb” (not “testa”). To start the app again, I just tap on the “t” again. *Much* better than going through the menu structure each time.

When debugging an app, I always now create one of these short cuts, then I can restart the application over and over again after each deployment with a single tap.

Oh, how do you change the app name?  Look for the “app.name” property in the project’s “build.properties” file.

Unit tests are of course really useful to make sure your code works and continues to work. I use JUnit for writing unit tests, but if you throw them into your ‘src’ directory, they get bundled into the penlet itself!  Not very desirable.

Having a quick look at the ant script invoked to deploy a penlet, it would be possible to extend this script.  But the files are supplied by LiveScribe (and may in a future release be updated by LiveScribe again), so I decided it was better to not touch them.

Instead, I use settings in Eclipse to tell Eclipse about the test classes (put in a separate directory) without the ant script knowing about them.  That way I can at least run the unit tests in Eclipse without the test cases sneaking into the deployed penlet.

Testing LiveScribe classes… Not!

I don’t have a solution at present for classes that use LiveScribe pen classes (like StrokeStorage) – there is an emulator coming which *might* provide useful stub classes here, but I am not sure.  It’s also tricky if you are using the ICRContext for handwriting recognition (or other similar complex functionality).  My best solution to date is to separate application logic from LiveScribe classes as much as possible, making it at least possible to test your application logic with unit tests.

The steps

If you create a new project (using the LiveScribe supplied wizard) you will have a ‘src’ directory.  The following steps describe one way to add unit test classes to the project without them ending up in the deployed penlet.

• Right click on the project and select ‘New’ –> ‘Folder’.  I called the new folder ‘test-src’.
• Right click on the project and select ‘Properties’ (right down the bottom).  Select ‘Java Build Path’, then on the ‘Source’ tab, click ‘Add Folder…’.  Select the test-src directory you created above.
• Tick the checkbox “Allow output folders for source folders” at the bottom of the Java Build Path page.
• Click on the ‘Output Folder’ setting of the test-src directory (on the ‘Source’ tab of the ‘Java Build Path’ window).  Enter a directory such as ‘test-classes’ for where compiled class files can go (rather than into the default location).  (I am not sure this step is strictly necessary, but I do it anyway to be sure.)
• Go to the ‘Libraries’ tab of the ‘Java Build Path’ window and make sure JUnit3 is listed.  (Add it if not there.)
• On the ‘Export and Order’ tab, make sure JUnit3 and Penlet Libraries are not ticked (so they are not exported).  (I think this is the default set up by the wizard.)
• Put your unit test classes under the test-src directory.  I personally for a class such as Chord would call the test class ChordTest (rather than TestChord) as it groups the Chord and ChordTest classes together better.  (This is however completely personal.)
•  Right click and select ‘Run As…’ –> ‘JUnit Test’.  The test should run (and hopefully pass!).  Deploying a penet will not include the test classes.

Conclusions

Unit tests are great to make sure your non-LiveScribe dependent application logic is working.  Unit tests also help make sure that over time if you make changes you have some confidence you have not broken the code.  I don’t have a solution for testing code dependent on LiveScribe provided penlet functionality (such as asking for strokes).  I am curious to see if the emulator coming out Real Soon Now will help with testing or not.  We shall see.  Until then, separate your application code an LiveScribe code and unit test the application code.

A common thing for a LiveScribe pen penlet (pen application) to do is require the user to draw a button that they can later tap on.  Its sort of funky in a way – making the printed page “active”.  The standard “piano” application is one example of this (tap notes to play a tune) and the translator demo (write word in one language, then say word back in another language).  Oh, before you get to excited about the translator demo, it only knows ten or so words.

This post summarizes the main things you need to do when creating such “buttons” on a page.  Note that the LiveScribe provided “Getting Started Platform SDK” and “Developing Penlets” manual goes into technical details in some detail as well, and the “User Experience Guidelines” manual describes recommended user interaction models for consistency across applications.  The following is my overview.  I purposely have left out details to keep this post length down a bit.

Letting the user draw a button

First thing to do is to allow a user to draw a button.  An application may do this in different ways.

• It may require a user to select an item from a menu (navigated using the cross (up/down/left/right) navigation controls.  The user can then create buttons at any time and in any order.
• It may prompt the user upon start-up and ask for buttons to be drawn.  This will result in the application asking for buttons every time it is started (from a menu).
• It may use handwriting recognition to work out what the user has written and turn it into a button if recognized. 

It depends on your application which is most appropriate.  Personally I find the menu approach painful, although it appears the best choice if you have lots of buttons a user may want to write.  Using handwriting recognition is cute, but does not (easily) allow you to draw a box around the button afterwards.

I was playing with a little application for playing music chords.  You could write down the chord name, then tap it later.  E.g. “G”, “A#” (for A-sharp), “Bb” (for B-flat), etc.  More funky chords are possible such as minors, suspended fourths, major 7ths and so on.  The handwriting engine in the pen works *much* better when you tell it what input is legal.  Specifying a list of words is an easy approach to do this (a lexicon).  A separate blog post talks about using a grammar file instead, which is more appropriate for chords (you can say “a note followed by a modifier” without having to list all the possible combinations).  The point is however in this case it made good sense to use handwriting recognition, allowing the user to write a chord on the paper, which then became a button they could tap on later.

If the user decides when to enter a label (via a menu selection or when the application starts), then it is common to just capture the strokes written but not try to interpret them at all.  When the user pauses (or double taps on the page) that marks the end of the stroke input for the button.  You just trust the user to write down an appropriate label.

Capturing a button in code

To capture a button, a common approach is for the penlet’s main class to implement the StrokeListener interface, then register itself using context.addStrokeListener(this) in the activateApp() method and deregister itself using context.removeStrokeListener(this) in the deactiveApp() method.  The strokeCreated()  method (required by the StokeListener interface) is then called whenever a stroke is created on the page.

If you use handwriting recognition, then it is normal for the strokeCreated() method to pass the stroke to the handwriting engine via the addStroke() method of ICRContext.  If you penlet implements the HWRListener, then you implement hwrResult(), hwrUserPause(), hwrError(), and hwrCrossingOut().  Yes, ICR and HWR are really both used for the same concept.  hwrResult() is the important one – it reports the final result of the recognition engine of what you wrote.

So when strokeCreated() is called, how do you know which button it is for?  Well, its up to your application to know.  E.g. if a user selects a menu item, then you save that selection away in a class variable.  The strokeCreated() method then looks at the class variable to work out what it should do with the stroke.  See this blog post for some suggestions here.

To create a button you can tap on later, you have to create a “dynamic region” and register it with the pen.  It is common to use a bounding box of what was written on the page.  If you used handwriting recognition, you can use the ICRContext.getTextBoundingBox() method to get a bounding box.  Sometimes I create a new Rectangle object with a bit of extra padding around this bounding box to make it easier to tap.  If you just capture the strokes yourself, then you can ask the bounding box of a Stroke using Stroke.getBoundingBox().  (To get the stroke, you use StrokeStorage.getStroke(time) where the time is a parameter of the strokeCreated() method.)  If you want to capture multiple strokes (common), then you can build up a bounding box around all the strokes using Shape.getUnion(rect1, rect2) to merge the rectangles.

So how to work out when a button is finished?  Double taping is one approach (but beware of strokes if the user moves the pen fractionally when they tap on the page – you might not want them as part of your button!), and pausing is another approach.  If using the handwriting engine, it does the pausing code for you.  Otherwise you need to use the Timer class to trigger an alarm after the timeout has expired.  (You reset the alarm timer when strokeCreated() is called.)  I have another blog post talking about timers briefly.

Area IDs

Once you have your rectangle representing the button, you need to tell the pen about it.  You do this by calling context.getCurrentRegionCollection() to get a RegionCollection instance, on which you then call addRegion().  You pass this call the rectangle (or any other shape) for your button.  You also have to pass a Region object.  I find this class name a little confusing – it is not a shape but rather metadata about the shape of the region on the page to be made active.  In particular, you can specify an “area ID” which is a 16-bit unsigned integer.  When the user taps the button on the page, you get back the Region data structure.  (To be precise, the strokeCreated() method of StrokeListener and the penDown() and penUp() methods of PenTipListener are given the Region as an argument.)

So what do you use and Area ID for?  Well, it tells you what button was touched.  If you application has 5 different types of buttons, it could be set to 1 to 5 where each number represents a different button type.  (Note Area ID zero is reserved to mean “no region was touched”.  That is, its an unused area on the page so far.)  To chose what action to take, you can just do a switch on the area ID from the region.  (See Region.getAreaId().)

For my chords application, I had many possible combinations (due to the complexity of the grammar).  In the end I tweaked my grammar so there were around 50,000 legal combinations.  That way each one fitted into a 16-bit integer (64K).  Given a number I knew what chord (and modifier) was selected.  Note it was not perfect however.  I had to take short cuts when there were multiple ways to write the same chord.  So sometimes a user might have written “Asus4” but when the user tapped on the chord I displayed “Asus” (which is the same).  A trifle confusing when the user sees “Asus4” on the page.  (Providing feedback on the display I find useful to confirm what is being played, in case the handwriting recognition software got it wrong for example.)

Another approach I could have used was to use PenletStorage which provides a file system within the pen.  I could have allocated areaId’s one by one and create a new file for each chord written (for example).  Each file might contain the chord name and any other details about the chord I wanted to keep.  The filename might be the area ID.  (A single file could of course also be used with more sophisticated parsing code.  One file per areaId is just easier to explain.)  Another file might keep track of the next areaId to use.

There is nothing wrong with this last approach, but I personally try to avoid it as it opens up a whole new area of complexity (messing around with files) and associated bugs that can sneak in.  I like keeping things simpler.

It may be obvious, but you can of course split up your range of area IDs to be a mix of the above.  For example, IDs 1 to 99 might be reserved for commands, 100 to 999 for files, and 1000 onwards using some form of static allocation.  Whatever makes most sense for your application.

Summary

Creating a button (dynamic region) on a page is pretty easy.  Bounding boxes are a common way to make the region sensitive.  You need to put a little thought into how you are going to allocate area IDs for your buttons if you use them.  Hopefully the above is useful to someone.  All of the information is in the SDK documentation, but sometimes hearing the same information in different forms can help.

This is a quick post on timers in a LiveScribe pen penlet (pen application).  I had asked in the forums about how to track the pen position as it moved across the page.  The following is my take on the response I got.

Sample Code

In this case, I am going to start with code and then describe it.

package com.wordpress.alankent.livescribe.play;

import com.livescribe.penlet.Penlet;
import com.livescribe.penlet.PenletStateChangeException;
import com.livescribe.penlet.Region;
import com.livescribe.storage.StrokeStorage;
import com.livescribe.ui.ScrollLabel;
import com.livescribe.util.Timer;
import com.livescribe.util.TimerTask;
import com.livescribe.afp.PageInstance;
import com.livescribe.display.Display;
import com.livescribe.event.PenTipListener;
import com.livescribe.geom.Point;

public class TrackPenPosition extends Penlet
        implements PenTipListener {

    private Display display;
    private ScrollLabel label;
    private PollTask pollTask;
    private Timer pollTimer;

    public TrackPenPosition() {
    }

    public void initApp() throws PenletStateChangeException {
        display = this.context.getDisplay();
        label = new ScrollLabel();
        label.enableFlickScrub(false);
    }

    public void activateApp(int reason, Object[] args) {
        context.addPenTipListener(this);
        if (reason == Penlet.ACTIVATED_BY_MENU) {
            this.label.draw("Here is my scrolling label", true);
            this.label.enableFlickScrub(false);
            this.display.setCurrent(this.label);
        }
    }

    public void deactivateApp(int reason) {
    }

    public void destroyApp() throws PenletStateChangeException {
    }

    public void penDown(long time, Region region,
                        PageInstance page) {
        System.out.println(">>> PEN DOWN <<<");
        StrokeStorage ss = new StrokeStorage(page);
        pollTask = new PollTask(ss);
        pollTimer = new Timer();
        pollTimer.schedule(pollTask, 10, 15);
    }

    public void penUp(long time, Region region,
                      PageInstance page) {
        System.out.println(">>> PEN UP <<<");
        pollTimer.cancel();
    }

    public static class PollTask extends TimerTask {

        StrokeStorage ss;

        public PollTask(StrokeStorage ss) {
            this.ss = ss;
        }

        public void run() {
            Point p = ss.getLastCoord();

            // Do work with the new coordinate
            System.out.println("Coord = " +  p.toString());
        }
    }

    public void doubleTap(long time, int x, int y) {
    }

    public void singleTap(long time, int x, int y) {
    }
}

Description

What the above code does is create a new Timer instance when the pen goes down.  It also creates a new class that is called when the timer expires.  You can used a non-static inner class giving it full access to the parent penlet class, although you might prefer to use a separate class and pass all the required information across in the constructor.

Timers can issue multiple events, which I have used in this case.  The Timer.schedule() method takes the callback class to use, the initial timeout, and the time between subsequent timeouts as arguments.

To get the current pen position, in the timer it asks the stroke storage for the last coordinate.  See StrokeStorage.getLastCoord().

The end result is you get told about when each pen move occurs.

Beware however that you may need to use synchroized methods to protect penlet class variables.  The Timer class uses a separate thread.  Its not uncommon in real code to require a synchronized keyword on all the Penlet methods as soon as Timer instances are introduced in order to make sure that any code triggered by the Timer is thread safe with respect to the main Penlet class.  Its too big a job to describe synchronization and thread safety here – its important to understand however if you are going to use Timers safely.

A common question that comes up on the forum is how to wait for user input.  The answer is Penlets don’t wait for more input – they react to events.

(The following post is a massaged version of this thread on the forums.)

Consider a simple penlet where you want to ask the user to write some buttons on a page that they can tap on later.  The penlet might display a first label (with a verbal prompt as well if you are being fancy), wait for input until the user pauses say for a second or two or does a double tap, then displays the next prompt.  So how do you write the code?  Do you display a prompt, wait for input, display the next prompt, wait for input again?  Yes, but not as sequential code.

Bottom line is you don’t wait for input – you react to events then return, remembering the state you were in so when the next event occurs you know where you were up to.

I find that, as with most event based models, a state machine is a good way to go. That is, draw a diagram with every single state your application can be in. There will be an initial state, then maybe a state for drawing a control on a page where double tap completes the control drawing and moves the app on to another state. You end up with lots of states circles and ways to move from one state to another (arrows between circles where you write on the arrow when the arrow should be followed – e.g. on double-tap).

Another way of thinking about it is to write a big table where you have a column for each event type you are interested in, and a row for each state your application can be in. For each state you work out what you want to have happen for each column. (Normally there is some action you want to have occur plus the new state to go to.) The advantage of this approach is it makes you think about what to do for every event that can occur in every state that exists. But it captures the same information as the diagram.

Note: You may have events other than just pen up, pen down, double tap etc such as timers going off, or controls being tapped etc. Depends on the complexity of your application logic.

How to implement state? Basically you have an integer variable (or a combination of several variables) holding the current state. This is normally done using a field (member variable) of the base penlet class. In each callback method you do a switch on the state variable to work out what code should be executed when the app is in a particular state.

It may be obvious, but with an event based system you cannot (easily) write procedural code where you suddenly in the middle of the code say "I would like to collect user input now". You must exit from your penlet back to the pen OS and wait until you are called again with the next event.

This is often called the "reactor" pattern. You react to events that occur then exit your code until the next event comes along to react to. You can think of it like playing tennis/squash/ping pong etc. Something happens and you react (hit the ball back), then get ready for the next thing (event) that happens.

Note: I don’t like using switch statements that much actually (which is the normal way to do a state machine). In play apps I have been writing I have instead being writing a class per state, then having the main penlet class delegate calls through to my current state object. It just makes the code a bit more manageable when the code gets complex. If you only have 3 or 4 states a variable is fine. But as the code grows, you start having code for a single state scattered all over the place. To me its much nicer to group all the code for a particular state in one place (a class). Its easier to add new states later. It also allows you to add you own application specific event types such as "pen moved". Pen moved events you might generate by having a timer going off say every 20ms while the pen is down, then asking the stroke storage the current pen position.  Its nicer to write your application logic in terms of a “pen moved” event rather than having all the complex logic for detecting pen movements etc mixed in with your mail application logic.  But I discussed that a bit in a separate blog post.

I have been very tempted to buy an iMac. I have an iPhone and love it. I have been doing some reviews of apps for young children (dropped off a bit due to work commitments, trying to get back into it). I would love to have a crack at writing an app myself (even if I never released it). The price difference between the 21″ iMac and 27″ iMac seemed so small, I thought I would go with the bigger model.

Before buying, I thought I would do a quick search for news on the iMac. I mean, should not be necessary – Apple always do top quality stuff don’t they? Err, no. There are lots of reports of flickering screens, yellow tinges, etc. Apple released one firmware fix, but it did not appear to fix the problem for everyone. There is now talk of another fix coming. My best hope was this article where someone said running the fans faster fixed the problem for them. Basically seemed like a heat problem. This article was another approach – effectively putting it into screen saving mode for a bit to let things cool down (without turning off the whole computer).  It also reported the iMac being very hot.

So I might let things cool down before buying one myself.  Probably a good thing for those work deadlines too!

I originally thought the LiveScribe pen could only play one note at a time. However, what I did not realize at the time was the pen supports JSR-135, which is a MIDI interface. MIDI is how computers can talk to devices like digital pianos, keyboards, drum kits etc. Using MIDI, chords can be played on the pen – that is, multiple notes playing at the same time to make a chord sound.

Details on MIDI can be found at http://www.midi.org/. I found the table at http://www.midi.org/techspecs/midimessages.php particuarly useful.

The following is some sample code to help you get going. There are some other articles in the LiveScribe forums, but the following shows the essence of talking to the MIDI subsystem. I like wrapping it in a separate class to keep the main code cleaner.

package com.wordpress.alankent.livescribe.support; import java.io.IOException; import javax.microedition.media.Manager; import javax.microedition.media.MediaException; import javax.microedition.media.Player; import javax.microedition.media.control.MIDIControl; /** * Helper class designed for accessing the * LiveScribe pen internal MIDI interface. * This class uses JSR135, so should work with * other MIDI devices but may not make all the * the functionality of those devices visible. */ public class Midi {     private Player midiPlayer;     private MIDIControl midiControl;     /**      * A suggested channel number to use if only      * going to play one note at a time.      */     public static final int DEFAULT_CHANNEL = 0;     /**      * The number of concurrent channels that can      * play at once.      * Channels are numbered 0 to 15.      */     public static final int NUM_CHANNELS = 16;     /**      * The maximum velocity that can be used when      * playing a note.      */     public static final int MAX_VELOCITY = 127;     /**      * The MIDI note number for middle C.      */     public static final int NOTE_MIDDLE_C = 60;     /**      * Program number for a piano sound. See      * http://en.wikipedia.org/wiki/General_MIDI      * for a long list of program numbers. The      * LiveScribe pen is not a high quality MIDI      * device, so does not necessarily make      * sounds like the instruments listed.      */     public static final int PROGRAM_ACCOUSTIC_GRAND_PIANO = 0;     /**      * Constructor. Connects to the MIDI      * subsystem and prepares to play sounds.      *      * @throws MediaException      * @throws IOException      */     public Midi()             throws MediaException, IOException {         midiPlayer = Manager.createPlayer(                         Manager.MIDI_DEVICE_LOCATOR);         midiPlayer.prefetch();         midiPlayer.start();         midiControl = (MIDIControl)                 midiPlayer.getControl("MIDIControl");     }     /**      * Play a note on the specified channel at      * the specified velocity.      *      * @param channel      *            The channel to play the note on.      * @param midiNote      *            The note to play.      * @param velocity      *            The velocity of the note to play.      */     public void playNote(int channel,                          int midiNote,                          int velocity) {         midiControl.shortMidiEvent(                         MIDIControl.NOTE_ON | channel,                         midiNote,                         velocity);     }     /**      * Silence a channel (stop playing any sound).      *      * @param channel      *            The channel to silence.      */     public void turnSoundOff(int channel) {         final int CONTROL_ALL_SOUND_OFF = 0×78;         midiControl.shortMidiEvent(                 MIDIControl.CONTROL_CHANGE | channel,                 CONTROL_ALL_SOUND_OFF, 0);     }     /**      * Silence the MIDI device completely (all      * channels).      */     public void turnAllSoundOff() {         for (int channel = 0;                         channel < NUM_CHANNELS;                         channel++ ) {             turnSoundOff(channel);         }     }     /**      * Change the "program" (instrument sound) to      * be used by the specified channel.      *      * @param channel      *            The channel to use.      * @param program      *            The program from the bank to use.      */     public void setProgram(int channel, int program) {         midiControl.setProgram(channel, 0, program);     } }

I was going to write up a long explanation, but really the code is pretty simple. There is detailed documentation in the JSR-135 specification if you really want to know more.

The LiveScribe pulse pen has built in handwriting recognition software.  Unfortunately its not very good unless your pen application (penlet) gives it a lexicon.  With a lexicon, it works pretty well – however when my lexicon reached 50,000+ lines I found out about a currently undocumented feature, Ludef files.  Ludef files are a grammar file (instead of a flat list of terms), which suites chords perfectly.  My 50,000 lines dropped to under 50 lines.

Background on Chords

First, I am not a music expert.  I used to play keyboard a bit, and frequently cheated by playing the guitar chords instead of the real music.  But here is a bit of background for those with even less knowledge than myself.  The following may make anyone who knows about music cringe – I am trying to keep things simple for any non-music people, not describe it 100% correctly.

On a piano, there are 7 white keys “A” to “G”.  These make up an octave (the “oct” for 8 comes from including the starting note at the end as well).  There are sharps (“#”) and flats (something that looks like a “b”) which raise or lower the sound by a semitone.  A semitone is one note up or down on the piano, including the black keys.  “C#” for example is the black key to the right of the white key for note “C”.  “Cb” is actually just “B” as there is no black key on the piano between “B” and “C”.  People just write “B” instead of “Cb”.  “Db” is the same black key as “C#” – both names are used in practice.

Chords are three or more notes played together that sound good together.  The chord “C” involves playing “C”, “E”, and “G” together.  Chords can have more notes, and chords can have variations that make them sound different.  For example, the chord C-minor (“Cm” or “Cmin”) is “C”, “Eb”, “G”.  To make a normal chord into a minor chord you lower the second note of the chord by a semitone (“E” became “Eb”).  Minor chords sound more sorrowful.  There are lots of other types of chords – major, suspended, augmented, diminished, power, added, 7ths/9ths/etc.

On top of that, it is common to specify the inversion of a chord to use.  The chord “C” can actually be played as “C”, “E”, “G” or “E”, “G”, “C” (next octave up), or “G”, “C”, “E” (where C and E are next octave up).  To specify which inversion, the notation “C/E” is used.  This means the “C” chord where “E” is the lowest note played.  However, as a lazy keyboard player, I would play the easiest version of the “C” chord in the right hand and just play the lowest note in the left hand.  It sounds good.

Why all of the above?  Well, hopefully you can start to see there are rules to how chords are written.  You have a root note (“A” to “G”), you can have an optional accent (“#” or “b”), you can have various modifiers, and you can have an optional bass note (after a “/”).

Using a Lexicon

Back to writing a penlet.  The handwriting recognition software in the pen does its best, but is not ideal. For example, for my writing it often gets “G” and “9” mixed up. It also gets “A” and “#” mixed up at times as well.  A lexicon is a list of terms that are legal inputs, in my case chords (like “A”, “G#”, “Cm”, “Gsus4” etc).  If you supply a lexicon it gives the handwriting subsystem a hint as it knows what letters can appear where.  For example, chords always start with “A” to “G” (never a “#” or “9”).  After using a lexicon the recognition was pretty accurate.

If you want to develop an application with handwriting recognition, use the new project wizard in the LiveScribe Eclipse plugin.  It has an option to add handwriting recognition to the new project.  I picked the option to type in a subset of characters I wanted recognized.  I added “ABCDEFGabdgijmnosu#/123456789+-”.  As a zero never occurred, I left it out (avoiding confusion with “o”).  I also picked to specify a lexicon.

Well, there were a lot of variations – for example every chord could specify “/<note>” so I wrote a little program to generate all the combinations I wanted to support.  It might not have been perfect, but my first generation created a 16,000+ line file.  Next version with some more fixes was over 50,000 lines.  Egad!  It worked, but was a bit scary.

Gotcha!

When I started out trying the handwriting code with a lexicon, I was bitten by a known issue.  The pen internally converts the lexicon into some internal representation that it then caches.  Redeploying the modified lexicon has no effect unless you turn off and on the pen (which flushes the cache).  Thoroughly frustrating times may eventuate if you don’t know this minor detail – they sure did for me!  I would make changes that had no effect, then come back my next programming session (after pen auto-shutoff) and they would magically work!  Things went much more smoothly once I knew to power cycle the pen after each lexicon change.

Ludef Files

All this work with handwriting recognition was in part triggered by someone else on the forums having some troubles as well.  Maybe due to critical mass of questions, LiveScribe published a sample application using a Ludef file.  A Ludef file specifies a grammar rather than a list of legal values.  This is perfect for chords.  My Ludef file is below.  Note this is not a perfect representation of how chords can be written.  In the forums the sample code contains a different (more complex) grammar file.  I wrote my own because I wanted the grammar to match my code which parsed the writing afterwards.  (Warning, some line wrapping may have occurred.)

note = (C)|(C#)|(Db)|(D)|(D#)|(Eb)|(E)|(F)|(F#)|(Gb)|(G)|(G#)|(Ab)|(A)|(A#)|(Bb)|(B) octave = [1-5] slash = (/){note}{octave}? min = (m)|(-) aug = (aug)|(+) dim = (dim)|(o) maj = (maj) nth = (7)|(9)|(11)|(13) minor = {min}{nth}? augmented = {aug}{nth}? diminished = {dim}{nth}? major = {maj}{nth} dominant = {nth} add = (add) add2 = {add}?(2)|{add}(9) add4 = {add}?(4)|{add}(11) add6 = {add}?(6)|{add}(13) added = {add2}|{add4}|{add6} power = (5) sus = (sus) sus2 = {sus}(2) sus4 = {sus}(4)? suspended = {sus2}|{sus4}|{sus}{nth} modifier = {minor}|{augmented}|{diminished}|{major}|{dominant}|{added}|{power}|{suspended} pattern = {note}{modifier}?{slash}?|{slash} {pattern}

 

The grammar of a Ludef file is fairly clear.  Literal text is in parenthesis, rule names are referenced by putting them in braces, question mark means optional, vertical bar means alternatives, no operator means a sequence.  I am assuming readers are familiar with the concept of a grammar here so I am not going to go into details.

Enabling the Ludef file

To use the Ludef file, I created a project using the Eclipse wizard with a lexicon, then made the following changes.

• Comment out the line icrContext.createAppResource(“/icr/LEX_MyApp.res”).  This assumes your application was called MyApp of course.
• Add a line icrContext.createAppResource(“/icr/LUDEF_MyApp.res”).  This references the compiled MyApp Ludef file.
• In the src/icr directory rename the MyApp.lex file to MyApp.lex.IGNORE.  (You can delete it – I am just cautious and like being able to go back easily.)
• In the src/icr directory create a file MyApp.ludef and put your grammar into it.  The file extension is important.  Its how the build phase knows the file is a Ludef file.

And you are done!

Personally I have not noticed any difference in my application behaviour using a Ludef file or lexicon.  But this is to be expected.  There may be differences in file sizes or performance (I did not check).  What was important to me however was the length of the file (much easier to manage and inspect) and that I could now compare my source code parsing a chord as a string to the Ludef grammar.  Its much easier to make sure the two are in sync.

Having a go with a Ludef file is easy.  The wizard creates enough of a handwriting application to display recognized text on the pen display as it goes.  So its easy to give it a go yourself if you want to.  Of course, you have to get the LiveScribe Penlet SDK (registering as a developer).  But its free.  Oh, having a pen would be useful as well!!!

Conclusion

I don’t think the handwriting software built into the pen is that useful without a lexicon or Ludef file.  It just gets too many errors.  A lexicon file is great if you have a simple list of terms you want to match.  A list of English words for example works fine.  But if the input text matches a grammar, a Ludef file is probably more suitable.  The recognition rates goes up dramatically.  You do still get some errors, but not many.

By the time you see this article, its possible Ludef files are documented in more detail by LiveScribe.  Check out the LiveScribe forums at livescribe.com for more details.

The LiveScribe pen allows penlets (pen applications) to be written in J2ME (Java 2 Micro Edition) and deployed onto the pen.  There are not a lot of penlets around yet, so I thought I would share a few personal opinions on how to structure penlets well, based on early penlet writing experience (and years of commercial programming background).

Anatomy of a Simple Penlet

A penlet has a main class that implements the Penlet interface.  It has methods such as initApp(), activeApp(), deactivateApp() and destroyApp() which are called at different times during the pen’s lifecycle.

In simple applications, it is also common for the main class to implement several other interfaces and then register itself as a “listener” for specified types of events.  Examples of events are pen on/off paper, a new stroke has been written on paper, and a menu item has been selected.  The advantage of the one class implementing multiple interfaces like this is it makes it trivial to share variables between all the different event handlers and the lifecycle context.

It is important to understand that penlets are event driven.  They get called when something happens.  Like web applications and other event based systems, a penlet cannot display a prompt then sit and wait for the user’s response.  Instead, after displaying a prompt, the penlet must then return back to the pen OS.  When some event intended for the application occurs, the application is called to process the event.  The penlet is therefore continually reacting to events (hence the name “reactor” in the design pattern literature).  It remembers state between events in class fields.

Larger Penlets

For small penlets, having a single class with all the code in it is fine.  However, as the penlet grows, this approach soon becomes problematic.  In particular, the logic starts getting more complex and confusing and automated testing is difficult. Logic complexity and automated testing are addressed separately below.

Penlet versus Application Events

The issue with logic complexity is in practice then pen events are not exactly what the application is typically after.  For example, imagine an application where if you draw some text it parses the text written (using the handwriting recognition software built into the pen) and creates a “button” for it.  Later if you touch the text (the “button”) then it invokes that operation.  It is almost impossible to put the pen onto the page without a slight movement, so in both cases you will get pen up/down event(s) – possibly several during the writing phase – and stroke events.  However the application really wants to know when a button was created, and when an existing button was tapped.  In this case, the application code would be cleaner if the application had newButton() and buttonTouched() methods that were called at the appropriate times.

Diving deeper here, pen down and up events are triggered first, followed by a stroke event.  The handwriting subsystem accepts strokes until the user pauses, which time it hands its best attempt at parsing the writing to the application.  If an existing “button” is tapped, the pen down event includes an ID for the region that was touched.  In this case there is no need (or desire) to pass the stroke on to the handwriting subsystem.  So the “new stroke” event should do nothing if after a pen down over an existing region.  If new text was written, then when capture is complete a new region would be registered to detect later touches on it.  In this case the new stroke event should do some work.  The point is, the penlet events may not directly align with what the application wants to receive.  Some code may be desirable to map penlet events into application events, clarify the overall application structure.

Automated Testing

Automated testing is important for all applications (not just penlets).  Automated tests allow an application to be modified with confidence that existing functionality has not been broken.  When a bug in code is found, I always recommend writing a unit test first (demonstrating the code is broken) then fix the problem.  The unit test typically helps when fixing the problem.  Then you also have a unit test to make sure the same bug never comes back again.

The problem with automated testing for penlets is the number of LiveScribe classes that only exist on the pen itself that are referenced from code.  This includes events passed into the penlet, but also includes outputs such as playing of sounds or updating the pen display.

Given that it is currently difficult to impossible to test the LiveScribe specific logic, the next best approach is to split the application into LiveScribe specific code (code that can reference LiveScribe classes) and application specific code (code that never references LiveScribe classes).  The application code can then be unit tested using junit or similar.

The challenge is how to decouple the code.

Having a Penlet class call the application code with special application events is trivial.  All required data can be copied out of LiveScribe classes and into application data structures.

Having the application code output data back to the LiveScribe classes is harder.  The best approach is for the application to define interfaces for callback methods it requires.  For example, there may be a updatePenDisplay() method defined in an interface.  The Penlet main class can then provide implementations of any required application interfaces to the application class.  Unit tests would supply a different implementation of the callback interfaces allowing the tests to capture the output for verification.  This does result in more overall code (various interfaces with two implementations – the real code and test infrastructure code), but it does make unit testing possible (of the application code).

Is it Worth It?

For simple applications, introducing the separation between LiveScribe specific code and application code that makes no reference to any LiveScribe class is probably overkill.

However, as soon as automated testing is wanted, splitting the code is strongly recommended.  It also helps with the application code clarity as the application code is able to be defined in terms of events it wants to receive.  Having code in pen up/down etc methods for all application purposes can result in spaghetti code pretty quickly.

LiveScribe just had a developer day where LiveScribe talked to a number of topics.  Slides of the presentation are available here.  Highlights are:

• Partial results from a survey of LiveScribe users
• More details on on competition
• Roadmap of future features

They even had a special guest – James Gosling of Java fame.  Applications on the pen are written using J2ME.

Survey Results

How happy are users?  According to the survey, 72% said the pen had changed the way they work.  91% rated the pen as good or excellent (on a 5 point scale).

Note: they did not report how many said it was bad (as distinct from “average”)!  There have been quite a few complaints in the LiveScribe forums about the quality of the software driving the pen, including all sorts of upgrade issues to the most recent desktop.  Its hard to judge how widespread the problem is (many have reported everything worked perfectly for them).  Forums tend to be visited by people with problems rather than people without, so the audience is probably skewed.  On the other hand, the survey was probably taken before the most recent desktop upgrades were released.

What do users say they want from the survey?  Productivity tools, utilities, and education were the top three (in that order).  Reference was a bit lower, and everything else dropped away significantly.

How many pens are out there?  Around 300,000 pens have been sold.  A useful figure to know if you are thinking of developing an app for the pen.  (The SDK is free.  LiveScribe will take a commission from pen sales.  The only outlay is US$100 to get a third party testing group to approve your application before you can get it into the store.)

Competition

LiveScribe now have a beta application store up.  There are a few apps there.  To try and kick start additional apps,they are running a competition Dec 7th 2009 to Feb 13th 2010 with a few categories with a US$5,000 prize in each.

Roadmap

The roadmap was interesting, but a little disappointing.

• The “Desktop SDK” which will allow data to get out of the pen is now not available until 2010Q2.  It looks like full functionality will only initially be available from C# apps (sorry Mac users!).  Not being a C# programmer, this was a disappointment.  The desktop API just keeps to keep slipping later and later.
• They are working on a knowledge based for Q1.  That should be useful.  The current documentation is pretty basic, and lacking as soon as you try to write a non-trivial application.
• There is an emulator planned which should make debugging easier.  I assume you use the mouse instead of a pen – it will be interesting to see if you can control a mouse as accurate as a pen (for doing handwriting for example).  Not sure if this will help with automated testing of penlets (pen applications).

 

Summary

I still believe LiveScribe need to clean up the quality of their base functionality.  There are also many obvious gaps in desktop functionality.  My biggies are:

• Inability to export an audio session to a file for sharing (you can only do this now by uploading to the LiveScribe web site – no use for corporate usage)
• No way to delete pages, tag, or otherwise organize content.
• No way for apps to get data out of the pen.

The continued lack of desktop SDK is a sore point to me.  The app store value is a bit questionable at the moment.  Coming up with useful apps when you cannot get data off the pen is pretty hard.  But the emulator was a nice surprise.  And they have published a roadmap, something LiveScribe don’t have a great reputation for.

With a market size of 300,000, if say one in a hundred users by an app, that’s 3,000 sales.  I doubt iPhone style $1 apps are going to make a profit!