SmartFM Android Study Dictionary v1.1 released

So I even managed to find time for a few performance fixes and have released version 1.1 of the SmartFM Android Study Dictionary to the Android Market.

The main difference is pretty much on the back end with the app using a separate Java client to consume JSON (instead of XML) from the new "unofficial" SmartFM API. This necessitated a performance fix on the main item list view since checking whether there was a sound for an item required a separate HTTP request. This creates the main noticeable different, that you'll briefly see progress indicators for the sound icons when you first get a set of results back.

I was pleased that using ProgressBar, AsyncTask and a cache of my own design was effective in making the list scroll smoothly and I went on to use the same approach for image loading in the sentence list that is part of each individual item view. That's something that was always a performance issue in earlier versions, but only came up for items with more than 3 or 4 sentences.

I'm disappointed not to have been able to add more functionality or fix other minor gripes, but the old version one XML SmartFM API is being discontinued today or tomorrow, so I thought it better to kick out a version that works with the new API, and leave everything else for subsequent releases. "Release Early, Release Often".

Anyhow, I am moving house and starting a new job, so will probably be restricted to bugfixes for the next couple of months, but hope to release some new features and fixes by Christmas. I think my main goals are making the voice search go straight to results without requiring a second click, and pre-loading lots of content so users don't have to wait to download sounds. I think the multiple HTTP requests associated with each individual items could be put into AsyncTasks, to make the item display more responsive ...

The two most frequently requested features are support for some study, and OpenID. I do hope to include a study feature soon. As for OpenID, we'll see ... :-)

To follow my progress subscribe to this blog or the associated Google Group:

http://groups.google.com/group/smartfm-android

Another SmartFM Android Study Dictionary Beta, all media upload now supported

So I've managed to get all the media upload functionality working against the new API, I think ... If anyone is interested in helping me test the new beta it is available here:

http://www.neurogrid.net/public_html/smart_fm.apk

There are still some performance issues, but I think I have an angle of attack on those (AsyncTasks and some caching structure), so if I'm lucky I can put those in place before the August 31st deadline. Final beta before then I hope ...

One interesting note is that SmartFM now has some pretty impressive automated text-to-speech for Japanese Sentences. Words and sentences now have audio added automatically and at least to my untrained ears it sounds pretty good. Makes all the effort to support sound upload on the Android a little redundant ...

The problem with Java, or null-proof logging in Android

So this is really irritating. I keep getting exceptions in Android when my logging string happens to be null, e.g.

Log.d("DEBUG", my_string);

if my_string is null, then I get an exception and execution halts. Not what I want. I want the null nature of my_string to be reported and execution to continue.

The verbose location solution is:

if (my_string != null) {
Log.d("DEBUG", my_string);
}

or:

Log.d("DEBUG", my_string == null ? "NULL":my_string);

but really I'd like to modify the Log.d static method so that check is included by default. Something I could easily do in ruby, but no equivalent I know of in Java that doesn't involve creating some new class and then modifying all my existing calls to this method throughout my code ...

Early Beta of Smart.fm Study Dictionary v2 available for testing

So I've created a stand-along Java client library for the new version 2 of the smart.fm API. I've now used that successfully to create an early beta of the Smart.fm Study Dictionary v2, which is available here:

http://www.neurogrid.net/public_html/smart_fm_2.apk

If you want to help by beta-testing this application you can download directly to your phone by browsing to the above link (ensuring you have set your android phone to allow non-market installs).

I've re-implemented everything that was in the original except the creation & media upload functions - those are planned for next week :-)

Some issues I am aware of are the slowness when downloading or scrolling item lists, and that some items may have sentences from multiple languages inserted. These are due in part to changes in the API and I am working on fixes. I would be most grateful to hear of any other issues or suggestions for changes. My current work TODO list is available at the following link for your perusal:

http://groups.google.com/group/smartfm-android/web/smartfm-v2-todo-list

Android Speech Recognition

The Android speech recognition module is great (I use it in my SmartFM Study Dictionary), but it doesn't have much flexibility. I'd really love to be able to use the audio recording component in it for my own audio recording needs, and also be able to submit my own audio rather than being forced to use the existing audio recording component. I think ideally the speech recognition and recording components would be split up. There are a few enhancement requests already filed with Google that I added stars to in order to increase the chances of this happening. If you have a moment please star them as well.

http://code.google.com/p/android/issues/detail?id=4541

http://code.google.com/p/android/issues/detail?id=6930

http://code.google.com/p/android/issues/detail?id=10022

Towards Tatoeba on Android

Great multi-lingual example sentences project at Tatoeba.org. Wish they had an API, but they very kindly provide a dump of their database. 3rd party projects are currently using that. I'm chatting with the Tatoeba guys about an API and hearing that it is an issue of expense, since they are a "a free project run by a student". Makes me wonder if there is a way to use Google App Engine or other cloud computing platform to help them support an API, or Mashery or 3scale something ...

So anyway, I used RazorSQL to take their csv sentences file and convert that into an SQLite database. The conversion took about 10 hours once I had figured out an issue with RazorSQL trying to always find closing double quotes. So then I got an INSTALL_FAILED_INSUFFICIENT_STORAGE error trying to load an Android app with Tatoeba's full set of sentences stored in an sqlite db that was 25 Mb. I was considering that the better approach would be to download the file from a remote location and store on the SDCard, when I realized the problem was actually that I had left the 50Mb SQL Inserts file in the same folder. It got automatically folded into the Android APK file. I removed that and the application ran. Well I say ran, it had the new problem that it couldn't find the Sentences table, but that was my Android SQLite plumbing and not the file size :-)

Fixing that I got a new error, Data exceeds UNCOMPRESS_DATA_MAX (24825856 vs 1048576), which happened when I tried to import the Tatoeba database from resources into the file system. A little research suggested there was a hard limit of 1Mb on resource and asset files in Android packages. So I moved straight to the network download which is what we would need for market release of an app of this kind anyway.

Download from Google docs worked after I added support for 302 redirect, took a long time - but database did get stored on file system, although the DDMS client died, so I couldn't check the details.

Started to feel like it would be faster to wrap my own rails REST layer around db dump ...

So I got the thing displaying a list of sentences but unfortunately on first attempt the UTF-8 characters were mangled. SQLite appears to support UTF-8, so I wondered if this was an android issue. Android is also UTF-8 compliant, so it seems like it was actually another RazorSQL issue. Although the SQL inserts had been created in UTF-8 when I ran them through RazorSQL the double byte encoding had been lost. I was able to re-generate an SQLite file with intact double byte encoding by running the SQL inserts from the command line - which had the added benefit of being hugely faster.

A bit of jiggery pokery later and I had the entire sentences db of Tatoeba displaying in an Android app. Nice to see the ListActivity performantly handling the display of a table with over 400,000 rows.

So that was a fun day, but still to do are:

1) Work out the best option for download; Google Sites has 20Mb limit, and Google docs stared doing some weird virus redirects on the latest file ...
2) Add some basic search to the app
3) Drop in the relations csv and get things all prettied up :-)

Forking Codebases

So Cerego is going to discontinue official support of their version 1 SmartFM API as of August 31st, which is a shame, as I use that API in my SmartFM Mobile Study Dictionary app (available in the Android marketplace).

I'm going to be switching over to the unofficial version 2 SmartFM which is JSON rather than XML based, so I'm trying to rework the app before the August 31st deadline. I managed to get some preliminary JSON communications going, and since the android test harness is too damn slow I created a separate standalone non-android Java SmartFM JSON client library. So now I'm just about ready to blast through all the old XML communications and change to JSON, but first I wanted to check that this separate library was actually going to work as an android plugin, particularly since I had pulled in a different JSON library to get the standalone test framework to work.

So I created a brand new Android Shell to check that the linkage worked, and it was good, but I started wondering if I shouldn't be forking my existing codebase since I want to be able to apply small fixes to that, and perhaps pull them directly into the new codebase. Also, I need to work quickly to get this all done by the August 31st deadline, so I likely don't have time for the complete rewrite I would like ("release early, release often"), however maintaining a long term fork seems rather involved. I found a walk through of how to do that:

http://programblings.com/2008/07/21/setting-up-a-long-term-fork-with-git/

On balance it seems more trouble that its worth, so I'm going to work with a new repository and move things over piecemeal. Still I'll try and move code over without trying to fix too much, as experience tells me that trying to fix too many issues at once is asking for trouble. The main issues I want to address are removing global variables and unsightly spaghetti code, making the voice recognition auto search, and streamline the content creation process. Better get that JSON stuff working first though :-)