vOICe in our mobiles

The vOICe MIDlet for Mobile Camera Phones

Try it on your Nokia 6600, 6620 or Sony Ericsson K700i
3D audio on the stereo-enabled Nokia 6620, 6630, 6680
The vOICe camera-based visual sound technology for the totally blind is now available for most Java-enabled camera phones and PDAs! Free! The main Java requirement is Java ME (J2ME) MIDP-2.0 & MMAPI compliance. Moreover, The vOICe is expected to be future proof and run on all future Mobile Service Architecture (MSA, JSR-248) compliant mobile devices that fully support camera input and audio output for rendering the live audio map (pixelated acoustic map), and should run on all future JavaFX Mobile compliant phones.
The vOICe MIDlet software already runs on both Symbian and non-Symbian devices. Apart from the . The software even includes a talking color identifier, such that you can point the camera of your Java-enabled smartphone or PDA to any item of interest and hear the color name spoken. Moreover, the software is available free of charge for non-commercial personal and academic use!
Nokia 6600 with built-in camera (lens is on the backside). This smartphone is capable of running The vOICe MIDlet.




Example views. Hovering the mouse over a view shows a spectrographic reconstruction obtained from the linked stereo wav sound sample. The recognizable reconstructions prove that much visual information was preserved in the one-second visual sounds.

The vOICe is now available for multiple hardware platforms to suit different user preferences. Apart from The vOICe Learning Edition for an immersive "backpack" system with a UMPC, PC camera (webcam or video sunglasses) and stereo headphones, some users prefer using the much simplified but highly portable "magic wand" version that runs on mass-market modern camera phones and PDAs. For continuous use and maximum context-awareness, a head-mounted camera would be much preferred for best sensory feedback, but for occasional orientation and wayfinding purposes, for reading signs, or to have a look at graphs or other graphical material in print, on the blackboard or on displays, or for simple use as a light probe and light locator, the mobile camera phone version of The vOICe can form an attractive alternative. Depending on your personal interests, you may also consider it as a mobile augmented reality game - the game of sight (disclaimer). You can read a few accounts written by blind users of this software.

The vOICe perceptual interface approach does not require any infrastructural adaptations for making objects "visible" to the totally blind. However, it is possible to add talking signs, i.e., visual tags that speak: just try The vOICe for mobile phones on the specially crafted image shown on the right! Now use your phone as a smart camera companion, a stand-alone computer vision system where all image processing and audio synthesis is done by the phone.

Within a decade, second-hand camera phones could become an affordable platform for use of The vOICe by blind people living in developing countries! These phones can at the same time serve many general communication, Internet access and pervasive computing needs for the sighted poor, while doubling as a digital camera. The vOICe camera phone software can thus form an example of targetted "educational software for cell phones" (BBC News).




Download


The vOICe MIDP-2.0 MIDlet, version 1.20.0, can be downloaded directly to your UMTS/3G or GPRS enabled phone by entering the URL of the WAP (WML) page
phone.seeingwithsound.com ( or type the shorter tinyurl.com/qebgl, or use the semacode / QR code / shotcode visual tag if your phone has a visual code reader )

Alternatively, for instance if your phone does not support OTA ("over the air") downloads, or in case of connection problems, you can first download the software to your PC as a zip file from the URL

www.seeingwithsound.com/phone/vOICeMIDlet.zip



Note: Do not forget that on many camera phones you must right after installing or re-installing The vOICe change the multimedia permissions for The vOICe application via the phone's application (App) Manager to "Ask first time" or "Ask once".
Also beware that some phones automatically launch their built-in camera application upon opening the camera cover, and in that case you first need to close that other camera application before The vOICe can access the camera. This is because only one application can access the camera at any given time. In any case, you will have to slide open the camera lens cover if your phone has one, or else the camera view will remain "black".


Once started, The vOICe MIDlet will continuously grab and sound snapshots from the camera. There are no connection costs while using it, because The vOICe MIDlet runs off-line. Each camera snapshot is sounded via a polyphonic left-to-right scan through the snapshot while associating height with pitch and brightness with loudness. For example, a bright rising line on a dark background sounds as a rising pitch sweep, and a small bright spot sounds as a short beep.

More information is available in the The vOICe MIDlet manual (Microsoft Word format).

Please report if you encounter problems (or solutions) that are not listed on this web page, but first check the detailed remarks section near the bottom of this page, and check if you have the latest firmware for your phone (Nokia: *#0000# ; SE: >*<<*<*>). See the update history if you are considering an upgrade from an earlier version.







Now if you are blind, you may need initial sighted assistance with the installation, but after that you should be able to work with the software using fixed keystroke sequences. The details of that are not specified here, because these may vary with the brand and type of device. Beware that with most camera phones the lens is on the backside of the phone, so the visual object of interest should then be behind the phone when you are pointing the phone to the object while you are facing the keyboard just like you would when operating the keys.

What color is it?

The "*" (star, asterisk) key toggles the talking color identifier on and off. This mobile color recognizer for blind and color-blind people speaks the color of whatever shows at the center of the camera view. Note that results of color recognition depend on ambient light and camera quality. Recognized colors include (dark, normal, and light) red, green, blue, cyan, yellow, orange and magenta, as well as combination colors such as red-orange. Black, grey and white are also identified, bringing the total number of identified colors and shades to 47. Further options exist for color filters, even including a filter for detecting exposed skin, e.g., for face detection, for detecting people or to help find empty seats. The color filters can also be used by blind and color-blind people to pick wires of a user-selected color, e.g., to distinguish red or yellow wires in electronics. Beware that the choice of color names can be culturally biased: cyan is a color in between green and blue, while magenta is basically the same as the color purple. Also, light-magenta and light-red make for the color pink or very similar colors, while dark-orange appears as a shade of brown. Dark yellow-green makes for olive-green.

Reconfigurable

A number of other key commands are available for changing settings. The "0" key toggles the muted state. The "1" key toggles the negative video mode, which can help to see/hear small or thin dark items on a bright background. The "7" key toggles a mode that helps prevent visual sound stuttering and buzzing on devices that cannot handle simultaneous visual sound rendering and playing. The "9" key toggles between different contrast enhancement modes. The "#" (pound, hash) key toggles between different sound volume levels when not muted. The default sample rate is 16 kHz, but lower sample rates can be selected by using the "DOWN" key (joystick down), and higher sample rates can be selected by using the "UP" key (joystick up). Available sample rates are 8 kHz, 11 kHz, 16 kHz and 22 kHz, but phones need not support all of these sample rates. Lower sample rates give lower sound quality, but may offer somewhat less sluggish responses. The "RIGHT" key doubles the visual sound duration to two seconds, while the "LEFT" key returns to the default one-second visual sound duration. Note that on some devices the "UP", "DOWN", "LEFT" and "RIGHT" keys may be mapped through the "2", "8", "4" and "6" keys, respectively.

More information is available in the The vOICe MIDlet manual (Microsoft Word format). Enjoy!



The table below gives a summary of the available key commands:

The vOICe Key Action Default
0 Toggles muted state Off
1 Toggles negative video Off
3 Toggles speech feedback On
7 Toggles "anti-stutter/buzzing" mode Off
9 Cycles contrast enhancement 100%
* Toggles talking color identifier Off
# Cycles sound volume levels 25%
UP Higher sample rate, up to 22 kHz 16 kHz
DOWN Lower sample rate, down to 8 kHz 16 kHz
LEFT 1 second visual sound 1 second
RIGHT 2 second visual sound 1 second
FIRE [ Flash and] say color Off
00 Mute and pause (low CPU load) Off
** Color identifier, no visual sounds Off
## Toggles blinders (narrow view) Off
r Red-only color filter Off
g Green-only color filter Off
b Blue-only color filter Off
c Cyan-only color filter Off
y Yellow-only color filter Off
o Orange-only color filter Off
m Magenta-only color filter Off
11 or s Skin-only color filter Off
a Analyze colors by cycling filters Off
p Save snapshot picture to memory card

If you seek a screen reader for your mobile phone, you may wish to check out products like Nuance TALKS (formerly SpeechPAK TALKS from ScanSoft), Mobile Speak (Pocket for SmartPhones, MSP) plus MCR from Code Factory, Pocket Hal from Dolphin, ADnota speaker from adnota GmbH, VSpeak from VoiceSignal and Text to Speech for Pocket PC from Digital Future. Compatibility with The vOICe for mobile phones is not guaranteed, although both Nuance TALKS and Mobile Speak have been successfully applied in combination with The vOICe. Beware though that Mobile Speak versions 1.125 and 2.03 reportedly have serious compatibility problems with Java MIDlets. Whenever possible, try before you buy. The user's choice of text-to-speech engine for the screen reader may affect compatibility with The vOICe. The vOICe supports two menu styles: the "Textual" style for the submenus is only advised for old versions of Nuance TALKS, while the "Normal" style is advised in all other cases. Yes, it is still far from perfect...
For best results with synthetic vision, consider The vOICe Learning Edition for Microsoft Windows, running on a subnotebook PC or UMPC.
The implementation of The vOICe MIDlet for MIDP-2.0 and MMAPI compliant devices is on the edge of what is technically possible with the devices that are currently available on the market, especially with respect to limitations in processing power and poor loudspeaker quality. Several compromises between quality and speed had to be made. Devices based on an optimized Java virtual machine and JIT compilers such as Sun's CLDC HI (HotSpot Implementation, also known as Monty VM) may work best, as will future devices based on high-performance processors such as ARM's Cortex-A8. Also, the MIDP-2.0 standard does not specify any mandatory sound file formats, and The vOICe assumes device support for the 8000 Hz, 11025 Hz, 16000 Hz (default) and 22050 Hz mono 8-bit PCM WAV formats. Mobile phone vendors can use The vOICe MIDlet to test and verify the (un)specified multimedia capabilities of their camera phones.

The zip file "vOICeMIDlet.zip" also contains "The_vOICe.prc" Palm Resource (PRC) application file for use with Palm devices (Treo, Tungsten, Zire, LifeDrive, other Palm OS devices), but MIDP-2.0 with MMAPI camera support appears to be incomplete for these devices. You can try it and report (The Treo 600 and 650 reportedly give a "video capture failed" due to incomplete MMAPI support with Java virtual machines available under Palm OS; nothing is known yet about the new "Treo 750").

Although The vOICe MIDlet was developed mainly for the latest generation of Java ME MIDP-2.0 and MMAPI compliant imaging phones, it should also (still) run on the MIDP-1.0 and MMAPI compliant Nokia 3600, Nokia 3620, Nokia 3650 and Nokia 3660 phones. However, a dedicated version for these phones - but without the color reader (color recognizer) - is also available as The vOICe BEB from Blue Edge Bulgaria. The Nokia 7650 cannot be used for lack of camera support with MIDlets (incomplete MMAPI support).



However, before starting The vOICe or during its operation, you may on some phones have to silence TALKS by doing a "talks + clear", hearing "silence equals yes", to avoid that TALKS blocks some of The vOICe (speech) sounds such as the spoken color names during color identification by the talking color detector. Just try what works for you. In case of persistent compatibility problems, you might need to quit your phone screen reader at some point and work with fixed key sequences to start and operate The vOICe. Specifically targetted at colorblind people there is also Tenebraex' eyePilot for camera phones, which performs functions that are similar to The vOICe color identifier.

Mobile TV?
For phones that include RealOne/RealPlayer, such as the Nokia 6600, there is a video clip about the PC version of The vOICe, namely the 2002 G4 TechTV television broadcast, downsized to 176 × 128 pixels (735 KB non-streaming file, tinyurl.com/s832f). With Nokia phones, PC Suite can be used to put the downloaded video clip on the phone.
Similarly, if your phone supports the AMR format, you can use your phone to listen to the 2005 CBC Radio One broadcast titled "See, If You Can Hear This" (podcast: 2 MB audio file in AMR format, tinyurl.com/l9pej).

In case you seek an accessible MP3 player for use with a screen reader, check out Mp3Player from Viking Informatics, or the Symbian Helix Player.



Stereo MIDlet with 3D audio
The vOICe MIDlet supports stereo panning and 3D audio spatialization on suitable stereo-enabled camera phones, such as the Nokia 6620, 6630 (tested OK), 6680 (tested OK), and many others. Note that many Symbian based phones will only give stereo with wired stereo headsets and not with bluetooth wireless stereo headsets, due to operating system limitations: Symbian OS 9 or later is needed for stereo via bluetooth. Options for Mono (default), Stereo and 3D audio are available in the Channels submenu under phone's Options key. Phones lacking stereo capabilities may give distorted mono sound when set to the Stereo or 3D audio mode. Also note that while using a screen reader it may be necessary to first mute The vOICe (with key "0") to hear the screen reader speak all menu and submenu items under the Options key. After changing settings, one can then unmute The vOICe again with key "0".
The vOICe's stereo and 3D audio support is completely independent of the microQ 3D audio engine from QSound Labs (used with Vodafone VFX) and the 3D Audio Engine from Wolfson (formerly Sonaptic), while The vOICe's stereo and 3D audio is optimized for application in immersive synthetic vision rather than for psychoacoustic positional audio realism, and it is designed for use with a discrete stereo headset rather than a surround sound speaker set.