Emu/Talkbank Signal Display Widget API

Steve Cassidy
SHLRC, Macquarie University

In order to promote interoperability between speech annotation tools we are working towards an abstract programing interface for signal display widgets in such tools. This API is intended to provide all of the required controls on signal displays as well as to provide for feedback from display widgets of things like cursor motion, region selection and playback progress. The API was developed in discussion with Steven Bird of LDC and Claude Barraas of LIMSI.

This page describes a sample implementation of the API in Tcl/Tk using the display widgets provided by our Emu system and the Wavesurfer/Snack toolkits from KTH. Although this implementation uses Tcl, the intention of the API design is that display widgets may be written in or controlled from any language.

Signal Display Concepts

The API makes use of a number of display concepts:

The View is the portion of the current signal that is currently displayed by the widget.
The Region is a subsection of the View which is highlighted in some way. The region is often used as the target of operations such as cutting a signal or creating an annotation on part of a signal.
The Cursor is a locator provided by the widget which indicates a single time point in the signal. This will often be a single vertical line in the display of a time waveform but might have different interpretations in, for example, a video display widget.
A Mark is a time marker maintained by the widget which is associated with a symbolic name. These can be used to place locators on a signal, for example they might be used to mark the locations of label boundaries.

API Procedures

Load signal: Load a signal into the widget; the display need not be updated.
Play start end: Play the signal between the given times.
Stop: Stop playback.
GetView: Returns a pair giving the start and end times in seconds of the part of the signal currently displayed by the widget.
SetView start end: Change the view on the signal to show between the given times (in seconds). This should result in the signal display being updated.
GetGeometry: Return the geometry of the widget as a list of four numbers giving the location of the top left corner and the width and height of the widget in pixels.
SetGeometry x y width height: Set the geometry of the display widget, x and y give the location of the top left corner of the widget.
GetRegion: Return the start and end times of the current region in the widget. The region is a subsequence of the displayed signal which is often highlighted in the display.
SetRegion start end: Set the region in the widget, this should update the display to highlight the region as appropriate.
GetCursor: Return the time corresponding to the current cursor position in the widget.
SetCursor time: Move the cursor to the given time in seconds.
GetMark tagname: Return the time in seconds of the tag with the given tagname.
SetMark tagname time: Create a mark in the widget associated with the symbolic name tagname and display it at the given time in seconds. If a mark exists with this tagname, modify its time.
UnsetMark tagname: Remove the mark associated with the symbolic name tagname.

This API should be implemented by any signal display widget that is to be integrated into a conformant system. In order to provide complete functionality, widgets also need to provide callbacks when certain events occur via their own user interface. These callbacks have the same API as the widgets themselves and should be generated in the following circumstances:

Play
Stop
SetView
SetGeometry
SetRegion
SetCursor
SetMark -- if a widget supports creation/modification of marks

A standard method of initiating callbacks needs to be defined but this is likely to be language and implementation dependent.

The Sample Implementation

A sample implementation using Tcl/Tk, Incr Tcl, IWidgets and Emu is available from the Emu CVS repository with the commands:

cvs -d:pserver:anonymous@cvs.emu.sourceforge.net:/cvsroot/emu login 

cvs -z3 -d:pserver:anonymous@cvs.emu.sourceforge.net:/cvsroot/emu co tbdisplay

Please let me know if you can't access cvs and would like a copy of the latest sources.

The sample implementation is written as an Incr Tcl wrapper around various signal display widgets from the Emu system. In this implementation, a master widget is used to control one or more display widgets. A master supports the same API as the display widgets and coordinates the actions of it's subordinate widgets, for example it might synchronise cursor movement or zooming of the displays. An abstract TBMaster class is provided along with a TBPanedMaster class which displays individual widgets as panes in a larger window.

The TBMaster class supports the manage method which instructs it to manage a new display widget. A simple demonstration program is as follows:


set master [TBPanedMaster .top]

set g1 [wsurfdisplay .t]
set g2 [trackdisplay .t2]

$master manage $g1 
$master manage $g2

pack $master -expand 1 -fill both

$g1 Load /home/steve/kutu21w.wav
$g2 Load /home/steve/kutu21w.wav

$master SetView 0 0

Here wsurfdisplay and trackdisplay are procedures which create individual display widgets. Note that in the current implementation signals must be loaded into the display widgets, this should be coordinated by the master in some way; in Emu the master would need an utterance name and could resolve this into a file name for the kind of data displayed by each widget using the database template.

The master configures each widget with a callback option which is then used by the widget to invoke the API of the master. For example the TBMaster class issues:

$widget configure -callback "$this callback $widget"

and the TrackDisplay widget uses this to inform the master that the region has changed (this procedure is bound to a mouse click in the display widget):

body TrackDisplay::start_region {x} {
    set regionanchor $x 
    SetRegion $regionanchor $regionanchor
    eval $itk_option(-callback) SetRegion $regionanchor $regionanchor
}

Here's a snapshot of the above application showing the wavesurfer widget and the emu track widget displaying the same waveform with a synchronised cursor: