Mapping Gestures to Features

March 12, 2009 by slholmes
Filed under: General

I spent the last several weeks working on a prototype application I’m calling RDFBurner. The purpose of this application is to provide a user interface that allows for the rapid composition of triples. The triples will be parsed and used to compose RDF. The user interface provides four entries:

Domain
Subject
Predicate
Object

The idea is that you type a new Domain in or choose an existing one from a list, then type or choose a subject, predicate and object. Then save that to create a new triple. Below the entries, you’ll see a list of previously entered triples. The intent behind the user interface is to provide instant visual feedback of your work. Seeing the triples laid out in a tabular modular fashion provides you with many clues as to what sorts of triples you would need to describe your particular domain.

Composing Triples with RDFBurner

During the construction of the prototype, I became frustrated with the complexity of the code. Simply developing the user interface, I could see already how the code was turning into a tangled web of gibberish. This is a fairly typical result when coding to Microsoft’s “Event Driven” programming paradigm. The idea of Events letting your program know that some thing has occurred such as a key press or a record selection is, on the surface, a good idea. But in practice, if you’re not careful (and Lord knows I’m careful but I want to see results quickly so I’m not that careful) you end up with a big mess. Crafting OOP classes helps to some degree but generally, the paradigm leads to sloppy difficult to follow code.

So, once again, I began to think about the challenge of managing the complexities of a user interface. I realize that it might be possible to de-couple the intent of the user from the features in the program, thereby adding some much needed programming structure. If a user gestures by pressing a key or clicking a record in a grid, I could simply make a note of it and move on. Later, a timer in the program would wake up periodically and process these gestures. When an event fires, the logic would be extremely simple as the only task is to queue up the user’s intent. Features in the program would be simplified as they could be coded as very atomic local scope processes.

When an event fires such as Control_KeyDown or Form_SelectionChange, create a Gesture object that represents the intent of the user’s action.
Using a map, translate that gesture object into one or more program features invocation objects.
Send an invocation object to each active form/control that supports those features.

The code snippet below shows that during an event, we simply queue up a new Gesture object (that’s what LogGesture does). Presumably, some program feature is going to know what to do with the mouse and keyboard focus so the event framework’s key press is suppressed by setting the KeyCode equal to zero.

 Visual Basic |   copy code | ?  
Private Sub Entry_KeyDown(KeyCode As Integer, Shift As Integer)
Select Case KeyCode
    Case 9 'Tab key
        LogGesture Me.Tag & "_Entry_Tab", CStr(Shift)
        KeyCode = 0
    Case 40 ' Arrow Key Down
        LogGesture Me.Tag & "_Entry_Arrow", "Down"
        KeyCode = 0
    Case 38 ' Arrow Key Up
        LogGesture Me.Tag & "_Entry_Arrow", "Up"
        KeyCode = 0
End Select

Now later, a timer event will fire and process the Gesture objects waiting in the Queue. In my prototype, I have the timer set to 50 msec. This gives the application a nice snappy feeling. From the code snippet below, you should be able to see that the Gesture is translated into one or more Features. Each feature is supported by one or more forms. Each form is notified that the feature needs to be invoked.

 Visual Basic |   copy code | ?  
Private Sub Form_Timer()
' Translate GlobalGestures Into Features
Dim pGesture As Gesture, pFeature As Feature, pFeatureQueue As FeatureQueue, pFormWithFeature As FormWithFeature, pFormsWithFeatureList As FormsWithFeatureList
For Each pGesture In globalGestures ' Process Gestures Loop
    Set pFeatureQueue = TranslateGestureIntoFeatures(pGesture)
    If Not (pFeatureQueue Is Nothing) Then
        For Each pFeature In pFeatureQueue
            Set pFormsWithFeatureList = GetFormsThatSupportFeature(pFeature)
            If Not (pFormsWithFeatureList Is Nothing) Then
                For Each pFormWithFeature In pFormsWithFeatureList ' For every form that supports the feature
                    pFormWithFeature.Form.FeatureQueue.Add pFeature ' queue up the feature - note this a form level object unlike the GestureQueue which is global
                    pFormsWithFeatureList.Remove pFormWithFeature ' We're done so get rid of the form from the list
                Next pFormWithFeature
            End If
        Next pFeature
    End If
Next pGesture
globalGestures.RemoveAll
Set pFeature = Nothing
' Check for any incoming feature requests
For Each pFeature In FeatureQueue ' Process incoming Features Loop
    DispatchFeature pFeature ' Execute the subroutine associated with the feature
Next pFeature
FeatureQueue.RemoveAll
End Sub

Finally, the code above is running in what I consider the “master” form which has it’s own queue of feature requests. The queues here are very simple. The DispatchFeature does nothing but checks the name of the Feature from the FeatureObject and calls the subroutine with the same name.

What about RDF?

Because I am building an RDF composition utility, it makes sense to encode the gesture to program feature map in RDF and use SPARQL to retrieve (parts of) the map at runtime. I’ve not done this yet in my current prototype. The purpose of the RDFBurner prototype is to compose triples that will be converted into RDF - a task which I have yet to program. The gesture to program feature mapping is currently implemented as an SQL Query against my rather simple triple store. Once I start actually generating RDF with the utility, I’ll refactor the translator portion to use RDF and SPARQL.

Inserting an RDF map into the Event logic of a typical application has some interesting connotations:

The resulting RDF could possibly be combined the mapping RDF of other applications, implying the compositions of a type of mash-up that works at the user interaction level.
Events are not the same as Gestures. For instance, when a user presses the up-arrow key while the focus is on the top row of a grid, the Event might be evntGrid_KeyPressed(UpArrow) while the gesture or intent of the end user is to move the focus to some other control above the grid, i.e. gstrGrid_Leave(Up). An Event is hardware/form/control kind of thing. A Gesture may be triggered by an Event but represents what the user really wants to do.
It highly likely that there should be a translation mapping in RDF between Events and Gestures.
Once the mapping is in RDF, it may be possible to infer patterns and sets of patterns, especially if the RDF is coaxed into OWL. We could use such inferred knowledge to generate at minimum a skeleton of the programming logic. Obviously other parts of a given application would make use of RDF knowledgebases and the combination of the classes involved could lead to an Ontology Driven Design.

Tags:

Comments

Feel free to leave a comment...
and oh, if you want a pic to show with your comment, go get a gravatar!

You must be logged in to post a comment.

Articles
- Intro to Semantic Web
- Ontology Design
My Accounts
- Facebook
- twitter
My Contacts
OWL
- CO-ODE
- OWL Editor: Protégé-OWL
Categories
- General
Scott Lawrence Holmes scottlholmes at gmail.com

Philadelphia , PA , 19107 US

semantic web ontologies dot net electronic music second life
This hCard created with the hCard creator.
Scott L Holmes's Profile

Create Your Badge

Search for:
Location
Latest Tweets
- I subscribed to smosh's channel on YouTube. http://www.youtube.com/user/smosh?feature=autoshare 12 hours ago
- I liked a YouTube video -- Ridley Scott on Life In A Day http://youtu.be/kGYACultjCY?a 2010/07/23
- I liked a YouTube video -- Kevin Macdonald on Life In A Day http://youtu.be/C_4uii96xqM?a 2010/07/23
- I subscribed to lifeinaday's channel on YouTube. http://www.youtube.com/user/lifeinaday?feature=autoshare 2010/07/23
Recent Posts
Meta
Tags
AI Amaya Apache Artificial Intelligence C# Dot Net DTD Entities Enumerable GridView HTML Inkscape Jasc Joseki JRuby LINQ MathML miscellaneous mod_jk MVC NaturalOWL ontology OWL Philly.net POR Protege RDF Redland Ruby Second Life Semantic Web SPARQL subquery SVG Tomcat WordPress Xerces
Archives

slholmes.org

Mapping Gestures to Features

What about RDF?

Comments

Articles

My Accounts

My Contacts

OWL

Categories

Location

Latest Tweets

Recent Posts

Meta

Tags

Archives