Wooden-Robot.net http://wooden-robot.net Commonsense AI, NLP, interaction design, web apps Thu, 16 Oct 2008 05:30:51 +0000 http://wordpress.org/?v=2.3.1 en Micro-theory of gaze http://wooden-robot.net/2008/10/15/micro-theory-of-gaze/ http://wooden-robot.net/2008/10/15/micro-theory-of-gaze/#comments Thu, 16 Oct 2008 02:07:16 +0000 david http://wooden-robot.net/2008/10/15/micro-theory-of-gaze/

Gaze (aka “looking at something”) can be automatic (scanning the environment), strategic (getting a closer look at something interesting), and even communicative (indicating interest, anger, or eagerness to cooperate). I’m aiming for rules that would allow not just simulating an agent, but allow an agent to predict or explain another agent.

Following my evaluation of potential inference tools last week, I’ve tentatively settled on using either JIProlog or AmziProlog because Prolog is the only rule language that allows me to be expressive enough, and these two tools allow Java and Prolog to invoke each other (while Prolog is embedded in a Java Runtime (JVM)).

It’s important that the tool run in a JVM because I want outside users not to be restricted to using the same platform as I do, and because I want to use industry-standard libraries for graphics, etc.

The high-level architecture of my system (named ATOM = automated theory of mind) is:

  • A set of Prolog facts for each agent representing that agent’s initial mental attitudes.
  • A set of Prolog rules for each agent representing what mental attitudes cause others under what conditions.
  • A Java object that simulates a physical environment. When the Prolog module infers an “attempt” attitude, this is converted into a call into the Java module to see what follows from the attempt. When the Java module determines a causal change that should be observable to an agent, then agent-specific “percept” facts are injected into the Prolog module and may trigger new inference there.
  • A set of JUnit tests that swap out files for different initial states in Prolog (and different simulated environments in Java) to verify that the outputs match expected values. There may also be unit tests that swap out rules to test predictions about agents that veer from the “norm”.

Because a fair amount of research goes into the formulation of each rule, each rule will be in its own Prolog file accompanied by comments describing the scenarios that inspired the rule (also present as unit tests) and the particular pages in articles that inspired the rules or any changes to them. (This may seem fussy, but a career in reading related research tells me we all need to be much better at providing such “provenance”.)

As mentioned in the summary above, gaze is a good human behavior to start with. But read here for even better motivation: Vertegaal et al, 2001: “Eye Gaze Patterns in Conversations: There is More to Conversational Agents Than Meets the Eyes”.

Some initial generalizations I wanted to capture:

  • The blurriness of peripheral percepts is high.
  • Moving the eyes in the dir of a peripheral percept will change it to a focal percept, and whatever was focal will become peripheral.
  • The blurriness of a focal percept is usually less than if it were peripheral.
  • For intriguing resemblances with high blurriness, one would want less blurriness.
  • For intriguing resemblances in focus, one usually wants to ignore peripheral resemblances that are less intriguing.
  • For surprising peripheral items, one automatically looks even if intending not to.

I want to write rules for these in such a way that, not only could the rules drive the behavior of one agent, but they could also be used by that agent to explain the gaze behavior of another agent that one is focused on. That is, it’s the start of an automated “theory of mind”.


Comparison of tools for rule-based inference http://wooden-robot.net/2008/10/15/comparison-of-tools-for-rule-based-inference/ http://wooden-robot.net/2008/10/15/comparison-of-tools-for-rule-based-inference/#comments Thu, 16 Oct 2008 01:51:58 +0000 david http://wooden-robot.net/2008/10/15/comparison-of-tools-for-rule-based-inference/

Although I started writing pseudo-code rules a few weeks ago, I’ve suspended that while I look for a tool for writing and testing rules so I don’t assume too much about the quality of the rules so far. This post describes what I’m looking for and the tools I’ve looked at. I’m likely to go with JIProlog.


I’m looking for an expert system-type tool that offers these features:

  1. Simple, first-order-predicate-calculus (FOPC) syntax rather than C-style code
  2. Can be embedded easily in Java programs
    • Prefer Java because it’s platform-independent
    • Java has widespread use in simulation research that I might use or partner with
  3. Has a large, active community that might be interested in my work, and from whom I might get tech help
  4. Has good IDE (editor tool) support for auto-completion, syntax-highlighting, breakpointing, etc
  5. Allows for easy changing of any rules governing beliefs, desires, and intentions (aka BDI)
  6. Has a way of tracking what papers/scenarios influenced the conditions and actions of each rule (aka “provenance tracking” in my terms)

This is what I’ve found:




  • Uses a Prolog-like syntax (i.e. fairly similar to FOPC)
  • Has a jEdit-based IDE that allows inspecting the mind of each agent, and history of actions among the agents
  • There is a book with in-depth how-to info (which I’ve ordered)


  • Structured as a “BDI” system, which sounds at first like a big positive, but on deeper inspection seems to mean that it handles beliefs, desires, and intentions in a pre-defined way. I’m not sure yet how limiting this might be.
  • I asked the creators if there was a compelling reason to use it over a simple inference engine (apart from the ability of MAS frameworks to allow agents to run over a network), and they said there isn’t much advantage in that respect.
  • Might not support back-chaining or facts with universally-quantified vars




  • Java based
  • Has its own BDI debugger tool


  • BDI support seems “cooked in” and likely to be hard to change
  • Rules must be expressed in an especially verbose form of xml
  • Might not support back-chaining or facts with universally-quantified vars




  • Java-based BDI
  • Distinguishes between achievement goals and maintenance goals
  • Seems more comprehensive in BDI concepts than similar systems


  • Procedural rather than FOPC-style syntax
  • Seems to have same limits on expressiveness (i.e. fairly restricted compared to FOPC or Prolog) as other forward-chaining expert system tools using the Rete clause-indexing algorithm


Discrete Event Calculus Reasoner


  • The “event calculus” (EC) is a set of rules for dealing with the frame problem (aka “commonsense law of inertia” — things tend to stay as they are unless explicitly changed). The frame problem occurs when a rule system knows a fact was true in the past but isn’t sure whether to assume it’s still true. CCSS is likely to run into the frame problem often, since our systems are likely to be triggered by combinations of mental attitudes, but some of those attitudes may seem “stale” due to being triggered in the past. The “discrete” part in the name is due to assuming a timeline with discrete points rather than a continuous one.
  • Has a book that shows how to encode BDI and OCC in EC


  • Requires integrating Unix env, Python, “PLY”, and one of a small set of SAt logic solvers. This seems too brittle and platform-specific to me.
  • While EC has a good-sized community, this inference engine seems to have a small one
  • Seemingly no IDE support
  • Might not support  facts with universally-quantified vars




  • FOPC-like syntax


  • C-based, so integrations would have to use C, JINI, or system-level calls



[Jess vs Prolog]


  • Clips syntax implemented in Java
  • Has an Eclipse (IDE) plugin
  • Widely used


  • Roundabout support for facts with universally-quantified vars: One must use the special ‘nil’ slot value for such vars. Not sure if this will actually work.
  • Roundabout support for backchaining: Instead of indicating that a particular rule is backchaining, one must commit to making a particular predicate backchainable. This seems like it could have unwanted side-effects.




Uses special jargon — “chunk” is what most other systems call a fact; “declarative memory” (for chunks/facts) is what other systems call “working memory”; “procedural memory” is where production rules are kept (is this an oxymoron only to me?)

Structured very similarly to BDI toolkits — there is a goal buffer and a retrieval/belief buffer, but unlike BDI these buffers are allowed to hold only one fact/chunk at a time.


  • Very popular among “cognitive modeling” community


  • Limiting the size and number of buffers seems unnecessarily limiting — it limits the expressiveness of rule conditions
  • Seems to be no support for universally-quantified vars in facts, especially because vars are not allowed to bind with ‘nil’ in facts
  • Coded in Lisp, and special purpose IDE is very limited but does have a stepper (but perhaps not breakpointing)



Similar jargon as ACT-R.


  • Fair-sized community in AI and Psychology (e.g. John Laird of UMichigan)
  • Formalization of Cohen & Levesque’s teamwork theory exists, STEAM, created at USC by Milind Tambe’s group. (I strongly suspect any team- or cooperation-related work we do will have to build on C&L’s theory.)
  • Java-based
  • Has its own IDE


  • Would prefer Eclipse plugin to specialized IDE
  • Syntax allows for separate stages of “proposing” and “adopting” operators, and a lot of flexibility in conflict resolution. I don’t currently need this, though, and the syntax is somewhat different from traditional FOPC. I think I could migrate to Soar later if needed.
  • Not sure if Soar has negative connotations for roboticists and AI folk, due to its somewhat dogmatic views on cognitive architecture


JBoss Rules/Drools

[IBM tutorial]


  • Very active community (enterprise)
  • Not only an Eclipse plugin but also a browser-based “rule management system” (not sure of its features yet)
  • Java-based


  • Two primary rule languages - one in “natural” but highly constrained English, and one that mixes FOPC with procedural constructs. There is a nascent effort to provide CLIPS/Jess-style language support also.
  • Doesn’t support backchaining
  • Probably doesn’t support facts with universally-quantified vars (unless through a workaround like Jess’)


Prolog in Java [external overview]

PRO: Prolog supports backchaining and facts with universally-quantified vars

  • JIProlog
    • PRO: Java can call Prolog, and vice versa
    • PRO: Available as shareware (free)
    • PRO/CON: Special-purpose IDE
    • PRO: Actively supported
  • JPL
    • PRO: Java can call Prolog, and vice versa. Commonly used with free SWI-Prolog
    • CON: Not actively supported since 2003 (at least not the docs)
  • tuProlog
    • PRO: open source, and appears actively supported
  • JLogic
    • PRO: Free
    • PRO/CON: Special-purpose IDE
  • Jinni
    • CON: Not free
  • SICStus Jasper
    • CON: Not free
  • GNU Prolog for Java 0.1.0
    • PRO: Free
    • CON: Very early release
    • CON: Appears not to be actively supported
    • CON: No IDE


Cyc engine and tools (but not KB) - TBD

Currently can’t run on Vista; will try on XP or Linux soon

RePast - TBD

Swarm - TBD




Targeted at situation-assessment and decision-making applications. Uses Bayesian network as part of its tech.



  • Appears to be not available, except perhaps through a belief net app from Charles River Analytics
  • “On the situation assessment side, one major problem we are confronted with is that the type of belief network currently used in COGENT does not model temporal information explicitly.”



  • Popular in research community


  • Seems limited to numeric-oriented simulation rather than rule-based ones


Prevent HTC phone from switching to Ulaanbaatar time from Singapore/Malaysia time http://wooden-robot.net/2008/10/05/prevent-htc-phone-from-switching-to-ulaanbaatar-time-from-singaporemalaysia-time/ http://wooden-robot.net/2008/10/05/prevent-htc-phone-from-switching-to-ulaanbaatar-time-from-singaporemalaysia-time/#comments Mon, 06 Oct 2008 01:08:51 +0000 david http://wooden-robot.net/2008/10/05/prevent-htc-phone-from-switching-to-ulaanbaatar-time-from-singaporemalaysia-time/ My HTC Hermes (AT&T 8525) keeps switching from the time and zone I set in Settings — in particular, from Singapore/Malaysia’s timezone to Ulaanbaatar’s. I found just one Google match about this problem: http://www.ppcsg.com/lofiversion/index.php/t93410.html, and it indicates the problem also affects the HTC Touch. The solution is to go into the Phone app, Menu, Options, find the Time Zones tab on the far lower right, and uncheck “Automatic change time zone and clock”.

Implementing Calendar Sync via Funambol http://wooden-robot.net/2008/07/21/implementing-calendar-sync-via-funambol/ http://wooden-robot.net/2008/07/21/implementing-calendar-sync-via-funambol/#comments Mon, 21 Jul 2008 20:05:51 +0000 david http://wooden-robot.net/2008/07/21/implementing-calendar-sync-via-funambol/ Sync software for managing contacts, events, etc across devices can be very helpful when it works, and very frustrating when it doesn’t. In my experience, when an error occurs it’s often unclear how to rectify it; and even when there is no error, it’s hard to judge whether it worked because many updates may have been made to my data and it’s unclear whether the updates were correct until I run into an instance of not being able to find a contact, or finding that I have multiple instances of the same reminder.

I was really motivated to find a solution, and recently I had the opportunity because my employer chose to offer a sync service by using the open-source software shepherded by Funambol.com. This article describes how to use Funambol’s software to allow users to sync with a pre-existing store of their event and task data.

Funambol provides a very capable sync platform that gets one past the fundamental technical challenges and will allow you to focus on the usability issues that all current sync experiences have.

What Funambol Provides

The core piece of Funambol’s software suite is the Data Synchronization Server (DSS), comprised of Tomcat 5.x running a webapp that handles the syncML xml-over-http protocol plus a dbms (mysql, postgres, or hypersonic) for state persistence. There is also a webapp (”webdemo”) providing an html interface for managing events and contacts, to be stored in the same dbms, but this article is about using pre-existing storage, so you won’t want to use this webapp or its db tables.

Another major part of the suite is the large stable of plugins for mobile devices. Most smartphones and PDAs (even the iPod) have calendar and contact functionality, where the data is stored in a local file or dbms. Many of these devices also have factory-installed (”native”) sync software, which works by sending a “begin sync” request encoded in syncML wirelessly(*) to a server that knows how to coordinate sync attempts. (*Strictly speaking, a device doesn’t have to have a wireless connection; instead, like the iPod, it might require a cable to a desktop computer, which itself has to be connected to a network unless you’re syncing only with the desktop itself.) For devices which do have contacts or events storage but no native sync client, one can usually find a “plugin” from Funambol. Unfortunately, finding out whether your device already has a sync client can be difficult, since the app might be hiding several folders deep from your main menu.

If you don’t have a sync client, but do have the ability to install software directly on your device, you can look for the plugin at https://www.forge.funambol.org/download/. If you don’t have the ability to install directly, but your device can receive binary SMS messages, then you can signup for a free user-oriented account at http://my.funambol.com, where you can indicate your device type and trigger an SMS containing the plugin software — all you have to do is click on an install link in the message.

There are some surprises in the devices that Funambol does support and those it doesn’t. It does support Microsoft Outlook (via a plugin), allowing data there to be synced with your pre-existing storage even without an Exchange server being involved. But it doesn’t support OS X’s Sync application. It does support the iPhone for syncing contacts only, but not events (because, I’m told, the iPhone SDK doesn’t support access to the device’s calendar storage). It does support the iPod, but only if you can connect it to a Windows desktop (via the iPod’s USB cable, which talks to a plugin from Funambol that you must install); there is no support for using an iPod via a Mac (and since the iPod has no wireless capability, it’s dependent on being tethered to some desktop). Another wrinkle is that some carriers disable the native sync clients on phones they sell, or they block Funambol’s plugins from using the wireless connection unless Funambol has been certified. With new phones arriving all the time, this makes for a moving target.

Two more major pieces of Funambol’s suite are the “sync portal” and the “PIM listener”. These are not available in the free, open Funambol software but can be gotten in the “carrier edition” through agreement with Funambol. The portal is another webapp using the same dbms, providing an http-based API, which can be used to build a website allowing users to indicate their device type and trigger an SMS containing an installable plugin (similar to what my.funambol.com provides).

The PIM listener is useful for users who have several devices that must stay in synch; it notices when one of a user’s devices has been synced, and triggers an SMS to all the user’s other devices (which are known to support such SMSs) instructing them to automatically initiate their own sync to pick up the changes. Obviously, this won’t help with Outlook or an iPod (because they can’t receive SMS notifications), but is a good way to satisfy one’s most-engaged users, who tend to have lots of other toys. I believe that if one updates device A, makes a separate update to device B, and syncs A, then not only will B be auto-synced, but the result of that sync will trigger an auto-sync of A to pickup the new data from B. However, as the number of a user’s devices goes up, the number of auto-syncs may thus increase factorially/exponentially; I know of no tests to see if this leads to significant battery drain or application latency. For a demo, check out the video interview with Funambol’s CEO at TalkTv.

As a last note, although the sync portal allows plugins to be pre-configured to connect with one’s own DSS, I don’t know if it’s easy to configure such plugins for locales other than US English. Also, I believe there is no publicly-available list of all error messages that a user might encounter, for use in one’s own Help page.

How Sync with DSS Works

Once a user has located the native sync client on their device, or installed a Funambol plugin (actually, any syncML client from any software provider should work, since Funambol’s DSS is syncML-compliant), and has gotten a login from the data provider (e.g. mail.aol.com or my.funambol.com) to enter into the client, they are ready to start. Actually, there’s one more step: Make sure the data provider url is correct in the sync client.

When the “synchronize” button is pressed, a “begin sync” syncML message is sent to the data provider’s url along with the user’s login, a deviceId (unique to the device compared to all other devices in the world), and an indicator of what kind of data to sync (i.e., contacts, events, or tasks). The data type is actually mapped to a “sourceUri” in the device — for example, some devices allow syncing events using either sourceUri “scal” (indicating the SIF Event format) or sourceUri “event” (indicating the VCalendar format). The sourceUri values are arbitrary and depend on how the DSS has been configured, but the values given here are Funambol’s plugin defaults. I recommend using SIF formats if possible, since they appear to allow for a wider variety of event data, and are common on Microsoft devices and thus have a large user base and the implicit testing that comes with that.

If the login is approved, the DSS combines the userId and deviceId into a “principal id” and checks its fnbl_last_sync table to see if this principalId+sourceUri has synced with it before; if not, it’s treated as a “slow” (aka full, complete) sync; if it has synced before, it’s treated as a “fast” (aka incremental, partial) sync. For slow syncs, DSS responds in syncML by asking the device to send an id for each data item of the requested type; for fast syncs, DSS asks the device to send only id’s for the data added, updated, or deleted since the last sync (as indicated by the date in fnbl_last_sync). DSS makes a similar slow-or-fast request to the data provider end.

Who or what is this data provider? If you use the DSS’s “webdemo” webapp for managing contacts and events, it’s a set of tables in DSS’s dbms. But in our case, it’s a “calendar server” that serves many other client applications and which we access via the network. More on this later.

For a slow sync, DSS asks the device for the full data behind each data item id, and passes that to the data provider telling it to add it. The data provider must reply with an id for the new item. Similarly, DSS asks the data provider for the full data behind every id, and passes that to the device telling it to add it. The device must also reply with an id for the new item. DSS stores the pairs of {device event id, data provider event id} in db table fnbl_client_mapping, keyed by principalId and sourceUri. (Note: if you reinstall a plugin, the deviceId is likely to change, so the principalId would change, which means that syncing, then reinstalling and syncing again, is likely to lead to duplicates on both the device and data provider ends, because there is no longer a matching principalId in fnbl_last_sync and thus a slow sync is done the second time.)

For a fast sync, DSS asks the device and data provider ends what has been added since the time of the last sync, sends the new items to the other end, and updates the mapping table with the new id’s. It then makes a similar request to both ends for items updated since the last sync, and then for items deleted since the last time.

On the data provider end, DSS makes all its requests through “modules” which must implement a predefined API. But before a sync can occur, the userId and password that a user entered in the device must be authenticated, and each module is responsible for indicating what Java class should do the auth check. Funambol calls such an auth class an “Officer”, and provides a default one that checks the co-installed dbms for a matching login. Since we are focused on using pre-existing data providers, you will want to write your own Officer to connect with your existing auth service for user accounts.

The module API requires implementation of these signatures:

  • beginSync - Called by DSS if the officer indicated successful auth; the principalId and sourceUri are passed in via a context parameter
  • getAllSyncItemKeys - DSS uses this to get data item id’s when starting a slow sync
  • getNewSyncItemKeys - DSS uses this to get data item id’s for fast syncs, to get items added since last time
  • getUpdatedSyncItemKeys - DSS uses this to get data item id’s for fast syncs, to get items updated since last time
  • getDeletedSyncItemKeys - DSS uses this to get data item id’s for fast syncs, to get items deleted since last time
  • getSyncItemById - If DSS wants the device to add or update something, it gets it from the data provider this way
  • getSyncItemKeysFromTwin - The user might have added similar items on both the device and data provider ends; this call gives the data provider a chance to report items it thinks are similar to the given device item, to avoid adding duplicates at both ends.
  • addSyncItem - If the device has an item the data provider should have, DSS uses this to put it there
  • updateSyncItem - If the device has an item that should replace one the data provider has, DSS uses this to put it there
  • removeSyncItem - If an item has been deleted on the device, and a similar one should be deleted from the data provider, DSS uses this to delete it
  • mergeSyncItems - If the module has been designed to resolve conflicting similar items from the device and data provider ends by merging them, DSS uses this to get that merged version as a step before sending the update to the device and data provider ends. (This method is available only if your PIMCalendarSyncSource extends MergeableSyncSource.)
  • commitSync - This is the next-to-last call DSS makes into a module for a sync. It calls it even if a SyncSourceException has been thrown by a previous call, which IMO is a design flaw since it doesn’t follow the traditional semantics of a ‘commit’; for example, if one plans to do all updates and deletions to the data provider via a batch, one would want to do that here — but that allows for loss of data integrity because a related update to the device might have failed.
  • endSync - Always the last call DSS makes into a module

If one wants to provide support for events and contacts using the SIF, VCalendar, and VCard formats, there are several ways to split up the work across modules. One could make a separate module for each combination, responsible for only one format of one kind of data, or have a single module that handles all types and formats. The major constraint on modules is that each can have only one auth Officer. While this constraint didn’t limit our design choices, we did happen to implement contacts and events in separate modules in separate source trees but where both modules would use the same officer code. Having separate trees had the unfortunate consequence of needing to copy/paste the officer code from one tree to the other, putting it in a different package, and configuring the two modules to use these different officer packages. Clearly, relying on copy/paste opens the door to maintenance problems; I recommend using a single source tree and single officer codebase if at all possible.

Putting a Module Together

In this section, we walk through the creation of a module to handle multiple data types and formats, where the data provider is a remote host rather than Funambol’s “webdemo” default.

First, find the source for DSS at Funambol’s downloads site, or via objectweb (e.g. http://cvs.forge.objectweb.org/cgi-bin/viewcvs.cgi/sync4j/funambol/modules/foundation/connector/src/main/java/com/funambol/foundation/engine/source/Attic/PIMCalendarSyncSource.java?hidecvsroot=0&search=None&hideattic=1&sortby=file&logsort=date&rev=1.18&content-type=text%2Fvnd.viewcvs-markup&diff_format=h). You’re looking for these files:

  • PIMCalendarSyncSource.java (and parent PIMSyncSource.java)
  • PIMCalendarManager.java (and parent PIMEntityManager.java)
  • init_schema.sql
  • SIFTaskSource.xml
  • VCalendarSource.xml
  • Funambol.xml
  • PersistentStoreManager.xml

Notice that PIMCalendarSyncSource implements the module API mentioned earlier by making calls into PIMCalendarManager, and PIMCalendarManager manipulates the co-installed dbms on behalf of the “webdemo” default webapp. You should hollow-out the method implementations of PIMCalendarManager and code them so they work for your existing data provider. I don’t have more to say about that part of the job.

The remainder of the work is configuring the module. Look in init_schema.sql at how the fnbl_sync_source table maps an incoming sourceUri (e.g. stask) to a bean file (e.g. SIFTaskSource.xml) that describes how to configure PIMCalendarSyncSource for that sourceUri. The key insight here is that every sync attempt triggers the creation of a new PIMCalendarSyncSource object, and that object manages state between the beginSync and endSync calls. By creating an object for every attempt, we can use the same Java class to handle different data types (events or tasks, even contacts) for different formats. Differences across data types and formats are largely hidden from you by the convenience methods in PIMCalendarSyncSource for marshalling and unmarshalling formats into Funambol data objects (e.g. Event, Task, and their common parent com.funambol.common.pim.calendar.CalendarContent). If you need to know the data type within your module code, you can use PIMCalendarSyncSource’s entityType member (or add your own member field and an entry in the bean files to populate it).

A module can support multiple data types and formats by adding an entry for each to the fnbl_sync_source table.

Be sure to make these changes:

  • Change the package and class name of the ‘object’ element in each bean file to match yours
  • Do the same for the ‘class’ attribute of the fnbl_sync_source_type table in init_schema.sql
  • Note that the values of the ‘config’ attribute of the fnbl_sync_source table aren’t filepaths in your source tree; instead, they reflect where the files will be unpacked from a zip after running bin/install-modules.sh. You need to check build.xml to make sure these files are pulled from the right place during the build.
  • Since you will be using your own solution for user acct management, you need to disable Funambol’s by providing a stub. In init_schema.sql, all entries in fnbl_sync_source_type should reference the stub class for the ‘admin_class’ attribute.

Next, look in Funambol.xml and notice the officer, store, and serverURI properties. The ’serverURI’ value is probably ‘{serverURI}’, which indicates a placeholder that will be filled using install.properties when install-modules.sh is run; using placeholders is a pretty convenient way to get config into your module; you can define your own bean java and xml files, add your values to install.properties, and update install-module.xml to map those properties to placeholders in your bean xml file. You can also use a placeholder for the entire ‘officer’ property and put the xml in install.properties (which is helpful if you’re using different officer classes in different environments).

The ’store’ property is important because it points to PersistentStoreManager.xml, which you will edit for your module. If your module will support fast/incremental sync, it needs its own db storage to keep track of the item id’s involved in the previous sync (aka “anchors”). For example, if the data provider provides a way of asking for all items changed since a certain time, but it doesn’t distinguish between which are new and which are updated, and doesn’t indicate any of the deleted ones, then you need to compare those id’s with the anchors: anchors that aren’t in the list from the data provider should be considered “deleted”; items from the provider that aren’t among the anchors should be considered “new”; all the rest can be considered “updated”.

To store and read anchors, you should create files create_schema.sql and drop_schema.sql (referenced in install-modules.xml) for creating and dropping the table(s) you need; if your module is listed in the modules-to-install property of install.properties when you run install-modules.sh, you will be prompted whether to recreate a table; answering ‘y’ will run drop_table.sql and then create_table.sql (so, you generally want to answer ‘n’ when installing unless you want to force all users to do slow syncs next time). To allow the module to read from and update the anchors, create java and xml bean code where the xml defines the SQL for inserting, updating, querying, and deleting. Add the bean xml filename to the list of such beans in PersistentStoreManager.xml.

Some of the files I’ve mentioned may not be in Funambol’s public source, and I’m sorry for that oversight. But they are helpful folks and I’m sure they would send you some samples.

Lessons Learned

Other than issues I’ve already mentioned, here are some issues you should be aware of up-front:

  1. There is no automated update process for getting the latest info about the world of devices. The sync portal needs this so it can offer support for the latest phones, and so it can keep up with changes in support for existing ones. You need to arrange with Funambol to be sent regular “phone pack” updates.
  2. There is no support for sending a message from the server to be displayed in the client. For errors, the best one can do is set a message in a thrown SyncSourceException, since the message will appear in the client log if it uses log level “error”.
  3. There is no way for a module to trigger a change from fast to slow sync in case, say, an error is found with anchor storage.
  4. The module API requires addItem to return the id that your storage will use for the item. This makes it hard if not impossible to include add’s with update’s and delete’s in a single batch during commitSync(); instead, you may need a synchronous network roundtrip for every call to addItem().
  5. It appears that the SIF Event content type used by Microsoft uses UTC-based times (e.g. 20080704T010000Z for 9amPT on July 4) for all values unless it’s an all-day event, in which case it uses YYY-MM-DD with no timezone indicator.  Funambol’s convert() in PIMCalendarSyncSource leaves these all-day values as-is instead of changing them to UTC format. I didn’t find any guiding principle about when UTC versus local time was used.
  6. The default dbms in the open source version of DSS is hypersonic. If you plan to use mysql, be sure to indicate your engine is InnoDB in your DDL scripts since Funambol’s dev wiki mentions there’s a risk of memory leaks without that.
  7. Since you’re using your own storage, make sure to remove the “webdemo” webapp (and associated db tables) so users don’t stumble across it by accident.
  8. When dealing with com.funambol.common.pim.calendar.RecurrencePattern, note that recurrence types with “NTH” in their name are for use when instance!=0, not when interval>0.
  9. Funambol’s Outlook plugin has a bug where if it is given an event that repeats every N years, it will change it to repeating yearly.
  10. Outlook has a bug where if it is given an event that repeats monthly, that’s timed/not-allday, and whose start time crosses a day boundary when converted to UTC (e.g. 4pm Pacific in summer becomes 1am UTC the next day), then Outlook will move it a day late (e.g. from Friday to Saturday). There is a similar problem when Outlook provides such an event, instead of consuming it.
  11. If an event is timed/not-allday but it has a duration of 24 hours, then the Outlook plugin will change it to an allday event, thereby losing start and end times.
  12. When a repeating event is created, synced, and then some individual dates on each end are modified or deleted (they must be different dates on each end), then on the next sync they will show as dupes. This is because Funambol’s default merge method has a bug…such changed dates are represented as an “exclusions list” in the RecurrencePattern obj, and merge should take the union of the device’s list and the data provider’s.
  13. Most devices don’t support multiple calendars, so Funambol has no special support for them, but your data provider may have many users who have these. As long as your module uses a concatenation of calendar id and event id as the full id given to DSS (to guarantee each item id that DSS has is unique within a user’s set of id’s), there should be no problem.
  14. It’s not feasible to host, say, a contacts module in one DSS and a calendar module in another DSS, and rely on sniffing to route http requests to the correct server. The problem is that the sourceUri is the one piece of info that would allow such decisions, but it’s embedded in the request payload, not the http headers.
  15. If a user did a manual sync at the same time that one of his devices happened to do an auto-sync, I believe there is no mechanism in DSS to detect this and block one of the requests, to avoid loss of data integrity for the user. For example, imagine there are similar events on device A, data provider B, and auto-syncing device C; while A and B are merging, C might update the event on B; when the merge of A and B is done and the update happens on B, it wipes out the info from C.
  16. If you use the carrier edition, be aware that your init_schema.sql isn’t the only file controlling what sourceUri’s the DSS thinks it supports. There are also portal/database/cared-coredb-mysql.sql and portal/database/cared-mysql.sql. Be sure to remove INSERT calls from these for sourceUri’s you won’t support, to ensure the user sees an error saying something like “Remote name not recognized” rather than a generic error.
  17. If you see error “Fatal error creating the PersistentStore object” in your log, a possible cause is having the wrong line endings in the config files of your s4j file. For example, if you build the s4j on a Windows machine and try to run it on a *nix machine, you are likely to see this error.
  18. Trivia item: The second param to the ICalendarParser constructor, a String, represents the device’s charset, not its timezone.

Community Mailing Lists

Good luck!

How to search for a string pattern in a directory tree in *nix http://wooden-robot.net/2008/06/11/how-to-search-for-a-string-pattern-in-a-directory-tree-in-nix/ http://wooden-robot.net/2008/06/11/how-to-search-for-a-string-pattern-in-a-directory-tree-in-nix/#comments Wed, 11 Jun 2008 20:58:38 +0000 david http://wooden-robot.net/2008/06/11/how-to-search-for-a-string-pattern-in-a-directory-tree-in-nix/ grep --include='FILEPATTERN' -rin SEARCHPATTERN * Where: FILEPATTERN is the file name pattern you want to match, for example 'h*' for all files that start with h -r is recursive i is ignore case for SEARCHPATTERN matching n is print out the line number SEARCHPATTERN is what you want to search * is to include all files (which is usually what you use if you use --include) ]]> http://wooden-robot.net/2008/06/11/how-to-search-for-a-string-pattern-in-a-directory-tree-in-nix/feed/ How to avoid the error beep in MySQL client http://wooden-robot.net/2008/04/10/how-to-avoid-the-error-beep-in-mysql-client/ http://wooden-robot.net/2008/04/10/how-to-avoid-the-error-beep-in-mysql-client/#comments Thu, 10 Apr 2008 22:30:51 +0000 david http://wooden-robot.net/2008/04/10/how-to-avoid-the-error-beep-in-mysql-client/ Continuing today’s theme of avoiding ill-considered audio feedback in some desktop tools, the MySQL client plays a beep on errors, and one can’t avoid it even if volume is muted.

To avoid it, invoke the client with a –no-beep argument like this:

“C:\Program Files\MySQL\MySQL Server 5.0\bin\mysql.exe” -u userid -p –no-beep

How to avoid the “breaking glass” sound in TortoiseCVS http://wooden-robot.net/2008/04/10/how-to-avoid-the-breaking-glass-sound-in-tortoisecvs/ http://wooden-robot.net/2008/04/10/how-to-avoid-the-breaking-glass-sound-in-tortoisecvs/#comments Thu, 10 Apr 2008 22:27:12 +0000 david http://wooden-robot.net/2008/04/10/how-to-avoid-the-breaking-glass-sound-in-tortoisecvs/ When an error occurs in the TortoiseCVS client, it plays a “breaking glass” sound that can be pretty annoying. To avoid this, rename or delete C:\Program Files\TortoiseCVS\TortoiseCVSError.wav

Note that the error sound plays even if uses Preferences to set Progress Messages to “Really Quiet” (although I suspect this preference controls the verbosity of text feedback, not audio — a poor labelling choice).

Firebug-like dev tool for Safari 3 http://wooden-robot.net/2008/03/20/firebug-like-dev-tool-for-safari-3/ http://wooden-robot.net/2008/03/20/firebug-like-dev-tool-for-safari-3/#comments Thu, 20 Mar 2008 18:58:30 +0000 david http://wooden-robot.net/2008/03/20/firebug-like-dev-tool-for-safari-3/ Haven’t tried it yet, but at first glance looks pretty useful: http://www.simplebits.com/notebook/2008/03/18/safari.html

EOFException from getBinaryStream http://wooden-robot.net/2008/03/14/eofexception-from-getbinarystream/ http://wooden-robot.net/2008/03/14/eofexception-from-getbinarystream/#comments Fri, 14 Mar 2008 21:07:17 +0000 david http://wooden-robot.net/2008/03/14/eofexception-from-getbinarystream/ If you get EOFException from a call to getBinaryStream on a result set, here are two suggestions:

  • Check that when you write the stream, you aren’t trying to read from the same stream twice. For example, you might try to update a row first, and if that returns 0 rows changed, then you may be trying to do an insert using the same stream that you read from for the update. This won’t work because once the stream is read once, the stream pointer is at the end of the content when you try to read the second time.
  • Try deserializing the stream right after serializing and without putting it into a file or dbms.  If this doesn’t work, it’s likely that your object type doesn’t implement Serializable nor Externalizable.
Great advice from xml.com about XHTML gotchas http://wooden-robot.net/2008/01/07/great-advice-from-xmlcom-about-xhtml-gotchas/ http://wooden-robot.net/2008/01/07/great-advice-from-xmlcom-about-xhtml-gotchas/#comments Mon, 07 Jan 2008 19:11:59 +0000 david http://wooden-robot.net/2008/01/07/great-advice-from-xmlcom-about-xhtml-gotchas/ There’s a great post on xml.com about getting XHTML markup right. A sampling:

  • Any styles defined on the body element should be defined on the html element also (or the style might not work)
  • When using createElement (instead of document.write) to add a DOM element, use createElementNS and specify the namespace as "http://www.w3.org/1999/xhtml"