MobileServer preview

If you have been reading these posts lately you know that I am working on getting a peer to peer scenario to work with my remote data access tools. This will allow two PDAs to synchronize data just by getting close to one another within the 32 feet required by Bluetooth.

Right now this is working over Wi-Fi and Bluetooth serial COM ports (no support for sockets in the Widcomm stack, alas).

The thing that has been missing from the picture is the server code. I started by providing a standalone native application to serve a TCP / IP sockets connection. Later I split the server feature and put it in a separate DLL. The latest change was to move all that code to the device server DLL (RemSqlCe.dll) and to expose an API for the sockets server. After adding the serial port server I started writing a very simple managed code class library for users to integrate in their own applications.

This class library allows the consuming application to create a server, configure, start, stop, query its status and dispose of it. The above picture shows an instance of the sample MobileServer application running two servers on an HP iPAQ 2210 serving two different desktop PCs, one via Wi-Fi and the other via a Bluetooth COM port.

Now I have to add Bluetooth server abilities to this little beast and this means supporting both the Microsoft stack and the popular Widcomm stack (that's why I'm using the 2210). The server will be able to expose its services (either COM or sockets) via enumeration in order to make ad-hoc device pairing easier.

I expect to release the device client very soon with some P2P samples ready to run.

SQL Mobile over Bluetooth

Well, not yet what you might expect but I just finished the first version of a serial communications Pipe. This allows you to connect to a Bluetooth-enabled device through a virtual COM port as I just did with my laptop (with Targus USB Bluetooth dongle and the standard Microsoft stack) and my iPAQ 2210 (a very reliable beast with the Broadcom stack).

Nothing automatic here - I just configured a serial link between the devices and wrote the serial code to glue them both. I am ashamed to say that this was not a swift experience because my years as a mobile developer led me to overlook some of the finer points of desktop development, namely overlapped IO. Yes, porting the device code to the desktop did not quite work as I expected. Nothing at all, really. After researching into the overlapped IO mechanism, it was quite easy to set up the internal threading and queueing to simulate a RAPI Pipe over a serial port.

Next steps? 1 - Port the client code to the mobile device so you can access another devices SQL CE / Mobile databases over a Bluetooth COM port. 2 - Add support for the Microsoft stack so a device will be able to identify which partners can serve SQL Mobile data and connect via sockets. 3 - Add support for the Broadcom stack. From what I have seen, there is no sockets support in the Broadcom stack. 4 - Why not infrared?

SQL Mobile Peer to Peer

My first P2P experience with SQL Mobile has just happened: using my iPAQ 2210 with a SanDisk SD Wi-Fi card was able to connect via a wireless network to my i-mate K-Jam, open a local SQL Mobile database and enumerate all tables.

This is especially cool because I am not using revolutionary new technology - I just adapted what I had been using in my products a long time ago. Let's see a little history.

When I first shipped Data Port Wizard almost two years ago, the device component DLL (RemSqlCe.dll) was only a RAPI streamed server. Soon I understood that I could adapt the same stream code and build more transports on top of it and the TCP/IP single-threaded server came about in almost no time. In its first incarnation the server resided in a separate device DLL file, but it was not a very pratical solution. The next release of DesktopSqlCe will have a new version of RemSqlCe.dll with the socket server code included as well as a CF (1.0 and 2.0) sample that shows how to use it. And this is what I used on the K-Jam for the server.

The client uses an adapted version of the desktop's dssce.dll and Primeworks.DesktopSqlCe.dll. (In a future version, these DLL files will change their names to better reflect what they do.) This is how I was able to quickly produce pwdclice.dll (dssce.dll device version) and the new Primeworks.Data.Client.dll (Primeworks.DesktopSqlCe.dll device version). I ported the code for both DLL files in about one day (cool, huh?).

Now, besides being able to open a local device connection I can also open a TCP/IP client connection to a server device. Et voilá: P2P! Shipping in September, but I'm open to send the early builds to anyone willing to test the code.

Three final thoughts:

Why should you want to have a local connection on your device? After all, that is why you make a reference to System.Data.SqlServerCe in your application, right? Well, can you connect to either SQL CE 2.0 and SQL Mobile with the same data provider using the same application code? And can you remotely connect to another device just by changing connection properties? Now you can!

And why not add socket server code to the LocSqlCe.dll, the RemSqlCe.dll desktop counterpart? This would be a very easy way to set up a client server scenario over the internet...

On the issue of pluggable data streams, why not a virtual COM port over Bluetooth as well? Can I sync my sdf with yours via Bluetooth?

Stay tuned!

How do I know the database version of an sdf file?

Have you ever wondered how to detect the SQL engine version of an SDF file? Here is an answer.

DesktopSqlCe redistributables updated

The redistributable files of DesktopSqlCe (LocSqlCe.dll and RemSqlCe.dll) have been updated to version 1.46.1620. This code corrects a SQL command parameter length verification bug.

SQL CE Console Updated

SQL CE Console was just updated to version 1.2.605. Besides supporting SQL Everywhere on the desktop, I corrected a few minor bugs and added timer and row count reporting to both table dumps and SQL command results. Get the updated setup here.

Supporting SQL Everywhere

I have just updated SQL CE Console so that it supports the Microsoft SQL Server 2005 Everywhere Edition CTP. The news bit can be read in this page, where you will also find the links to the downloads.

Adapt your application to exotic screen resolutions

Microsoft has just launched the Adapt Your App site to help all the Mobile developer community to better adapt to the new exotic device screen resolutions. Here you will find all the resources and downloads to help you adapt you app to, say, landscape Smartphone.

SQL Everywhere CTP

In the same day I returned from MEDC Europe, I'm greeted with some great news: Microsoft released the SQL Server 2005 Everywhere Edition CTP! Thanks to Alex Yakhnin for the heads up.

Migrating Sync to SQL Server

The first phase of the Data Port Sync migration to SQL Server (2000, MSDE, 2005 and Express) is over with the port of the Jet preparation code. It is quite interesting to see how Microsoft implements the "same" features in all of these databases. With Jet, one can do everything with OLE DB whereas with SQL Server one must resort to T-SQL to get some of the schema information. As I found out when I was developing Data Port Wizard, SQL Server does not allow you to mix direct table access (inserts) with IDENTITY_INSERT: you have to use an SQL INSERT command. Also, Jet and SQL vary wildly in how they manage IDENTITY columns: Jet makes it a piece of cake, SQL is a bit harder especially when you have to know the current seed. Finally, I found out that there is no way to create the equivalent of a ROWGUIDCOL column in Access through a SQL CREATE command - you do have to go through OLE DB.

Having said all this, I am not sure that this will be the final preparation code for SQL Server. As a matter of fact, with a more advanced database engine I will ba able to use triggers to flag any changes to the tracked tables (or so I hope...).

Data Port Sync Released

Version 1.0.500 of Data Port Sync has been released. I will be adding more samples to the code to better illustrate what is an arguably complex piece of software. If you want to take a look at how the whole process works, I advise you to start by the JetSync sample (only available in C# but I'm porting it to VB). This sample does the whole synchronization process for one database:

1. Prepares the Access database for synchronization by adding the tracking columns and tables
2. Exports the database to the device in SDF format (either SQL CE 2.0 or SQL Mobile)
3. Tracks the changes on both ends and
4. Merges them according to a very simple rule: the Access database always wins

There is another C# sample (Prepare) that you may use to prepare and "unprepare" Access databases and I have used it together win an end-user sample that I am also shipping (this time it's a VB sample that I will port to C#) that does steps 2 to 4 and is supposed to act as a template for a custom solution.

All thoughts and suggestions are most welcome for this product.

Standalone device emulator

Good news: the standalone device emulator is here!

Synchronization stability

Data Port Sync is now undergoing a testing phase and although things are looking bright, I had overlooked one important detail: what happens when something fails?

There are four phases to the synchronization process: 1) preparation of the Access database; 2) exporting the Access database to the device in SDF format; 3) tracking changes on both ends and 4) merging the changes. When I designed the tracking code I assumed it would be okay to immediately mark the changes in the database, both in the user table and in the tracking table. As a matter of fact, this is a reasonable solution if you just want to track changed data, but it is not very safe if you need to process this information subsequently. If that process fails, you essentially lose the tracking information (at least with the way I was doing it).

To solve this issue, the data tracker now makes no change to the database. It generates a list of changed rows (for both databases) and the generation of this list is only affected by the posterior changes made to the database. The importance of this is that if the merge process fails, I can roll everything back and rerun the data tracker. All changes will be there.

Data Port Sync will become a product by the end of this month, before I go to MEDC Europe in Nice. Stay tuned.

SQL Mobile DateTime parameters

Looks like I have found another small issue with SQL Mobile. I found out about this after a customer reported not being able to run a parameterized INSERT command with a DataTime parameter through DesktopSqlCe. The OLE DB provider reports the DateTime value as having 8 bytes when it in fact has 16. The ADO .NET provider seems to work correctly.

Preparing for SQL Everywhere - II

After figuring out how to make SQL Mobile work on my XP PC (I have VS 2005 installed so the license works), I decided to port to the desktop the device component DLL I use in all my products. Most of the code (like 99% of it) was immediately reused, barring the occasional precompiler tweaks. The major challenge was not to change too much on the support code and this implied simulating a RAPI or TCP/IP connection and the corresponding data streams.

The solution I came to was quite simple: two asynchronous data buffers. The first is written to by the client code (dssce.dll) and read by the server (LocSqlCe.dll). The second data buffer is used in the exact reverse way.

When I finally managed to make the whole thing work I was amazed by how slow it was... The data streaming code was the culprit because it was chewing up too much CPU bandwidth in its own thread, especially while waiting. The solution? A criteriously inserted Sleep(0) on the reader thread.

The bottom line is that DesktopSqlCe users may now develop their applications against a local SQL Mobile database. Look Ma: no PDA!

Preparing for SQL Everywhere

Since Microsoft announced that SQL Mobile would run on all desktop platforms under the "SQL Everywhere" name, I have been wondering how easy it would be to set up my development PC in order to start testing the native OLE DB interfaces.

Well, it's easy enough. First, make sure you have Visual Studio 2005 installed (one of the versions that carries the SQL Mobile bits). Second, go to the C:\Program Files\Microsoft Visual Studio 8\Common7\IDE directory and copy all the sqlce*.dll files to your C:\Windows\System32 directory. Third, register the sqlceoledb30.dll file using regsvr32.exe Finally, open the ssceoledb30.h file and copy the SQL Mobile definitions to a header file of their own.

After this very simple routine, I was able to quickly write a desktop C++ application to create an SDF file on my desktop using code borrowed from eVC3/4! You just need to include your new header file after atldbcli.h and you are done... I will give all the details in an upcoming article.

Ah, and don't forget: this only works on development machines with VS 2005 and Windows XP Tablet PC edition. These license limitations should be lifted when we get the Everywhere bits by the end of this calendar year.

SQL Server Everywhere Edition

Paul Flessner has just announced what many of us have been asking for: SQL Mobile on desktop PCs. Read all about it here.

Synchronizing multiple PDAs - success!

I write this post after my first successful synchronization of one Access database and two PDAs with SQL Mobile (an iPAQ 2210 and an i-mate K-Jam). The sync code now recognizes each client through a GUID value that is stored by the client PDA. IDENTITY space partitioning is working and all PDAs are generating non-conflicting negative IDENTITY values where the Access database generates positive ones. All of this can be configured by the user or consuming application.

My final hurdles are to implement conflict rules and constraint-directed updates. Right now, there is only one conflict rule: if both the PC and the PDA have updated the same row (as identified by a GUID), the PC wins. As for constraint-directed updates (this is a pompous term that means "insert, update and delete in foreign key order") I have already devised a means to calculate an "INSERT order", the order by which it should be safe to insert new rows without FK clashes. It so happens that it is the reverse order for DELETE.

Lots of tests ahead...

MEDC Europe 2006

I am going to MEDC Europe 2006 - are you? This is the place to be if you want to stay ahead on Windows Mobile technologies. As the banner says, register now for an early-bird rebate. See you in June!

Synchronizing multiple PDAs

So far, the Data Port Sync preview has already some synchronization capabilities. It will only work safely with one PDA and still has no collision handling code. When the same row is updated on both ends the PDA prevails, but this will change soon as collision-handling policies become available.

The day's work was spent on calculating INSERT order for tables (reverse of DELETE) so that database structure and constraints are not violated. Also, work is under way to support multiple PDAs synchronizing against the same database.

Hope that by the end of the week I will have more to write about.