Native Mobile

New blog

2010-04-28T13:57:00.004+01:00

I am moving all my blogging activity to a new roof, where I will discuss more than just native code on mobile devices. Please follow me on the Smiling Smith.

Windows Mobile 6.5.3 Developer Tool Kit

2010-02-17T09:32:00.002Z

You can now download the Windows Mobile 6.5.3 Developer Tool Kit from this link.

Windows Mobile 7 is here

2010-02-15T15:29:00.001Z

Actually, it's "Windows Phone 7 Series". Go have a look at the Windows Phone team blog!

Gesture magic

2010-02-04T10:40:00.001Z

The February edition of MSDN Magazine is now out with a very interesting and informative article on Windows Mobile 6.5 gestures: Gesture Magic. A must read, I would say...

Not yet...

2010-01-26T09:17:00.003Z

The Windows Mobile 6.5 SDK I announced was removed due to some issues (apparently it was prematurely uploaded). Here's the story from the::unwired. Apparently we will have to wait a bit more before te final version is released.

Windows Mobile 6.5.3 SDK

2010-01-24T13:28:00.002Z

The new Windows Mobile 6.5.3 SDK is now available for download here.

New development certificates

2010-01-14T09:22:00.003Z

Here I am, back from the December hibernation (not a vacation) with a renewed MVP award. Thank you for that, Microsoft!

It's time to get this blog rolling again, so here's a heads up for you:

New Windows Mobile development certificates

An alternative way to create the menu bar

2009-12-06T23:12:00.005Z

I don't like resources. I really don't. I can understand their usefulness for features such as string tables or embedded icons, but when it comes to the Windows Mobile menu bar definition, I just hate the whole thing. Take a good look at how the CrypSafe sample app's menu bar is defined:


ATL_IDW_MENU_BAR SHMENUBAR DISCARDABLE
BEGIN
IDR_MAINFRAME,
2,
I_IMAGENONE, ID_ACTION, TBSTATE_ENABLED, TBSTYLE_AUTOSIZE,
ID_ACTION, 0, NOMENU,
I_IMAGENONE, ID_MENU, TBSTATE_ENABLED, TBSTYLE_DROPDOWN  TBSTYLE_AUTOSIZE,
ID_MENU, 0, 0,
END

Intuitive, right? Incidentally, this resource is defined in the .rc2 file and the ATL_IDW_MENU_BAR identifier is used as a default parameter of the CreateSimpleCEMenuBar method of the CFrameWindowImplBase class template. So what do you have to do in order to change the menu bar? Gasp!

There is a better way, fortunately. A way that you can encapsulate and that makes the whole process a bit more transparent.

You create a Windows Mobile menu bar by calling the SHCreateMenuBar API typically from your main window's WM_CREATE handler. As you can see the function takes a single parameter, a pointer to a SHMENUBARINFO structure where you define the menu bar. There are two ways to create a menu bar: you either specify a menu bar resource ID or you provide your own HMENU by specifying the SHCMBF_HMENU flag. This is in fact a much more palatable alternative because you can declaratively create your menu bar in a single location of your application's code. Using my beloved WTL, here's how the code would look like on the main frame's OnCreate handler:


if(m_mainMenu.CreateMenu())
{
  m_mainMenu.AppendMenu(MF_BYCOMMAND  MF_ENABLED  MF_STRING,
                        ID_BACK, _T("Back"));
  m_mainMenu.AppendMenu(MF_BYCOMMAND  MF_ENABLED  MF_STRING,
                        ID_MENU, _T("Menu"));

  SHMENUBARINFO mbi = { 0 };
  mbi.cbSize        = sizeof(mbi);
  mbi.hwndParent    = m_hWnd;
  mbi.dwFlags       = SHCMBF_HMENU;
  mbi.nToolBarId    = (UINT)(HMENU)m_mainMenu;
  mbi.hInstRes      = ModuleHelper::GetResourceInstance();
  mbi.nBmpId        = 0;
  mbi.cBmpImages    = 0;
  mbi.hwndMB        = NULL;

  BOOL bRet = ::SHCreateMenuBar(&mbi);
  if(bRet != FALSE)
  {
      m_hWndCECommandBar = mbi.hwndMB;
      SizeToMenuBar();
  }
}

The m_mainMenu variable is a CMenu, of course. The advantage of this approach is that you create the main menu bar in a very explicit way and if you later want to change it you know where to find the code, and stop guessing what that crappy SHMENUBAR resource declaration means...

CodeProject

A simple header button

2009-12-01T19:33:00.003Z

Now that we have a header for the application, what about adding some useful stuff to it, like buttons? The picture on the left shows the second version of the touch header in action with a "back" button that implements the same functionality as the menu bar's "Back" menu. This button should only be visible when there is somewhere to go back to, so we should be able to add it and remove it as we please.

For now, the header button is a very simple object containing up to 3 CImage objects (for the normal, depressed and disabled states). This is not a regular Windows button but instead a "sensitive area" of the header that just behaves like a button (just like what happens with regular toolbars).

The button itself is implemented in the CTouchToolbarItem class (see sample code below). It derives from CRefCount so you can use the CRefPtr smart pointer class template to manage it and forget about deleting... The class contains three CImage instances, one for each possible state (these will later be changed to smart pointers as well in order to allow for better reuse).

Buttons are set through the virtual SetupTouchHeader method in each of the child views. Instead of just setting the header text, you can now set up to two buttons (left with index zero and right with index one). The button activation messaging is very similar to a regular menu: the WM_COMMAND message is sent to the parent window (the main frame) with the command ID as the wParam parameter.

A final thought: the header belongs to the frame window and is set up by the child view when they are shown. Is this a good design? I'm wondering if these should "belong" to the child view instead and be replaced along with it, possibly with some fancy animation. What do you think?

Sample code: CrypSafe08.zip (406 KB)

A basic header

2009-11-15T12:07:00.004Z

In this post I'm adding a very simple header to the CrypSafe sample, as you can see from the picture. Graphically it is a simple gray gradient with some text printed with an embossed look. Despite of the sleek look, the implementation is quite simple, as you can see from reading the CTouchHeader class source code (see sample source below).

The header is implemented as a simple WTL header window. The WTL protocol for headers and footers is quite simple: the frame window sends an empty WM_SIZE message (both width and height are set to zero) to the header and footer signaling these to position themselves in the main frame client rectangle. The remaining space will be occupied by the child view window.

Implementing this protocol is quite simple: handle the WM_SIZE message and reposition the header at the top of the main frame:


void CTouchHeader::OnSize(UINT nType, CSize size)
{
   if(size.cx == 0 && size.cy == 0)
   {
       CWindow wndParent(GetParent());
       CRect   rc;

       wndParent.GetClientRect(&rc);
       rc.bottom = rc.top + m_cyHeader;
       MoveWindow(&rc);
   }
}

The m_cyHeader variable contains the height in pixels of the header. Now, you just need to paint the header with your favorite gradient. Painting the text with the embossed look is also a simple task: you just need to paint it twice in different colors, offsetting the second painting by one or two pixels down. You get the best effect by painting first with a darker color.

As of this implementation, you don't get a very smart header control: it just paints the text it is given. However, we can make this text change according to some application context just like when you switch views. In the sample code below, the header is always present between view switches and the CChildViewManager merely asks each CChildView that is brought into view to update the header text through the virtual SetupTouchHeader function.

Now that this is in place, we can add more features to the header and make it more useful for the whole UI...

Sample code: CrypSafe07.zip (345 Kb)

HOWTO: Close Application on Minimize

2009-11-06T08:56:00.003Z

This is a popular question lately: how do I close my application when the user clicks the "smart minimize button"? This button does exactly that - it minimizes your application and does not close it. Your main application window is minimized when it receives a WM_SIZE message with the SIZE_MINIMIZED constant in the wParam parameter. All you have to do is call PostMessage(WM_CLOSE) and you are done. Here's a sample WTL handler:


LRESULT CCloseOnMinFrame::OnSize(UINT /*uMsg*/, 
                                 WPARAM wParam, 
                                 LPARAM /*lParam*/, 
                                 BOOL& bHandled)
{
    if(wParam == SIZE_MINIMIZED)
    {
        PostMessage(WM_CLOSE);
        bHandled = TRUE;
        return 0;
    }
    // Not handled here
    bHandled = FALSE;
    return 1;
}

Article about Windows Mobile 6.5 Gestures

2009-11-01T11:37:00.001Z

There is a new article on CodeProject about Windows Mobile 6.5 Gestures, and it's fully written in native code. Go there and have a look!

HOWTO: Implement a text ticker

2009-10-26T22:41:00.003Z

Here's the answer to yet another question asked on the Windows Mobile Developer Center Forums: How to implement a text ticker. The request implied for a flicker-free implementation, so a memory bitmap is required. This bitmap is generated whenever the screen changes size (CTextTickerView::OnSize) or when the text itself changes (CTextTickerView::SetTickerText). To calculate the text size when printed with the given font, a call is made to CDC::GetTextExtent which returns both the height and the width of the text in pixels. The off-screen is then created using the screen width and the text height plus a few padding pixels (CTextTickerView::ResizeBitmap).

Scrolling of the ticker is achieved through a timer that shifts the ticker printing position every 100 ms. The ticker is painted by invalidating its own rectangle and the whole painting process is performed on CTextTickerView::OnPaint. Note how the ticker text gets painted a second time tailing the first instance in order to give a continuous feel. Also note how the x offset is incremented by the timer handler. Go and have a look at the code!

Sample code: TextTicker.zip (114 KB)

HOWTO: Show the Today or Home screen

2009-10-26T13:42:00.003Z

I recently answered this question on an MSDN forum: how can I show the Today screen from my application? The answer is actually quite simple:


HWND hWndDesktop = GetDesktopWindow();
SetForegroundWindow((HWND)(((ULONG) hWndDesktop) | 0x01) );

The first line of code retrieves the "desktop" window handle (the Today screen) and in the second you set it to the foreground. Easy.

The Hold Gesture

2009-10-19T19:42:00.003+01:00

After writing my last post, I started wondering if I could implement other useful gestures to control the touch list. So I started to wonder what kind of gestures would be needed for a real list-based application, like the CrypSafe prototype I have been working on. One of the issues that you will be confronted with is the requirement of, somehow, single out a single list item and then apply some sort of command to it. Think about deleting a list item: wouldn't it be nice that by clicking the item in a special way you would be able to do just so? There are a lot of other actions you might want to perform in a list item this way (edit, move up or down, you name it). Microsoft actually solved this issue some time ago with the "tap-and-hold" gesture, but gave it a very specific implementation with those circles rolling around the screen to denote it. Alas, I don't like it so I decided to reinvent the wheel and tweak the CTouchGesture in order to implement this gesture. It's actually quite simple: if you press an area of the screen and you don't move or release the finger for some time, a hold gesture is returned by the timer event.

To illustrate the use of this gesture, I implemented a very simple demo based on AppStart (yes, I was lazy) a so that it shows a "context toolbar" when you hold a given list item (see picture). In this case I added an "edit" and a "delete" button that you could use to change the list item. To make the whole thing a bit more fun, I added a small animation to the hold gesture where you can see the item expanding to reveal the "toolbar". Getting information about the pressed toolbar button would be a simple matter of overriding the HitTest function. Note that when the toolbar is displayed you can still flick and click because the item merely changed its size.

Here is the sample app: AppStart05.zip (268 KB)

Ooops!

2009-10-18T16:48:00.001+01:00

Please discard the AppStart03.zip file as the project contains a bug. The corrected version is here: AppStart04.zip (169 KB).

Minor touch improvements

2009-10-18T11:37:00.002+01:00

In this blog post I'm returning to the touch window code to show you a couple of improvements that were needed.

First, there was the issue of stopping the list halfway through a scroll. If you start dragging the list and then stop, wait a little and then release the finger or stylus, the list would still scroll. This is not an intended (nor intuitive) behavior: the user scrolls the list up to a position and he / she wants it to stay put by stopping the finger / stylus for a fraction of a second and then releasing it.

The second issue happened when the list was "bouncing back" if you forced it to scroll down from the top (or scrolling up from the bottom). Under some occasions you could actually stop the list from scrolling back to its "rest" position leaving it in an awkward state.

To solve the first issue I changed the bulk of the code in the CTouchGesture class and turned it into a state machine with four inputs:

Press - The user pressed the finger / stylus
Move - The user is dragging the finger / stylus
Release - The user released the finger / stylus
Timer - Called every 200 ms in order to check transient conditions

Each input is implemented by a method with the same name and the novelty here is that each of the first three returns a value of GestureType (NoGesture, ClickGesture, FlickGesture). The return type is used by the CTouchWindow mouse event handlers to determine what to do (nothing, a click or a flick). When flicking through the list, the CTouchGesture returns the delta movement through the new GetDelta function.

By implementing the gesture recognition as a state machine we get a much better chance of determining and handling the harder to detect situations, such as:

Input noise such as when the user flicks and accidentally releases the screen for a very short period. This is now handled by the code and the state machine assumes that the previous flick is still happening.
Intermediate motion stop, such as when the user is flicking and stops halfway. This situation is detected by the timer event that essentially resets all displacement and time accumulators and assumes that the user wants to do nothing (this is not interpreted as a click, of course).

Finally, I changed the code a little bit so that it does not allow the list to be in an "unnatural" position. Whenever the list stops it is checked for such a situation (showing the canvas to the top or bottom when the list is larger than the window). When such a situation is detected, the list is automatically scrolled back into position.

To illustrate all of this, I turned back to the AppStart sample simply because it has a bigger list. Here it is for your viewing pleasure (or not...):

Sample code: AppStart03.zip (166 KB)

It's not a bug...

2009-10-11T18:56:00.003+01:00

... it's a feature! Really! I'm talking about my last post where I jumped to the conclusion that I had a bug in a particular line of code. In fact, that line of code is correct - the bug lies elsewhere. Let me explain.

As I stated before, there is no way (at least I never found a way to) use storage object with SQL Compact command parameters. To overcome this limitation the code must allocate the full size of the BLOB data on the binding buffer (which is not very efficient, but it's a viable solution). The problem with this is that the code has to know the size of the BLOB and adjust the binding buffer accordingly. If you use ADO .NET then you know how this issue is solved: the developer must create SqlCeParameters with the given maximum size of the parameter data. The whole purpose of this code is to avoid this situation and let the OLE DB library handle that chore for you.

Command parameter information is requested when you prepare a command and this is the reason why you see the dreaded 1024 constant in the code (the value is pretty arbitrary, by the way). This was the value I chose to represent the default value of the BLOB just to assume a value for it's length. The code then expects that you call SetParam for each parameter and explicitly set its size then. When the command is executed, the whole set of command parameters is looped through in order to get the size of the new binding buffer. If the new binding buffer is larger than the previous one it gets reallocated. And this is exactly where the bug was: the existing buffers were not being deleted and an obvious memory leaked was being created (the complaint reason from the reader who sent me the email message thet prompted me to look at this issue again).

To prove that the memory leak issue is gone in the new code, I set up a very simple demo project where three JPG images are inserted into a sample database table. The images are inserted using a SQL INSERT command and the code is arranged so that the images are processed in an increasing size order. This way you can actually see how the binding buffer increases.

Incidentally, the real bug was on the CCommand::BindParameters method where the m_pBuffer and m_pBindStatus arrays are allocated. These must be freed if non null before allocation.

Sample code: BigParams.zip (789 KB)

Note that the sample application expects both the sdf and jpg files on the \Storage Card\Temp directory. You can change this by editing the BigParamsFrame.cpp file.

Command Parameter Handling

2009-10-08T10:36:00.002+01:00

I recently got an email from an interested reader pointing out an error in the SQL command parameter handling OLE DB Client code. If you remember my previous ramblings on this issue, the OLE DB provider for SQL Compact is unable to handle BLOB SQL command parameter through storage objects. This means that you must provide some information about the BLOB before using the command, just like you have to do if you use the ADO .NET stack. Now, if you look at the CCommand::CreateParameter method you will see the error:

ULONG nParamSize = bIsLong ? 1024 : pParamInfo->ulParamSize;

Let me assure you that I was not taking any medication (or smoking funny stuff) when I wrote this line. This is a piece of code that I just did not return to so the error crept in and started to create lots of memory leaks. Solving this issue is not a simple matter of correcting this line to:

ULONG nParamSize = pParamInfo->ulParamSize;

Like I said, we must specify all the BLOB parameter sizes before we prepare the command and this means that CCommand::Prepare must also change. Instead of getting parameter information in the call to Prepare, the user will have the option of calling another method to get this information. In any case, there must be an option to provide custom command parameters. This is the topic of my next post.

Improvements

2009-09-28T09:07:00.002+01:00

After reading Vincent's comment on my last post, I took a second look at the scrolling code and scuttled the InvalidateRect scrolling mechanism. The fact is that while on some older devices this did work in an acceptable fashion, it did not work that well on more recent models and you can actually see some "bumps" in the scrolling speed. The code now uses a timer (again) but instead of indirectly painting the window through the InvalidateRect / UpdateWindow calls, it now directly invokes the painting function. The result is a very smooth scroll on all devices I have tested the code with.

I also changed the way the ScrollLeft and ScrollRight view transitions work. Instead of painting the bitmap at full speed, I added a very small sleep period to each iteration so that each transition is performed in approximately 250 milliseconds. The first paint operation is timed and a per-iteration sleep period is calculated so that the whole process takes the quarter-second period to unfold. With this improvement, you will be able to see the view transition on all devices (especially the ones with a faster CPU and QVGA screen).

Finally, I decided to add a couple of new view transitions: Fade and Explode / Implode. The first transition uses the AlphaBlend API function to fade from the existing view to the new view (like in a Power Point presentation). I have to say that this is not a very compelling transition, but inspired me to write the second. The Explode / Implode view transition also tries to mimic one of the most famous iPhone view transitions (when you launch an application). The existing view is zoomed in while the new view fades in while also zooming in from half its original size. The Implode effect does the reverse. The result was a bit disappointing because AlphaBlend operation seems to be very greedy (you can replace it by a simple StretchBlt and compare the results). The GDI is not very good at this stuff, unfortunately...

Sample code: CrypSafe06.zip (323KB)

View transitions

2009-09-20T19:03:00.003+01:00

Now that I managed to get some decent finger feedback from the list, it's time to get back to the CrypSafe prototype and add some view transitions. The code I'm using is based on an article I published last year on CodeProject: Animating View Transitions on Windows Mobile. Please read it before continuing as what you will see here is an extension of what I wrote there.

Essentially this article explored the ability to do some fashionable (read iPhone-like) view transitions using child views implemented as regular Win32 controls (such as a tree view, a list view or even a dialog). The view transitions were implemented using a very simple mechanism:

Capture the existing view as a bitmap;
Remove the from the frame container;
Paint the view bitmap on the frame client area;
Create the new child view window off screen;
Enter a loop where the bitmap is scrolled to the requested direction a few pixels;
Every time the bitmap is scrolled, move the new child window to the contiguous position thus creating the illusion that both windows are moving in lockstep.

That was a bit crude but worked. One of the lessons I learned from writing this article is that, generally, you cannot force an arbitrary window to paint itself to a memory DC. This would be very useful for some advanced visual effects. By reading the docs you might even be tempted (as I was) to send a WM_PAINT message to the hidden window with your memory DC in the WPARAM and expect it to paint there, but that just doesn't work for all windows. That's why I had to use MoveWindow for the animation.

Now that we are rendering the touch windows and lists directly on a memory DC, there is a better chance to improve on the original algorithm and implement a smoother view transition algorithm. This is what you will find in the sample code (see below).

The major change I made to the code was to CChildViewBase where you can now see a GetBitmap virtual method. If this method returns NULL, then the child view is not able to render itself on a bitmap and the old brute-force method will be used. If, on the other hand, the child view is able to render itself to a bitmap it must do so here. Note how this is implemented in the CCrypSafeView class: the painting engine is called directly and a new HBITMAP is returned for the child view manager to paint.

The results are here for you to see: acceptable on some devices, way too fast on others and slower on older devices. What we need to do here is quite simple: use the same kinematics equations to control the view transition motion and leave nothing to chance.

Comments and suggestions are welcome!

Sample code: CrypSafe05.zip (349 KB)

Feedback

2009-09-15T22:17:00.004+01:00

I just received a very nice email from one of you (you know who you are) reporting a bug in the OLE DB Client library, more specifically on the CDbValueRef class (file DbValue.cpp, line 758). The line must read:

*m_pLength = nStrLen * sizeof(wchar_t);

The original code did not multiply the string size with the Unicode character size, so you got a wrong byte count and a bug.

It's always very rewarding to receive emails with correction and suggestions from you (it means someone is reading this... - hehe). Thank you for all the feedback!

Better touch

2009-09-13T19:56:00.002+01:00

My professional life changed quite a bit recently and that's why I have not been posting as frequently as usual. I finally seem to have reached some balance and now's the time to resume my blog work.

One of the faults I found in the last version of the published code concerned finger sensitivity. This is a real issue in touch screens based on resistive technology (see a discussion about this topic on Marcus Perryman's blog). The bottom line is simple: resistive screens were made for a stylus, not for your finger. They are much more precise than capacitive screens (like the one found on the iPod Touch and iPhone), but require pressure. Capacitive screens are less precise but require only a very soft finger contact in order to operate. Resistive screens expect a very small pressure area (the tip of a stylus). An adult index finger uses up a very large area on the touch screen and the net result is noise and erratic motion messages.

You can see this in the last sample code I published: flick the list and then click it. It should stop the list but it doesn't (that's how it works on my HTC Touch Pro). What happens is that a single finger press is interpreted by the highly sensitive resistive screen as a sequence of one WM_LBUTTONDOWN message, a couple of WM_MOUSEMOVE messages and the final WM_LBUTTONUP. Inspecting the LPARAM parameter you will see that the POINT structure shows some fluctuations on the x and y coordinates. This means that a single finger press is actually equivalent to a small doodle drawn with your stylus. You will be lucky if the first and last POINT values are the same (they almost never are).

To mitigate this issue and provide better (more reliable) finger action feedback I made some adjustments to the code. The first thing I realized is that one of the sources of "position noise" was the first WM_MOUSEMOVE message. The offset between the WM_LBUTTONDOWN position and the first WM_MOUSEMOVE message seems to provide more variance than the offset to the second WM_MOUSEMOVE. To even things out a little bit I now discard the first offset and start calculating the gesture from the first WM_MOUSEMOVE message. Also, I brought back the concept of sensitivity to filter out smaller movements. Finally take a look at the code in CTouchGesture::Move method. Instead of calculating the move offset in a single if clause, I split the code into two, one for the vertical motion and the other for the horizontal motion.

The final result feels a bit better and more reliable on all the machines I have tested the code with, but the final word is yours.

Sample code: AppStart02.zip (164 KB)

Shaken, not stirred...

2009-09-10T11:20:00.005+01:00

The September 2009 issue of MSDN Magazine presents us with a very nice article on how to detect shaking with the HTC accelerometer. Note that the article is for managed developers, but that should be no problem for you...

Touch Improvements

2009-08-25T10:02:00.003+01:00

The last version of the touch window and touch lists still needed improvements, notably on scrolling smoothness and clicking accuracy. In this post I'm presenting new versions of both classes where I tried to address these issues. The bottom line is that clicking feels much better but scrolling is still a bit jagged. I have to say that in order to get a game-like smooth screen scrolling one would have to choose something like DirectDraw because GDI was not made for this. Anyway, you will be the judge.

First I changed how the window scrolls by removing the timer. How do you force the window to scroll without a timer? Well, it's kind of easy: set up a scrolling mechanism that determines the next position based on the current time (I showed how to do that with the kinematic equations), set up a cutoff point (time and / or position) and just invalidate the whole window after painting. If you just invalidate the window, you allow all other messages to be processed and when the message queue is empty, the OS will send you a WM_PAINT message. If the window is still scrolling, you calculate the new position (based on the current time), paint the window and if the motion has not stopped, you invalidate the window again. This way you get a more natural way of scrolling the window and you know that you will get the best screen update frequency that GDI can muster. Note that in order to implement this in a generic way, I had to implement the OnPaint handler in CTouchWindow. This class delegates to its children through the "static virtual" Paint method.

In this process I had to update the way the kinematic equations are calculated due to some rounding errors I found on QVGA devices. To solve the issue, I changed the main equation back to the more classical form but reduced the timer precision from millisecond to hundredths of a second (divided by 10 in order to get smaller number from multiplication). The rounding error seems to be gone. I have also added a double criteria to test if the motion did end: now the code tests for both time limit and end point.

To help with both scrolling and clicking, I added a new class named CTouchGesture that essentially handles all the data coming from the stylus (mouse) messages and helps CTouchWindow determine if the last gesture was a click or a "flick". Note that for flicking, the class adds up all the partial moves (compensating for changes in direction) and reports an "average" flick speed (total displacement and time) instead of the last move.

Clicking the items is now a bit more reliable because the code tests for clicking eligibility. If the window is moving and you click it, there will not be an OnClick call: you will only get one if the window is still to begin with.

I hope that these modifications did improve the code and the user experience, but with my limited assortment of devices I cannot have a good judgment on that. So please tell me what you think!

Sample code: AppStart01.zip (161 KB)

A simple application launcher

2009-08-19T16:03:00.004+01:00

As I promised on my last post, here is the code for a very simple application launcher. It essentially lists all the applications registered in the "Programs" folder of your Windows Mobile device and launches them when the user clicks the corresponding list item.

I have to confess that I went into a bit of an ordeal in order to implement this very simple application. Quite simply because I tried to outsmart the Windows Mobile shell and to collect all the required information from the .lnk files. If you are on this same path, please read these words carefully: all has been done for you already, so don't try to reinvent the wheel.

My first pain was to get the icons from the shortcuts. You can almost do this by using ExtractIconEx, but this function will not work for some of the system DLLs, such as ceshell.dll. Why? I really don't know why, but the "Search" application icon (the magnifying glass) is impossible to get to by using this function. After asking around a bit, I was steered to the right direction by Christopher Fairbairn: use SHGetFileInfo. The only caveat is that you must use this function against the .lnk file itself, not against the target. If you do so, your icons will show up just right as you can see from the image.

The item name is taken from the shortcut file name - just remove the extension and the path. To make this easier I creted a very simple class named CFilePath that, for the time being, contains only one function: GetFileNameWithoutExtension (now where have I seen a similar function name?).

Finally, in order to execute the application, you just send the full .lnk file name to the ShellExecuteEx function and you are done with it. No need to mess around with the .lnk file format and all of its quirks.

For your viewing pleasure, here is the sample code: AppStart00.zip

Shell Shortcuts

2009-08-17T12:06:00.003+01:00

Windows Mobile shell shortcuts have a very simple file format, something like (taken from my HTC Touch Pro):

22#:MSPIMG?pimgres.dll,-101

The number 22 is supposed to tell us how many characters there are in the string, but this example seems to be wrong. Incidentally, I have seen examples where this value is zero so apparently you can forget about it.

After the # character, you can read the associated command. In this case, you get a "shell macro" that needs to be expanded using a technique that I explained on my previous post. After the ? character there is a module name where the display icon can be found. This section is optional as well as the final section after the comma: the display icon identifier. The negative value means that this is an icon ID and you should be able to feed it directly to the ExtractIconEx API to get the icon handle.

One of the interesting things I've found while doing some research on these files is that you can add a comment section to the beginning of the file, something like:

;Pictures & Videos
22#:MSPIMG?pimgres.dll,-101

Interesting, huh?

I'm putting together a very simple application launcher that uses this knowledge to read from the Program Files directory all the existing shell shortcuts and displays them on a touch list with the right icon and name. This will be published on my next post, so stay tuned.

Default Windows Mobile Applications

2009-08-12T16:09:00.003+01:00

How do you start the default Windows Mobile applications like Calendar, Contacts or Messaging from your application? I recently came across this question in the Visual Studio Smart Device Development - Native C++ Project (quite a long name, huh?) MSDN forum. The answer is quite simple and relies on a registry entry that has been around at least since Windows Mobile 2003:

HKEY_LOCAL_MACHINE\Software\Microsoft\Shell\Rai

Here you find a list of varying size containing all the default applications in your machine (both a human-readable name and the required command line). For instance, on my HTC Touch Pro the calendar is stored in the following entry:

HKEY_LOCAL_MACHINE\Software\Microsoft\Shell\Rai\:MSCALENDAR

The application's human-readable name is stored in the "0" (zero) key while the command line is stored in the "1" key. If you look up this in your device's registry you will probably find something like:

:MSPOUTLOOK calendar

Note that this requires further expansion by looking up the following entry:

HKEY_LOCAL_MACHINE\Software\Microsoft\Shell\Rai\:MSPOUTLOOK

The value of the "1" key contains "poutlook.exe" so the final command line to start the Calendar application must be "poutlook.exe calendar". Easy huh?

Item Activation

2009-08-10T16:57:00.003+01:00

I'm still not sure how this is going to end, but right now items can be focused, selected and activated. An item gets the focus when you use the arrow keys to navigate the list and that is meant to mean the "current" item in the list. When you select an item it gets added to a selection list (in case you need to use multiple-selection lists). Note that focus does not mean selection nor selection means focus. Finally, an activated item is the one you just "pressed" or clicked.

The code I'm publishing today shows how to set the focus on a particular item (just added a highlighter to the DrawItem method) by navigating the list with the arrow keys and how it can be activated by directly clicking it or by pressing the enter key on the focused item.

I had to make some changes to the CTouchWindow class template in order to make sure that scrolling (or flicking through) the list does not get confused with a click. If you hook the application with Remote Spy you will see an interesting pattern of messages when clicking an item: WM_LBUTTONDOWN, WM_MOUSEMOVE (may occur more than once) and WM_LBUTTONUP. What's interesting is that the "mouse move" message always shows up even if it reports the same screen position as the "button down" message. Is this a bug, Microsoft? Anyway, the code already handles null movements so...

When an item is selected, the CCrypSafeView::OnItemChangedState method is called by the base class and to handle the item activation you must handle the TLN_ITEMACTIVATED code. Just take a look at the sample code to see how it is done. Note that you test nmtl.dwState for nonzero meaning activation - it is zero when the item is "deactivated".

Sample code: CrypSafe04.zip (325 KB - now includes the OleDbClient lib)

Compiling CrypSafe

2009-08-03T09:41:00.002+01:00

Back from the sunny Algarve and ready for more work. I have to confess that during these last two weeks I barely touched my laptop, but I surely needed the rest.

Compiling the CrypSafe prototype has proven to be a bit of a challenge for some of the readers and I have received some complaints about this very subject. Here is a list of what you need to do, both with VS 2005 and VS 2008 to get the sample to compile. Please note that I'm distributing the VS 2008 solution files only but you should be able to recreate the project for VS 2005 without major issues.

Here's what you must do to compile the sample:

Download WTL 8.0 or 8.1. You can get the latest version of WTL from its sourceforge page.
Unzip the WTL distribution file to a directory of your choice.
Under the installation directory, you will find some directories named AppWiz, AppWizCE and AppWizMobile. Navigate to each of these in turn and run the setup90.js (for Visual Studio 2008) and/or setup80.js (for Visual Studio 2005) scripts in order to install the Visual Studio application wizards. Please note that under Vista (and probably also under Windows 7) you must open a command prompt with elevated privileges because these scripts write files to the Visual Studio install directories.
Now you have to open Visual Studio and let it know where to find the WTL include files. To do this, open Visual Studio and go to the Tools / Options menu and select "Projects and Solutions" item in the tree. Select the "VC++ Directories" under that item and then select the "Include files" option of the "Show directories for:" combo box. Now, for each platform listed on the left combo box, you must enter your WTL include directory in the list below.
Now you can open the CrypSafe solution in Visual Studio and try to compile it. If it still fails, please make sure you correct each project's "Additional Include Directories" property under the project's Configuration Properties / C++ / General option.

If you still find errors while compiling, please let me know through a comment to this post.

And now for something completely different...

2009-07-21T16:50:00.002+01:00

I just spotted a native code pearl on Todo PocketPC: Nitrogen 1.1 a native code media player for Windows Mobile 5. The interesting thing about this application is that it comes with full source code (licensed under the GNU GPL v3). The media player looks very good and I will be giving it a spin (I really don't like any of the media players bundled with my HTC Touch Pro).

Kinematics

2009-07-11T18:33:00.003+01:00

How do you make the touch list behave like a real physical object? What's the secret behind making the list bounce smoothly back to its original position? How do you decelerate it when you quickly flick through it like if it had some sort of "friction"? In short: how do you make it work like an iPhone list? The answer is simple: kinematics, the study of motion.

In this post I'm adding some kinematic effects to the list to make it work like if it had both weight and some "springs" to attach it to the extremes of the window. These "springs" will make the list bounce back when the list is either showing its first or last element and you force it to scroll down or up (respectively). Also, when scrolling through very large lists, you want them to slow down in a realistic fashion, like a real object would.

There are a lot of changes to the base CTouchWindow class in order to include these kinematic additions. Interestingly, some parts of the code actually became simpler. This base class is used to implement the touch list that only performs vertical scrolling, but it must also accommodate for a possible horizontal scrolling. To make the kinematics implementation easier, I created a helper class named CTouchMotion that is used to calculate the list position as a function of time. You initialize each instance (vertical and horizontal) with the desired acceleration (in pixels per squared second), initial speed (in pixels per second), the initial and final positions of the motion (the final position is redundant, me thinks) and the total motion time. The x and y positions of the motion (just y for the touch list) are calculated by the GetPosition function that takes the time offset (since the beginning of the motion) as milliseconds. The function just computes the classical x = 0.5 * a * t * t + vi * t + xi equation, where x is the position for time t, a is the acceleration, vi is the initial speed and xi is the initial position. Notice that the function must transform the input in milliseconds to seconds in order to get the appropriate result. This is a concern because I must perform all calculations using integers to get the best performance possible. If you look at the implementation of the GetPosition function you will see that I calculate the 0.5 * a * t term first and convert it to a pixels per second speed. Next I add the initial speed, multiply the result by t and then convert the result to pixels by dividing by 1000. Finally, xi is added and we get the proper result. Also notice how each step is accurately rounded to make sure nothing is lost during conversions.

While I was debugging this code I found a very interesting bug. Originally, the function was using the time offset as a DWORD (an unsigned value). Occasionally the list would simply disappear while scrolling. The bug was due to the conversions performed between signed and unsigned integers and sometimes they behaved differently from what I expected, and would throw the list origin to very large negative numbers. The problem was solved by casting the time offset to a signed integer.

You can also see that the function uses a time limit that works as a "break" if you accidentally call it with a time offset larger than the one you specified. Why is this? Simple: the equation is a simple parabola that computes a trajectory until the speed (the first derivative) is zero. This is when the list stops. If you don't limit the motion, you would actually see the list bouncing back with a positive acceleration like when you throw a ball up, and it comes back down. You only want the upward movement - the one where the speed decreases until zero.

Another big change happened in the way the motion timer is calculated. The previous version used the speed of the last WM_MOUSEMOVE (displacement and time). This concept was discarded and I assumed that the movements would have a set of fixed behaviors:

Bounce-back movements would last 500 milliseconds.
List deceleration would take 2 seconds.

You can see these calculations in the CTouchWindow::SetupMotion function. This is called when the list is about to move (on the WM_LBUTTONUP handler) and during the timer (OnTimer) function in order to test for boundary conditions. There is something that's still amiss here: when you transition from a scroll to a bounce-back, the initial speed is not being considered so you see a bit of a harsh transition. I will correct this later.

Finally, and because you can now have an empty space drawn before the list (and also after, but this was already in the previous implementation), I added a fourth color to the list: the canvas color. This is what lies "beneath" the list and helps to convey the notion that the list exists above another surface (thanks for the suggestion, Anatoly).

The code is not fully cleaned up yet and you will see some more changes to this kinematic code. Meanwhile, here is the project for your perusal:

Sample code: CrypSafe03.zip (99 KB)

Windows Mobile 6.5 Gestures documented

2009-07-09T21:03:00.001+01:00

The documentation for Windows Mobile 6.5 Gestures is now online here.

Bouncing the list

2009-07-09T10:03:00.002+01:00

While I was preparing the code for the next post, I realized that something really important is missing from the touch list: it does not "bounce". Use the sample I published in the last post and try to move the list. It will not budge. Similar implementations allow you to move the list out of it's stable position (minimal energy position?) and it will move back to where it was. It gives you a nice sense that you are dealing with a physical inertial entity.

The sample code I'm publishing today just allows for this and simply resets the list to it's stable position without any animation or kinematic feedback. Right now I'm going through some high-school kinematic equations in order to implement some properly accelerated animations to the list. Meanwhile, check the source code here: CrypSafe02.zip. As you can see, the list painting now allows for empty space both before and after the list so that when you force it down it will not leave garbage at the top.

A better look

2009-06-29T11:33:00.004+01:00

Here's a better look for the CrypSafe main view. As you can see I'm using a new font for the item text, added a separation line between the items and painted the background with a different brush for odd-numbered items. The changes to the code are not very dramatic, so let's take a look at them.

The first change is the new font. To use a new font I generally declare a CFont object on the view class like this:

private:
CFont m_largeFont;

This way I know that the font will exist throughout the life cycle of the containing view. Creating the font itself is always a bit of a pain, but i prefer to use the following method because it does give me full control over what is going on. The following code snippet was taken from the view constructor in CCrypSafeView.cpp:

LOGFONT lf;
int nLogPixelsY = DRA::LogPixelsY();

lf.lfHeight = DRA::HIDPIMulDiv(-12, nLogPixelsY, 72);
lf.lfWidth = 0;
lf.lfEscapement = 0;
lf.lfOrientation = 0;
lf.lfWeight = FW_NORMAL;
lf.lfItalic = FALSE;
lf.lfUnderline = FALSE;
lf.lfStrikeOut = 0;
lf.lfCharSet = ANSI_CHARSET;
lf.lfOutPrecision = OUT_DEFAULT_PRECIS;
lf.lfClipPrecision = CLIP_DEFAULT_PRECIS;
lf.lfQuality = CLEARTYPE_QUALITY;
lf.lfPitchAndFamily = DEFAULT_PITCH | FF_SWISS;
_tcscpy(lf.lfFaceName, TEXT("Tahoma"));
m_largeFont.CreateFontIndirect(&lf);

The code at the top is the most important one because it makes sure that we create the font with the right size in points, independently of the resolution of your device's screen.

Now that the new font is ready to use, let's see how to use it on the DrawItem method. First, you need to declare an HFONT variable:

HFONT hFont;

This is used to store the DC's currently selected font while we use our custom font. Setting the new font is easy:

hFont = dc.SelectFont(m_largeFont);

When we are done with the new font, we must set the old DC font back:

dc.SelectFont(hFont);

To paint the text with a left border, I copied the item CRect to a new instance:

CRect rcText(rcItem);

Before painting, the left value is adjusted:

rcText.left += DRA::SCALEX(4);

Note that this will correctly scale in higher DPI screens. As you can see from the included image, the even-numbered items have a white background while the odd-numbered ones have a very light gray background. As I explained in the previous post, the client code in DrawItem is supposed to paint the background, and doing it is extremely easy:

COLORREF rgb(m_rgbBack);
if(iItem % 2) rgb = RGB(0xf0, 0xf0, 0xf0);
brush.CreateSolidBrush(rgb);
dc.FillRect(&rcItem, brush);

The m_rgbBack variable is declared by CTouchListBase and contains the default background color (this color is used to fill in the unpainted space at the bottom of the list). If the item has an odd number, I replace the default white with a light gray and use this RGB value to create a brush. Painting the item's background is a simple issue of calling FillRect with the supplied item CRect (rcItem).

Now we can paint the item's text making sure that it is painted with a transparent background mode:

int nBkMode = dc.SetBkMode(TRANSPARENT);
dc.DrawText(pItem->GetText(), -1, &rcText, DT_LEFT | DT_VCENTER);
dc.SetBkMode(nBkMode);

Note that the text is painted using the rcText rectangle, not the item's. This way we get the left margin effect. Finally we can print the bottom separation line:

CPen pen;
HPEN hPen;
pen.CreatePen(PS_SOLID, 1, m_rgbLine);
hPen = dc.SelectPen(pen);
dc.MoveTo(0, rcItem.bottom - 1);
dc.LineTo(rcItem.right, rcItem.bottom - 1);
dc.SelectPen(hPen);

The m_rgbLine contains the default color for the separation line (gray). As you can see, this is not rocket sience. Next, I will explore how to add a navigation cursor and key handling.

Sample code: CrypSafe01.zip (96 KB)

The CrypSafe prototype

2009-06-26T11:51:00.006+01:00

Today I present the very first iteration of the mobile password safe application. First I decided to name it "CrypSafe". Second, I also decided to post the code in very small increments so that the whole process of using the WM framework becomes clearer (and also to allow me to add changes as I see fit). Third, I'm starting off with a prototype.

The application architecture is actually very simple (as usual, I'm using WTL 8.0):

A main frame window that contains the content views and any decorations
A category view that shows the information categories in the safe
A detail view showing the individual items in each category
An item editing dialog

More items will be added if need arises, but for now I would say that this is enough. When you create a new project using the WTL Mobile Application Wizard, you get a lot of code created for you. In fact, after creating the project you can compile and run (and it will run - just make sure that the WTL include files are reachable for both the C++ and resource compilers). I usually create an SDI application with a child view because it's the most flexible arrangement. You can skip the child view and pain directly on the frame's client area, but this will make it harder for you to use multiple views. Incidentally, the code that flips views does paint on the frame's client area, but that's another story.

After creating your initial project, our first stop is to change the view window (see the source code):

class CCrypSafeView : public CTouchList<CCrypSafeView>, public CChildView<CCrypSafeView>

This means that our view will implement both a "touch list" and will also behave like a switchable child view. Next we need to remove the standard painting protocol and replace it with the touch list's:

virtual void DrawItem(HDC hDC, int iItem, CRect &rcItem);

Yes, we only need this because the list is painted item by item. Remove the OnPaint handler because it is not needed (it's implemented by the base class). I also added a default constructor where I placed the initial list of categories for the prototype:

CCrypSafeView::CCrypSafeView()
{
AddItem(new CTouchListItem(L"Credit Cards"));
AddItem(new CTouchListItem(L"Debit Cards"));
AddItem(new CTouchListItem(L"Mail Accounts"));
AddItem(new CTouchListItem(L"SIM Cards"));
AddItem(new CTouchListItem(L"User Accounts"));
AddItem(new CTouchListItem(L"Web Sites"));
}

Note that these items are not explicitly delete anywhere in the code because their pointers are reference counted (internally). Finally, let's see how to paint the items:

void CCrypSafeView::DrawItem(HDC hDC, int iItem, CRect &rcItem)
{
CDCHandle dc(hDC);
PTouchListItem pItem = GetItem(iItem);

dc.DrawText(pItem-&gtGetText(), -1, &rcItem, DT_LEFT);
}

Note that the PTouchListItem type is a typedef that denotes the reference-counted version of a pointer to a CTouchListItem object.

As you can see from the image, the result is less than satisfactory. The whole list is painted on an off-screen bitmap that is initialized to solid black (hence the black areas of the screen). The bottom of the list is automatically filled with the background color, but not the individual items - you must fill in the background for yourself. We also need to make the item font larger (possibly bold and with ClearType rendering for a better reading experience). As you can see each row has a height of 40 pixels (80 on high DPI machines) in order to accommodate a 32x32 (64x64 on high DPI machines) image. The text will have to be correctly formatted in order to better fill each row.

For the time being, you can take a peek at the first version of the code here.

P.S.: Anatoly, I have not included all of your changes yet.

Back to native

2009-06-24T12:41:00.002+01:00

After a few days struggling with the BlackBerry JDK, JDE and Eclipse, it's time to get back to predictability and safe ground. Boy, have I missed Visual Studio!

Now that I have some readers actively using, criticizing and suggesting improvements to the Windows Mobile native framework (thank you so much, Anatoly!), I think now is the time to start exercising the code with a real application. Contrary to my early plans, I will start with something that I really need on a day-to-day basis: a mobile password safe. Nowadays I struggle to remember all the mobile phone PINs and PUKs, debit and credit card PINs, online service passwords, just to name a few. I know there are some excellent applications out there ready to be purchased, but what's the fun in that? My idea is very simple: use an encrypted SQL Compact database to store all the sensitive data. The user will be able to add, modify and delete both data and structure (each information category will be stored in a different table). The user interface must be very simple:

A main list containing the user's information categories (credit cards, phone PINs...);
A secondary list containing the category details;
A dialog for editing the details

So, after committing to this, let me roll my sleeves and head back to VS. On the next post I will start publishing the first version of the code.

A simple GPS viewer

2009-06-15T11:19:00.004+01:00

Here's a simple GPS viewer application. I wrote it to help me in testing the transition to reference counted pointers in my budding Windows Mobile native (WTL-based) framework. In the process I have learned a few things about reference counted pointers, especially when NOT to use them.

One of the changes I made to the way the CPropertyListView class is used is that it contains pointers to items created on the heap and not on the stack. This means that you now have to create items like this:

m_pGpsFix = new CPropertyString(L"Fix", L"Unknown");

Note that the m_pGpsFix variable is actually a "smart pointer" and you don't have to worry about deleting it in the destructor. Nor does the list, in fact... When each item's reference count reaches zero, the last smart pointer that references the object will delete it (and that's why you cannot have stack allocated items). This is when you should use smart pointers, so when should you use simple C++ pointers? I found out the hard way when adding these items to the list. My original (and right) prototype was:

void CTouchListBase::AddItem(CTouchListItem* pItem)

When I replaced the C++ pointer with a smart one, I immediately lost one of the best features these things have: the ability of a base class pointer to refer to a derived class object. Understanding that I don't know enough about C++ class templates to make this work (not even sure if that's possible - will have to look at Jossutti's book), I decided to fall back to straight C++ pointers. This poses no problem because the method internally will use the smart pointer class constructor to build a new instance using the raw pointer and add it to the underlying array, so the reference counting is respected.

The sample application just starts the GPSID and displays some data from the GPS on a dialog. As you can see from the ZIP file, there are two additional library projects that make up the framework. Note that these are subject to change and do have bugs!

Sample code: GpsView.zip (93 KB)

unresolved external symbol CLSID_ImagingFactory

2009-06-11T14:44:00.005+01:00

Just started out a new sample project that illustrates how to use the WM libraries I'm writing, and got the following link errors:

1>wmfw.lib(Image.obj) : error LNK2001: unresolved external symbol CLSID_ImagingFactory
1>wmfw.lib(Image.obj) : error LNK2001: unresolved external symbol IID_IImagingFactory

This means that the linker cannot find the symbols CLSID_ImagingFactory and
IID_IImagingFactory, required to use the imaging library on Windows Mobile. If you look at the documentation, you will see that a link to the imaging.lib file is required but it will not work (at least it does not work for me).

The solution I found is to force the inclusion of these symbols using a dummy file that I usually name guids.cpp. Here's the content:

#include "stdafx.h"
#include <initguid.h>
#include <imgguids.h>

The initguid.h include redefines the DEFINE_GUID macro that is used by imgguids.h so that it declares a GUID along with its value instead of making an external reference to the same symbol.

Compile and link and the error goes away.

More native memory in WM 6.x

2009-06-11T12:00:00.002+01:00

Here's a very interesting blog post from The Windows Mobile RSS: More Memory For ALL in WM 6.x. As you can see, folks at Microsoft still care about us, native mobile developers.

VS 2010 and Smart Device Development

2009-06-03T18:13:00.002+01:00

This updated link explains it all: Smart Device development is not supported by Visual Studio 2010 Beta 1, but will (somehow) be supported by the final release. Quoting:

"Microsoft is committed to making Visual Studio a great development tool for the mobile device developer, and will deliver mobile device tools for Visual Studio 2010, but cannot share details now. For existing Visual Studio 2008 mobile developers, Microsoft will release a new Windows Mobile 6.5 emulator in the upcoming months that works with the Windows Mobile 6 SDK."

Now, back to work.

Is Native Code Hard?

2009-05-30T15:49:00.005+01:00

If you read this blog then probably your answer is, like mine, no. So what's the difference between native and managed that makes people avoid native code like the plague? What makes people not want to come back? In my opinion there are a few reasons, especially for Windows Mobile native code:

Lack of tools - just try to write a C# application using notepad and the command line compiler and you will see how much tools make a managed developer life easy. There is no comparable level of tool support for native code developers, and this may be linked to the next point.
Lack of comprehensive and cohesive libraries and frameworks - the Win32 API is way too low-level for most uses. It's very nice to be able to use it directly without paying the P/Invoke price, but this comes at a cost of having to write dozens of lines of code to achieve a very simple outcome. The OLE DB and MAPI COM interfaces are a perfect example of this - way too complex to use as they are offered. Just compare the "stock" OLE DB samples with the OLE DB Client library I published here...
Memory management - this is a minor issue for small applications but when the scale starts to grow it becomes a headache. When are you supposed to delete that object? Is it safe to do it now? Are there other objects referring to it?

I have recently been confronted with the memory management issue in a relatively complex application when I found myself wanting to have the memory management of a particular set of objects to be done automatically. These objects are referred to by an arbitrary number of other objects and determining when their life cycle ended was becoming more and more difficult, especially because they were being referred to by different threads.

There are two basic approaches to automatic memory management that have been used for quite a long time:

Reference counting - The iPhone uses a reference counting mechanism to determine when a given object can be removed from memory. The price you pay for this type of memory management is that you have to remember to increment and decrement the memory reference counts. Failing to do this will result in memory leaks. The iPhone uses an optional "autorelease pool" object that kind of protects you from this.
Garbage collection - The Java and .NET platforms both use garbage collection to reclaim the memory of objects that are no longer active. The price you pay for this is that the GC has a mind of its own and stops your code in order to clean up the mess. On mobile devices, if you don't manage your memory by reusing objects and by avoiding excessive allocations, the GC will kill your application performance.

One of the interesting advantages of C++ is that it implements deteministic object destruction: when your statically allocated object goes out of scope, the destructor is called. This behavior allows us to implement classes like CWaitCursor that displays the wait cursor on the constructor and hide it or replace it on the destructor. You just have to create a variable of that type and the wait cursor will be shown while the object is alive.

If we couple this with a simple reference counting mechanism, we can actually get all the benefits of a reference-counted memory management scheme without the pain that is imposed on us by the iPhone: we don't have to remember when to increment and decrement the references. How? The dead-old solution is to use reference counted smart pointers to manage our objects' life cycles.

After performing a search for reference-counted smart pointers, and realizing that my needs were quite simple, I decided to roll my own. The first thing we need is a place where to store the reference counts. Some implementations use a proxy object to do this but, for simplicity sake, I decided to put the reference count right on the object through inheritance. This has the disadvantage of having to make all the reference-counted objects inherit from one class, but it's a simple class that will only add four bytes to the object size:


class CRefCount
{
private:
LONG m_nRefCount;

public:
CRefCount(void) : m_nRefCount(0) { };

void RefIncrement()
{
InterlockedIncrement(&m_nRefCount);
}

bool RefDecrement()
{
return InterlockedDecrement(&m_nRefCount) <= 0;  }   };

Note that I'm using the InterlockedIncrement and InterlockedDecrement to manage the reference count because these methods can be used from different threads. Also, note that the reference count starts at zero (and not at one like on the iPhone) because I want to count how many smart pointers are referring to this object. Let's see how the smart pointer class is defined:


template <class T>
class CRefPtr
{
private:
T *p;

public:
CRefPtr() : p(NULL) { }
CRefPtr(T *pT) : p(pT)
{
   if(p)
       p->RefIncrement();
}

CRefPtr(const CRefPtr& ptr) : p(ptr.p)
{
   if(p)
       p->RefIncrement();
}

~CRefPtr() throw()
{
   Release();
}

void Release()
{
   T* pTemp = p;

   if(pTemp)
   {
       p = NULL;
       if(pTemp->RefDecrement())
           delete pTemp;
   }
}

void Attach(T* p1)
{
   Release();
   p = p1;
}

operator T*() const throw()
{
   return p;
}

T& operator*() const
{
   ATLENSURE(p!=NULL);
   return *p;
}

bool operator!() const throw()
{
   return (p == NULL);
}

T* operator=(const CRefPtr& rhs)
{
   if(this != &rhs)
   {
       Release();
       p = rhs.p;
       if(p)
           p->RefIncrement();
   }
   return *this;
}

T* operator=(T* pT)
{
   if(p != pT)
   {
       Release();
       p = pT;
       if(p)
           p->RefIncrement();
   }
   return *this;
}

T* operator->() const
{
   ATLASSERT(p != NULL);
   return p;
}
};

As you can see each smart pointer takes up the same amount of memory as a regular pointer, so you can safely pass and return it by value from your own methods and functions. This is a very bare-bones implementation and may lack some more sophisticated features, but so far has worked quite well for me. You use it like a CComPtr (I confess that I did look it up when writing this code).

Gynoid

2009-05-29T16:32:00.002+01:00

There is a new article on Code Project by Vincent Richomme where he presents the Gynoid framework for cross-platform mobile development. In the included sample, Vincent was so kind as to use the "touch list" WTL control that I have published in this blog. Thank you Vincent!

P.S.: I'm not a guru, I just love this stuff and write about it... ;-)

Integer Trigonometry

2009-05-23T15:37:00.009+01:00

Adding a live GPS feed to a map means, besides near-second updates, the requirement to display a location marker with the current heading. The Window Mobile GPSID provides the current heading in the GPS_POSITION structure as the fHeading value, representing the angle in degrees between the current heading and true North (zero degrees). So how do you go about drawing a compass like the one in the picture?

The drawing is composed of two simple triangles: red for North and blue for South. The drawing is performed by using the Polygon GDI function that takes a list of points describing a closed shape and draws it using the selected pen and brush. Nothing new here. The interesting challenge is how to draw the image in the different angles without a big performance hit (translated: how to implement fast trigonometry functions).

Current Windows Mobile devices are not guaranteed to have an FPU unit that will do all the floating point math for you. If you use the Compact Framework, you are sure that all floating point calculations are performed in a software emulated FPU, so they are much slower than what they could be when performed with a real FPU. When using native code, you have the advantage of using the FPU if one is available, but the code will fall back to the emulator if the FPU is missing.

My approach here was to skip any floating point processing at all. There are two reasons for this:

Drawing a map on a Windos Mobile device is a resource-intensive operation, especially if you are updating it frequently and you want to spend the least amount of time drawing any decorations on top of it. So a fast implementation is required.
This object requires very simple transformations (rotation plus offset) and the required accuracy of the final result is not very high. In fact, it is somewhat low considering the relatively small resolution of WM device screens. The bottom line is that you can live with a bit of error when drawing your compass (this is not rocket science).

So how do you calculate a rotation transformation? Please refer to a good book on the subject (I'm using this lovely book as a reference: Essential Mathematics for Games and Interactive Applications) and you will see that it all boils down to these two simple equations (transforms a point through a counter-clockwise rotation around the origin):

x' = x cos(a) - y sin(a)
y' = x sin(a) + y cos(a)

Where (x, y) are the original point coordinates, (x', y') are the transformed point coordinates and a is the rotation angle. The first thing you need to do is to "draw" your compass by defining the vertexes assuming that the center of your coordinate system is in the center of the image. In this case, I used the following points:

(0, 32) - North
(-7, 0) - West
(0, -32) - South
(8, 0) - East

The reason why I'm not using (7, 0) for East is simple: the final image looks much better on a discrete screen if it has an odd width. If you look at the sample project you will see that I split the compass into two distinct triangles in order to draw them with different pens and brushes.

Now the interesting bit is how I implemented the sin and cos functions: through a table lookup. Assuming that you can live with an error of one degree when specifying the angle, all you need is a precomputed list of the sin values between 0 and 90 degrees. Using basic trigonometry you can derive all sin and cos values from the first quadrant sin values. If you look at the code, you will see that the precomputed sin values are scaled by 10,000. When calculating x sin(a) the code uses the precomputed value and divides the result by 10,000 to get the approximate result (you can use a higher scale for a higher resolution calculation, but that might not be required for this purpose). You can see the implementation of these functions in the FastTrig.h and FastTrig.cpp files in the sample.

Before we can draw our compass on the screen, we must take care of two important issues:

The original compass coordinates are centered around the origin, so we must transform them further to make it fully visible on the screen;
The screen coordinates of a Windows Mobile device are a bit different: while x grows to the right as we expect, y grows down and not up.

This means that beside the rotation we must apply a translation and a "mirror" transformation. In the sample code, I translate by adding the x and y offsets corresponding to the middle of the screen and then the y coordinate is flipped by subtracting it from the window height. Enough of this talk, here's the sample project: GpsCompass.zip.

On a future post I will update this code so that it uses a double buffer painting method to eliminate flickering and connect the compass direction to the GPS heading.

One GPS, multiple windows

2009-05-20T08:56:00.003+01:00

After publishing the simple GPS class on my last post, I found myself in a situation where I needed the same CGps object to notify more than one window whenever a new GPS event arrived. My scenario is quite simple: while feeding a window with a map, I also need the same CGps object to notify another window where I display some GPS information like the current latitude and longitude coordinates.

The original code supports notifications for one window only so, to solve this issue I could create another instance of the CGps class and have it notify the new window. Although this would work, it would also create an extra thread in the appliction (if you look at the CGps implementation, you will see that it uses a thread to wait on the GPSID events) which is unneeded and consumes more system resources.

I ended up by devising a simpler solution: add multiple window handle - message pairs to the notification mechanism. To implement this I created a new class - CWindowNotifier - that is nothing more than a simple array of HWND and UINT pairs. When the consuming object wants to send a notification to a set of windows, it just calls the appropriate method (Post or Send). If one of the notification windows was closed, the code automatically removes it from the notification list. Using this class is very simple: just call the AddWindow method with the HWND you want to notify and a UINT value containing the message code for the notification. When you want to send a notification use either the Post or Send methods. The Post method uses the PostMessage API so it returns immediately and is the one I recommend for most uses. The Send method uses the SendMessage API, so it works synchronously and waits before all notification windows have responded. You can get the class source code here.

The updated CGps source code is here. Note that this new version uses a new wrapper for the CRITICAL_SECTION objects which you can find here. Enjoy.

A simple GPS reader class

2009-05-15T09:05:00.005+01:00

I've been recently involved in a number of projects that require information from the Window Mobile GPS Intermediate Driver. The usage pattern is quite similar in all cases as the application needs to update a window based on the GPS information stream. There are two possible approaches for this: you can use a window timer and poll the GPSID or you can create a worker thread that waits on the GPS stream to wake up and notify the window through a PostMessage call.

The CGps class (download here) is very simple to use. On your window class just declare a member variable of type CGps, like this:

CGps m_gps;

To start receiving GPS data, just call

m_gps.Start(m_hWnd, WM_GPS);

where WM_GPS is a user-defined window message. You can use something like:

#define WM_GPS (WM_USER + 0x654)

Now you need to handle the WM_GPS message in your code (this will be somewhat different if you are using straight API, MFC or WTL). Note that when handling the message, both wParam and lParam are zero - the message works as a simple notification mechanism to let the window class or procedure know that the GPS updated its data. You read the GPS information quite simply:

GPS_POSITION gpsPos;

m_gps.GetPosition(&gpsPos);

To stop the GPS events you can either call Stop() or let the destructor do that for you when the object goes out of scope and is destroyed.

When implementing this class, I had to work around the now-known GPSID API bug. On some devices, the GPSGetPosition API uses a different size for the GPS_POSITION structure (something that has been explained by John Spaith here). The workaround is simple: just feed the thing with a fake buffer and cast it back to a GPS_POSITION. If you look at the implementation, you will see all this trickery at work. But don't worry - it does work!

Windows Mobile Marketplace now open

2009-05-12T09:52:00.001+01:00

The Windows Mobile Marketplace is now open (but only for US developers?)

AllKeys, documented

2009-05-07T09:15:00.001+01:00

Here is the key capturing API we've all been waiting for since the demise of GAPI: AllKeys.

WTL frames, toolbars and headers

2009-05-06T15:00:00.003+01:00

The WTL CFrameWindowImpl class template uses a very simple approach to manage the client, toolbar and status bar windows. The preset arrangement for these windows is very simple:

a client view that occupies most of the frame's client area;
an optional toolbar window that is placed at the top of the frame's client area and above the client view;
an optional status bar placed at the bottom of the frame's client area and below the client view.

The frame window already knows how to handle the three child windows. Each frame contains three member variables for them:

m_hWndClient for the client view;
m_hWndToolBar for the top toolbar window and
m_hWndStatusBar for bottom status bar

To make the three child windows work together you must create them and assign them to the appropriate member variable. The top and bottom windows must implement a special handling of the WM_SIZE message because this is how the frame tells each window to place itself in its client area and thus calculate the remaining area for the client view. This special message is nothing but a WM_SIZE message with both parameters (wParam and lParam) set to zero. Here's a sample from my production code:


void OnSize(UINT nType, CSize size)
{
    if(size.cx == 0 && size.cy == 0)
    {
        CWindow wndParent(GetParent());
        CRect   rc;

        wndParent.GetClientRect(&rc);
        rc.top = rc.bottom - m_cyToolbar;
        MoveWindow(&rc);
    }
}

Easy, huh?

Intercepting mail messages

2009-04-29T09:03:00.001+01:00

Ever wondered how to intercept an incoming mail message from native code? The Windows Mobile RSS blog explains how: a very interesting read!

SQL Compact and threads

2009-04-27T09:28:00.003+01:00

Do you remember the days of Pocket PC 2002 and 2003? Life was pretty simple with no need for certificates, a single screen size and SQL CE only ran on devices. Then Microsoft released SQL Compact 3.1 and suddenly the small database was also on the desktop. By this time I already had a small set of tools that remotely accessed the engine via a streamed RAPI interface and was confronted with the need to port the code to the desktop.

In its classical form, the code runs in two different processors with an asynchronous link. Porting the code to the desktop, where both the client and the server ran on the same process, meant either a major rewrite or a smart adaptation. I chose the latter and implemented a DMA data stream that is implemented by the "server" DLL on the desktop. The client merely uses a virtual stream that can be attached to a variety of server endpoints, like a RAPI server, a TCP/IP server or even a local server running in the same process (but on different threads).

This has worked quite successfully until last week when I got a report from a customer complaining about a mysterious error. After exporting a very large (and I mean very large) database from SQL Server to SQL Compact 3.5 on the desktop, the database file would end up with over 350 MB of size but zero tables. After repairing the database it would shrink down to the 20 KB empty databases have.

I first suspected of the transaction mechanism because the whole process (table creation, data transfer and final script execution) is executed under a single transaction. After confirming with Microsoft that there was no known transaction size limitation, I got back to the drawing board in bewilderment. To make matters worse, no error was being reported!

The product that caused this issue (Data Port Wizard) has a very simple usage pattern: you set the source and target databases, set some options and run the data transfer process. At the end the application reports success or error and then quits (after user confirmation). As it turned out, the problem happened when the application exited.

When transferring data to a desktop SDF file, everything works under the same desktop process. The SQL Compact data provider runs in its own thread (not on the UI thread) due to the reason I pointed out at the beginning: I wanted to reuse the remote provider code that runs on a different CPU, so I implemented it as running on a different thread with a DMA communication to the client code. When the client application closes, it signals the SQL Compact data provider to shut down and waits a little while for it to reply. If no reply arrives on time, the desktop client proceeds to terminate the application.

Apparently with very large databases, the SQL Compact engine takes a bit longer to shut down. This is in fact something to be expected because there are larger buffers to flush to storage and there is also the auto-shrink feature that runs when the connection closes. When this whole thing is running on a remote device, your desktop client can afford not to wait for the remote provider to shut down because it is running in a different CPU. When running under the same process, you don't have that luxury. If you don't wait for the SQL Compact engine to shut down in its own thread and you shut down the application, you will be preventing the database engine from correctly shutting down altogether. You can imagine the mayhem this causes to your database...

Lesson: if you run SQL Compact on a different thread, make sure you wait until the connection closes before you shut down your application. You have been warned!

Native GDI+ on CodeProject

2009-04-21T17:35:00.003+01:00

Vincent Richomme just posted an article on Code Project on how to use the GDI+ libaries through native code. If you think it's hard to add that extra cool look to your native GDI app, do take a look at Vincent's article!

IE8 and Visual Studio

2009-04-15T12:40:00.000+01:00

If you installed Internet Explorer 8 on your development machine, you may find it impossible to create new C++ projects. Here's an explanation and workaround.

Is IImage caching a good idea?

2009-04-14T17:49:00.002+01:00

On my previous post I described a very simple solution to improve the performance of IImage::Draw - caching the converted image. This does speed up the process of drawing the alpha blended picture but has one drawback: you store the image twice. From what I understood from the IImagingFactory::CreateImageFromBuffer online documentation, the supplied buffer is not immediately used and must be there throughout the IImage object lifetime. So when you cache the image you are essentially storing two representations of the same thing (the original encoding and the decoded version). While this may not be a big issue for small images, this becomes a serious problem when using lots of images and / or larger images. So what can we do about this?

The idea is to have a single 32 bpp ARGB HBITMAP in memory per image. As I showed before, using the shell API to load the images is a bad idea because it shrinks them to 16 bpp. Is there any other alternative to this? We can try to create a 32 bpp bitmap in memory and then use the IImage object to paint it. After releasing the IImage object, we should get a properly-formatted 32 bpp ARGB bitmap. On my next post I will illustrate this technique that was validated by master guru Alex Feinman.

Revisiting the Carousel - II

2009-04-13T19:21:00.004+01:00

I finally found a solution for the speed problem of rendering the carousel icons: caching the image. After a big wild goose chase, the solution was under my very nose all the time. To see the dramatic performance improvement, open the CarouselDemo2 project I published on my last post, and go to the image.h file. At the top of the file, there is a private method named CreateImage, where the IImage object is created. Right after creating the object:

hr = spFactory->CreateImageFromBuffer(m_pImageData, m_nImageSize, BufferDisposalFlagNone, &m_spImage);

just add the following lines:

if(SUCCEEDED(hr))
hr = m_spImage->SetImageFlags(ImageFlagsCaching);

Now run the demo again and see how much the performance has improved!

Now that this issue is solved, let me tell you about the wild goose chase. My original idea was to use the AlphaBlend API that has been added to Windows Mobile 5. This function does support per-pixel alpha blending so it looked very promising. Unfortunately, there seems to be no easy way to load a 32 bpp bitmap image with Windows Mobile. I tried using SHLoadImageResource (I read somewhere that this would keep the alpha channel intact) but the returned HBITMAP was smashed down to 16 bpp, so the alpha channel was lost. I still have not given up to find an easy way to load the PNG files into memory, and when I do you will be the first to know. Oh, by the way, if you do know please post a comment here!

Revisiting the Carousel

2009-04-11T13:24:00.007+01:00

What's the difference between the two QVGA carousel implementations above? And what about the two VGA implementations below?

The top carousel icons look much better than the ones below, especially the ones with round shapes. What's the difference? While the images at the bottom carousel are regular Windows icons drawn from an image list container, the images at the top carousel are PNG images drawn using the IImage object. These PNG files were created with an "alpha channel", an extra byte of information that specifies how transparent each pixel is. Contrary to Windows CE icons which support only the concept of full transparency or full opaqueness, an alpha value determines how transparent each pixel is, so it can blend with the background and create very smooth transitions (so much nicer and easier on the eye). Using the right tools (these icons were produced using Axialis Icon Workshop), you can easily create these images and render them on the device screen.

The sample project (CarouselDemo2) illustrates how this can be implemented in code. First, I removed all the icon-related code and added two new classes (both on the image.h file): CImage and CImageArray. The first class encapsulates an IImage object and provides minimal services for loading the image from resources and drawing it on a given DC. The second class merely groups instances of the first in an array for convenience.

The great thing about IImage is that it allows us to load a raw byte stream of data making up a recognizable bitmap (more codecs can be added) and transforms it into a usable bitmap (not an HBITMAP, mind you). The supported image formats include JPEG, GIF and PNG with the nice alpha channel feature. All you have to do is load the image and draw it. Simple!

Well, not so fast... Really: it's dreadfully slow! If you run the new demo application you will see what I mean: the carousel scrolls like mud.

I previously had a similar experience where using IImage::Draw on a graphics-intensive GDI application proved impossible to use. The solution was to pre-render the PNG files (no transparencies here) to an HBITMAP and cache them, so I was not really surprised that the painting would be slower (but not that slow!). Is there any way around it? Hopefully yes, and I will write about it (or about my failure to achieve it) on my next blog post.

The missing iso-8859-1 code page

2009-04-07T09:25:00.002+01:00

It's definitely not there. Some people have complained about this when parsing XML files using the .NET CF, but I found out about this yesterday night while trying to custom parse an iso-8859-1 encoded KML file. These files are used by Google Earth to add layers of geographical information on top of the displayed map. The content is regular XML with a specialized syntax that is recognized by the Google Earth application and you can find lots of sources of this type of files. Currently I'm developing a native Virtual Earth map browser for Windows mobile and I want to add the option to read these files and dynamically add the information to the map - that's why I met this issue.

When trying to convert a string using the MultiByteToWideChar API, I got an invalid parameter error when using the iso-8859-1 (28591) code page. A very brief search showed me that this is aknown issue and that, apparently, it's the device manufacturer's decision to include a given code page. Thankfully there seems to be a solution that should solve most of the conversion cases: use the windows-1252 code page.

Now I'm glad that I'm custom-parsing the KML file because I can cheat the encoding section and replace the iso-8859-1 with windows-1252 on-the-fly. I'm not sure I would be so successful if using an automated parser (native or managed). Now the question is: why is iso-8859-1 not there?

The Touch Pro menu

2009-03-31T14:03:00.006+01:00

This is not really a menu but a window instead, and it's also not what I want to write about. What really interested me was the effect of showing a single menu item at the right side of the menu bar. This is a very cool effect because you use the same toolbar button to both show and hide the menu list. Achieving this effect is actually quite easy: you must add an empty button on the left. Here's the script I used to mimic this effect:

IDM_MENU_EMPTY_CANCEL RCDATA
BEGIN
IDM_MENU_EMPTY_CANCEL, 2,
I_IMAGENONE, ID_MENU_EMPTY, TBSTATE_ENABLED, TBSTYLE_AUTOSIZE,
IDS_EMPTY, 0, NOMENU,
I_IMAGENONE, ID_MENU_CANCEL, TBSTATE_ENABLED, TBSTYLE_AUTOSIZE,
ID_MENU_CANCEL, 0, NOMENU,
END

The IDS_EMPTY string resource contains one space character only and the ID_MENU_EMPTY command is not handled. This seems to be the same approach that HTC used when writing the user interface: if you click the left soft button or the left menu area, you will see the visual feedback just like it had a real button.

Camera Capture on CodeProject

2009-03-27T08:47:00.003Z

Here are two interesting articles on using the SHCameraCapture API on C++ applications:

Using SHCameraCapture from a C++ Pocket PC 2003 Application

What to do if SHCameraCapture Returns E_FAIL?

IImage::Draw

2009-03-16T10:40:00.003Z

Drawing PNG images on the screen has become quite easy since Microsoft released the imaging library for Windows Mobile devices. Christopher Fairbairn illustrated the technique on one of his excellent blog posts where he illustrates the process, from acquiring the bitmap data to rendering it on the screen. As he points out, if you load a PNG file with an alpha layer, the transparency will be correctly painted to the image creatin all sorts of nice visual effects. But...

... this is a very expensive operation! In fact, IImage::Draw is slower than painting a non transparency bitmap using the regular BitBlt API. If you need raw performance (like when you are repeatedly painting lots of these to the screen, even with a double buffer) yo will get a terrible performance hit. I found out about this in my map rendering scenario, while adding KML-encoded layers. As the number of bitmaps grows, the rendering speed degrades very rapidly to the point where it becomes unusable. What can you do?

My solution was to pre-render the PNG file into a regular GDI bitmap (adding a white background) and then using the BitBlt API to render it. The performance was back at the expense of some unsightly white backgrounds on my map icons... Seems like this is going to be the solution if I want to stick with the GDI.

P.S: The white box issue was neatly solved by rendering the PNG against a Magenta (RGB(0xff, 0, 0xff)) background and using the TransparentImage API.

Seek and you shall find

2009-03-04T17:08:00.003Z

I recently found myself involved in a prototype Windows Mobile mapping application using the Virtual Earth tile system. Downloading the 256 x 256 bitmaps to a mobile device through a GPRS connection is slow and a bit painful, so I thought about creating an external cache mechanism. This would temporarily store the bitmaps for future use and, instead of writing the bitmaps to the file system I thought about using SQL Compact. There are a few advantages for using a database instead of the file system: you store all the bitmaps in a single file and you get the added benefit of easier maintenance (especially when deleting outdated bitmaps to clear the cache). So I looked back to the OLE DB library I have been working with for some help but I found a major whole (yes, there are more, I know!): there was no support for IRowsetIndex::Seek. Why do I need to use this method when I can equally retrieve individual records using a plain SELECT command? The answer is speed. When you go through the SQL Compact Query Processor your application pays a performance penalty that may be avoided (with varying degrees of effort). In this case I wanted to retrieve a single row from a table using an index (no fancy JOIN or GROUP BY clauses) so using a base table is very appropriate, performance-wise.

The idea is very simple: open a base table cursor and specify the index to use; seek using the required key values and retrieve the row data (if it's there). As I said, you can do this using SQL but there is performance penalty that I did not want to pay, especially when fast painting is required. My major problem was that there was no support for this in the previous version of the OLD DB library, so I had to implement it.

Implementing the IRowsetIndex::Seek supporting code implies adding an extra column binding code (similar to the one I wrote for the columns). In fact, you need an extra IAccessor object to bind the key columns as well as all the other supporting data structures. After a few hours worth of work, I managed to develop a working version of the code that I'm publishing here. If you look at the CRowset class implementation you will see how its complexity increased to a point where some new abstractions are just crying out to be created...

When using this code, remember to:

Add the DBPROP_IRowsetIndex property (set to true) to the rowset property set;
Call CRowset::SetKeyValue for each one of the index key values: ordinals start at one;
To seek to the requested row, call CRowset::Seek specifying the number of valid key columns (the number of valid values set with CRowset::SetKeyValue).

If you get an S_OK as return value, the row was found and the data loaded into the row and ready to use.

Sample code: OleDbClientLib3.zip

EDIT: The link has been fixed.

TouchBrowser - III

2009-02-27T18:03:00.003Z

Here's the latest update of the TouchBrowser code. This time I added a gradient to the selection cursor and also added the file size and last access date. The file size needs to be pretty-printed to make reading easier.

On touch-screen devices, the list now supports a HitTest function that, by delegating to the underlying list item, reports what part of the item was clicked by the user and what action is associated with it. In this case I made the icon area sensitive to the selection action so you now navigate into a directory by clicking the icon. If you just click the file name, nothing will happen beyond scrolling the list and setting the selection cursor.

Sample: TouchBrowser3.zip

SQL Compact Insert Performance

2009-02-27T09:55:00.001Z

I've just posted the SQL Compact Insert Performance demo on my managed code blog. It compares various approaches for inserting large volumes of data in a SQL Compact database and, although written for a managed code audience, is also relevant for native code and the OLE DB stack (the basic reality is the same, only slightly faster...).

TouchBrowser - II

2009-02-19T22:33:00.003Z

Here are a few more tweaks to the touch browser sample I published in my last post. One of the issues that nagged me was the item's height - too small for finger use. This new sample uses the large system image list which makes for bigger items that can be more easily selected with a finger.

As you can see from the image, directories and regular files are rendered differently. File items will also have the file size and last change date printed in the list for the next release.

In this version I also fixed the bub in the EnsureVisible function. When you use the direction keys to navigate the list the items are now correctly shown and the list scrolls with a small animation.

My major concern with this code is finger-friendliness. There are still a lot of rough edges and it works best with a stylus - not my intended purpose. I'm still wondering how to solve this issue and make the code consistently recognize a drag gesture. Finally, I will also have to change the way the code accepts an action on an item. One of the ideas I'm exploring for this is to define a "hot" area in the item that will be used to take the action (in this case to open a file or a directory). Each item will know where its hot area is and will report it back to the container in a HitTest - type of function.

Sample: TouchBrowser2.zip

TouchBrowser - I

2009-02-11T16:15:00.003Z

My planning for the first half of 2009 includes writing a SQL Compact explorer, something like the old isqlw mobile application. One of the components this application needs is a friendly way to select the database files - something along the lines of the open file dialog, only a bit better.

Towards this end, I picked up the "touch" window code used in the property list and adapted it to a very simple file browser. The sample application that I'm publishing includes a revised version of the atltouch.h file and implements a very simple-minded "file explorer" window.

You navigate either by scrolling the list or by using the arrow keys. To list a sub folder, just click it or select it and press the enter key. To back up to the previous level, use the main menu "Up" command.

There are still some issues to be sorted out in this code, like the apparent sensitivity when you start dragging a directory entry up or down - you may end up expanding that directory instead of dragging the list. I have struggled with this issue ever since I started to write this code and I'm starting to believe that a different approach may be needed, like the addition of an action button to the right of each file: only by pressing it will the user get the associated action (opening a sub-folder, for instance). This solution implies having larger list items so that a finger can be effectively used (but this is also an issue with WM devices that are expecting a stylus, not a bulky finger). I will be investigating these issues and posting the code in the next few posts.

Sample code: TouchBrowser.zip (27 KB)

Minor bug correction

2009-02-05T12:18:00.002Z

There is a bug in the last version of atltouch.h that causes a debug assertion when the user clicks the empty area of the list. To solve it open the atltouch.h file and look for the ActivateItem method on line 910. Before the last call to NotifyItemChange, insert the following line:

if(IsValidItem(iItem))

Thanks, Vincent!

Why does CeRapiInitEx return E_INVALIDARG?

2009-02-04T12:04:00.003Z

This was a very hard debugging session. Suddenly I found that while using exactly the same DLLs, Wizard was failing to open a RAPI connection while Console was working without any issues. Yes, the code and the binaries were the same. I tried everything, from file and directory rights to the .NET trust, not forgetting the application threading model. Nothing worked: Wizard consistently reported an E_INVALIDARG error when calling CeRapiInitEx (both static and dynamically linked). What to do?

Out of desperation I called the RAPI initialization code on the main form constructor. Surprise: it works! If I move the code to the button event handler it failed. Then I tried the code on the form load event, only to find that the code also failed. What could be happening between the main form constructor and the form event handling code?

Wait, let's try the code at the end of the main form constructor. Voilá, it failed. At the start of the constructor I get an S_OK and at the end I get an E_INVALIDARG. Why? Some time ago I tried to be really smart (lesson: don't try to be smart) and wrote some code that would test the presence of a particular database engine OLE DB CLSID so that the user would see a list of the available engines on her PC. This actually worked badly and instead of dynamically building the UI labels, I went back to static text but I forgot to remove the calls. They look like this:


DPTRACK_API int IsLocalEngineInstalled(GUID guid)
{
   CComPtr<IDBInitialize>    spInitialize;
   HRESULT                   hr;

   CoInitializeEx(NULL, COINIT_MULTITHREADED);
   hr = CoCreateInstance(guid, NULL, CLSCTX_INPROC_SERVER,
                         IID_IDBInitialize,
                         (void**)&spInitialize);
   CoUninitialize();

   return SUCCEEDED(hr);
}

After removing the calls to this toxic code, Wizard works again! Why is CeRapiInitEx sensitive to COM initialization?

Property List Notifications

2009-02-03T09:16:00.003Z

In my last post I presented a custom control based on the previously published CTouchWindow WTL class. In this post I'm updating the code to support individual item enabling and disabling and also to illustrate how the client code can manage item notifications.

Each CTouchListItem (the base class for CPropertyListItem) now supports two new functions to control and report the enabled state: SetEnabled and IsEnabled. After changing the item state your code will have to update the item in order to reflect its updated status (this is very likely to change in the near future).

The new sample code now implements a very simple notification handling mechanism that tests the check state of the check box. This is done in the dialog's OnItemActivated method, called after a particular item is activated or deactivated. In this case, I test if the activated item is the check box item and enable or disable the second group (collapsing it when disabling).

As always, comments are welcome!

Sample: PropList2.zip (46 KB)

The Property List

2009-01-21T14:06:00.006Z

Before I can go any further in the development of the SQL Explorer device application, I must make sure that I have some components ready for use. These will include (but will not be limited to):

A tree component to display the database schema;
A property list control to edit/display object properties such as connection, table and column properties;
A grid or table-like display for table and query results;
A splitter control for compound views;
A decent file open dialog box.

It surely would be easy to just use the controls that Windows Mobile has in store for us: the tree view and the list view. These are undoubtedly the most used controls but are somewhat déja-vu and are arguably hard to use (especially the list view). I decided to (guess what) write these controls from scratch using something the WTL CTouchWindow class that I published here a some months ago.

This class is a nice candidate to be a base class for the new tree, grid and property list controls because it enables a touch-like interface. My first approach was to implement the property list control taking some of my previous work as a model: A list-based form for Windows Mobile. There are two nagging issues with this code: it uses MFC and subclasses the list view control. As I've learned by implementing the WTL version I'm presenting here, subclassing and custom-painting the list view is more work than you need to implement the control.

The property list is shown here contains a list of editable items and groups. Groups are shown as collapsed (the + sign indicates that the item can be expanded) and, like the original MFC implementation, each group can contain only editable items, not other goups (I may change this later on).

A group should be used as a means to group related data items and to ease the list navigation. When clicked the group expands to display the contained editable items. By clicking again, the group collapses and hides the contained items. This is the same principle that you can see at work with a tree, so extending this concept to a "touch tree" should not be very difficult.

Each editable item may be in either of two states: focused or activated. An item gets the focus when it is clicked and when either the up or down keys are used to change the selection. A focused item shows the focus rectangle around it. When the enter key is pressed (or when the user clicks the item) it is activated (note that activating implies setting the focus). The notable exception to this rule is the text editor item that activates itself when it receives the focus.

When an item is activated, it generally creates a Windows control to perform the editing. The exceptions to this rule are the check box (changes its state) and the group item (expands / collapses the contained items).

As of this implementation, the supported data item types are:

Text editor
Date time editor
Check box
Combo box

Here is the expanding property list:

The scroll bar to the right is custom-painted and is not yet the final version (I'm still thinking about allowing the use of old style scroll bars). The advantage of this type of scroll bar is its transparency - you can actually read what's beneath it so you do get a few more pixels of screen real estate.

I will make lots of changes and additions to this code but, in the meantime, please do take a look at it and be so kind as to use it and criticize it.

Sample code: PropList.zip (45 KB)

Enumerating Unique Constraints

2009-01-09T11:07:00.004Z

There's no big deal about getting the list of UNIQUE constraints - they get listed as indexes by the CIndexesRowset class. When using this schema rowset you get everything that is implemented as an index by the database engine:

Indexes
Primary Key constraints
Unique constraints

While the schema rowset can distinguish between a regular index and a PRIMARY KEY, there's no telling if the listed index is a UNIQUE constraint or not. To correct this, you must first list all the UNIQUE constraints using the new CTableDefinition::FillUniqueArray method (see the updated sample below). As you can see, a unique index is also represented by a CIndex object, but stored in a different collection in the CTableSchema class. When the indexes are finally enumerated, the list is filtered for known UNIQUE constraints.

Another addition to the sample code is the ability to open any SQL Compact database on your device by using the Menu / Open sequence. The code that opens the database uses the installed SQL Compact OLE DB engines to "sense" the correct database format (see the updated CSchemaTreeFrame::OpenDataSource method).

On the next post I will start building the SQL Compact Explorer application by using this sample as a starting point. I will also use this application to bring back some of the code that I have been working on, namely the touch list and all the derivatives that have been developed but left unpublished.

Sample code: SchemaTree4.zip (155 KB)

Windows Mobile VM Article

2009-01-06T22:48:00.002Z

A very interesting article about Windows Mobile Virtual Memory has just turned up on Code Project:

Visualizing the Windows Mobile Virtual Memory Monster

Not only this is a worthwhile read, you will also get a couple of links into the Windows Mobile RSS (Reed and Steve Stuff) Feed blog where the Windows Mobile virtual memory manager is discussed.

On my next post I will publish the revised version of the SchemaTree sample with a few additions, like the ability to open any SQL Compact database on the device. From then on, the project will be renamed and I will start building an on-device SQL Compact explorer application. In native code, of course!

Enumerating Foreign Key Constraints

2008-12-19T10:52:00.002Z

After enumerating columns and indexes, now it's time to work on the first of the three types of constraints that are supported by SQL Compact: foreign key constraints. Constraint definitions are also retrieved by calling ITableCreation:.GetTableDefinition. These are returned in an array of DBCONSTRAINTDESC structures that contain all the supported constraints. Each record contains one constraint whose type is stored on the ConstraintType member. The SQL Compact header file defines the following values:

DBCONSTRAINTTYPE_UNIQUE - A unique constraint.
DBCONSTRAINTTYPE_FOREIGNKEY - A foreign key constraint.
DBCONSTRAINTTYPE_PRIMARYKEY - A primary key constraint.
DBCONSTRAINTTYPE_CHECK - A check constraint (not supported by the current SQL Compact versions)
DBCONSTRAINTTYPE_SSCE_DEFAULT - Apparently this is a default value constraint. I have found no documentation about this constant and will investigate it later.

For now, we concentrate on the foreign key constraints and how to show them on the SchemaTree sample aplpication. In this version of tha sample, foreign keys will be displayed in a folder of their own, but for future versions this folder will be renamed "Constraints" and will contain all table contraints, not just foreign keys.

A foreign key definition is stored in the new CForeignKey class. You can see these classes being built from the array of DBCONSTRAINTDESC structures in the new CTableDefinition::FillForeignKeyArray function, where each array item is tested for the correct type and then fed to the CForeignKey class constructor, where all thework takes place. Each foreign key contains a list of matched columns from the reference table and the base table (each pair is stored in an instance of the CForeignKeyPair class).

Displaying these in the schema tree is quite straightforward and follows the same rules I have used for columns and indexes. Please note that there are new accessors for foreign keys on the CTableSchema class.

On my next post I will also enumerate primary keys and unique constraints, put them all under the same folder and start moving towards an on-device SQL Compact editor. I already got a name suggestion from Alberto Silva: SQL Explorer. More names, anyone?

Sample code: SchemaTree3.zip (1.2 MB)

Native Pointers

2008-12-13T15:29:00.002Z

No, I'm not talking about memory addresses, just interesting bits of information for the Native Mobile developer.

MSDN forums have recently moved to a new platform and got new URLs. Here are my favorites:
Visual Studio Smart Device Development - Native C++ Project
SQL Server Compact

On another note, Christopher Fairbairn just wrote another great post in his blog where he discusses some very cool techniques for Native Mobile developers, such as playing sound, using COM and displaying PNG images. A must read!

Code changes

2008-12-11T10:23:00.002Z

After a second look at the array CAtlArray implementation, I'm planning to change all the containers from value to pointer (instead of CAtlArray, I will change this to CAtlArray). The reason is quite simple: when growing the buffer through Add, the existing items are moved to the new buffer with a simple Checked::memmove_s call. The contained class copy constructor is not called which may lead to memory leaks (depending on the contained class implementation). If the array merely contains pointers, everything works as expected and all memory relocations will be much faster (only the pointers are copied, not the objects themselves). The downsidr to this approach is that I will have to manually implement the code to release all the objects pointed to by the array.

Finally, don't miss this great post on the Windows Mobile Team Blog: Uninstalling Applications Programmatically in Windows Mobile.

The perfect time-waster

2008-12-10T14:55:00.002Z

Reinventing the wheel. That's what I have been doing for the last few days implementing object arrays in C++... Why? Apparently some template classes are not very easily exportable on a DLL so you cannot use useful stuff like CString or CAtlArray as exported class members. After having implemented a very simple string class and while working on object array allocation I decided it was enough: from now on the OLE DB Client library will not live in a DLL but either on a LIB or embedded in the client project. Enough is enough - I want to move ahead and implement the cool stuff, not arrays and strings so that the damned thing can be neatly exported as a DLL.

The code I'm presenting today has some additions to the OLE DB Client library, namely some new schema-related classes:

CSchema - Contains an array of table CTableSchema objects and a reference to a CSession.
CTableSchema - Contains a table schema information. This class is prepared to load this information on demand in order to avoid a performance penalty when enumerating the database schema (all tables are loaded and each table schema is loaded on demand).
CTableDefinition - Helper class that loads the table definition in a single OLE DB call. This populates the columns and constraints collections. Indexes are loaded separately through a specialized schema rowset (see the LoadIndexes method).
CColumn - Contains column schema information.
CIndex - Contains index schema information and a list of index columns.
CIndexColumn - An individual index column.

Please note that some of these classes will change in content and placement (I mean the .cpp file).

The sample project is the same - it enumerates in a tree the schema of the sample database placed on the device root. As you can see from the code, the tree lazily loads the table schema information (when the user expands either the "Columns" or the "Indexes" folder).

After loking a bit at this sample and to how the OLE DB Client has evolved, I started wondering about writing a "Query Analyzer" type of application. This would mimick most of the "old" SQL Compact Query Analyzer application and would add a few more features. Writing this application would require development in other areas such as the user interface, but I think that it will be a very interesting challenge.

What features would you like to see in the open source QA? What name would you give such app?

Sample: SchemaTree2.zip (1.21 MB)

Enumerating Columns with IDBSchemaRowset

2008-12-02T21:54:00.006Z

While it's freezing out there (for Lisbon standards, of course) I have been busy working on the schema enumeration code for the OLE DB Client library. My first approach is to enumerate database tables and columns using the IDBSchemaRowset interface. Access to this interface is encapsulated in the CRowsetInterface class that is derived in CTablesRowset and CColumnsRowset classes to enumerate tables and columns respectively. The IDBSchemaRowset interface works by accepting a set of "restrictions" and generates a schema rowset containing the requested data. Schema rowsets are formally defined on the Appendix B of the OLE DB Programmer's Guide, where you can see the returned columns and the accepted "restrictions". SQL Compact implements a subset of these, as described in the BOL.

Using the CTablesRowset is quite easy (and you have seen it at work in a previous sample where it was used to test for the presence of a given table):

CTablesRowset tables(m_session);
CRowset rowset;
HRESULT hr;
hr = tables.Open(NULL, NULL, NULL, NULL, rowset);

The first four NULL parameters mean that you don't want to filter on any of the supported restrictions, so you will get a list of all the tables in the database. SQL Compact does not support table catalogs nor table schemas so the first two parameters are always NULL for this provider. The third restriction is the table name and the fourth is the table type (see the possible values for this in the TABLES schema rowset reference).

When scrolling through the tables rowset, you retrieve the current table name like this:

CString strTable;
rowset.GetValue(3, strTable);

Now that we have the table name, we can enumerate its columns by using the CColumnsRowset and imposing a restriction on the table name (see the COLUMNS schema rowset reference):

CColumnsRowset columns(m_session);
CRowset rsColumns;

hr = columns.Open(NULL, NULL, strTable, NULL, rsColumns);

This retrieves all columns from the given table and you can get the column name from ordinal 4. As you can see from the COLUMNS schema rowset, there are lots of additional schema information about a column, like its OLE DB type, "nullability", size and more.

Although this is a quite convenient way to retrieve a table schema, we can use another OLE DB interface that in a single call returns column and constraint information. On the next post, I will look at ITableCreation.

Sample code:
SchemaTree.zip
OleDbClientLib2.zip

Connection Manager Article

2008-11-24T11:32:00.003Z

The latest issue of MSDN Magazine is out and it features a great article from Marcus Perryman: Going Places: How Connection Manager Connects. The reason why I'm also referring to this article here is that its code samples are in C, a rarity these days.

SQL Compact Schemas

2008-11-23T16:40:00.003Z

Now that I have looked into the most basic principles of using a SQL Compact database from native code (base table cursors and command execution), I will now look at the available tools to enumerate and manipulate the database schema. There are essentially two ways to go about this:

SQL commands
OLE DB interfaces

Using SQL commands
This may be the easiest approach and is also language-independent - you can use these techniques from both native and managed code. To determine what's in your database you can execute SELECT commands against the INFORMATION_SCHEMA system views. These work like regular SELECT commands and return specific rowsets:

COLUMNS
INDEXES
KEY_COLUMN_USAGE
PROVIDER_TYPES
TABLES
TABLE_CONSTRAINTS
REFERENTIAL_CONSTRAINTS (not documented for 2.0)

To retrieve information from one of these views, you just execute a simple query like:

SELECT * FROM INFORMATION_SCHEMA.TABLES

This will return all tables in your database. The advantage of using a SELECT command is that you can add a WHERE clause and filter the result. If you need to make changes to your database schema you can use all the supported DDL commands. Since version 3.0 came out, Microsoft even provided us with a stored procedure to rename tables: sp_rename. Unfortunately you cannot rename columns which sometimes is needed.

Using OLE DB interfaces
All of this (and a bit more) can also be done through some of the exposed OLE DB interfaces, namely the following Session interfaces:

IDBSchemaRowset - Generates rowsets that are similar to the ones returned by the INFORMATION_SCHEMA views.
IIndexDefinition - Creates and drops indexes.
ITableCreation - Retrieves the table definition (full schema).
ITableDefinition - Creates drops tables, adds and drops columns.
ITableDefinitionWithConstraints - Same as above, but also manages table constraints.
IAlterIndex - Only renames indexes in SQL Compact.
IAlterTable - Allows changing table properties like the table name, column defaults and identity properties. You can also use this interface to rename columns (not available in managed code).

Note that when using these interfaces, you are accessing the storage engine directly, circumventing the Query Processor.

On my next post I'm planning to publish a sample that illustrates how to use these interfaces.

Logo Certification Article

2008-11-20T14:54:00.003Z

Here's a very nice article for native mobile developers that addresses the required coding for some of the strict Windows Mobile Logo Certification. A worthwhile read.

Implementing SQL Compact Command Parameter BLOBs

2008-11-18T11:09:00.005Z

After much ramblings, tests and misconceptions, I finally closed the chapter on SQL Compact command parameters. The sample code (see below) illustrates how to use a BLOB command parameter using the "variable length" binding buffer. Whenever the total size of the columns exceeds the allocated binding buffer size, the binding process is repeated increasing the binding buffer size. Although this is not an optimal solution (being able to use storage objects for this purpose would be much better), it frees you from having to specify the maximum size for BLOB parameters - the code calculates this for you.

Before you can execute a command with parameters, you must prepare it by calling the Prepare method (failing to do so will cause a run-time error). Here the code determines the number of existing parameters and allocates memory for a "user buffer". This user buffer is an array of CDbValue objects, a VARIANT-like class whose sole purpose is to store parameter values. When the user sets a parameter it is first written to one of these objects and upon execution it is copied to the data buffer. Meanwhile, the code determines if the new set of parameters on the CDbValue array requires rebinding, does so if needed and finally copies the values to the data buffer. This way you just have to set the parameters and execute the command (see the sample code).

As a side-effect of all these travails, I ended up creating a new class - CDbValueRef - from which CDbValue derives that I use as the "engine" that sets and gets data from various types, performing the necessary data type conversions. As you can now see, the column data access on the CRowset class is performed through an array of such objects that directly reference the bound data buffer, and they know how to handle storage object BLOBs. Now you can access your CRowset data using a type-safe interface.

Sample: InsertBlob.zip (152 KB)

I was right...

2008-11-17T17:27:00.003Z

... for the wrong reasons. In fact, you cannot use storage objects to set BLOB command parameters. I found that the usual mechanism does not work because:

When retrieving command parameter information through GetParameterInfo, you don't get the DBPARAMFLAGS_ISLONG bit set for BLOBs;
Binding a BLOB parameter through a storage object causes the Execute method to fail reporting a DBSTATUS_E_CANTCONVERTVALUE on the BLOB column.

So I had to revert to the old code that buffers the parameter values in a buffer and forces a rebind when the new set of parameters is larger than the old parameter buffer (used by the accessor). Now that I got this straight, it's time to finish the new demo that inserts BLOBs using a parametrized INSERT command. Please stay tuned for the next post.

I was wrong

2008-11-13T21:26:00.009Z

Tomorrow I will deliver (using the English language) my second presentation in an international Microsoft conference. My first time ever was in 2005 in Nice for the now defunct MEDC (Mobile and Embedded Developers Conference). I have been splitting my time in attending interesting presentations, preparing my own presentation and completing the OLE DB Client library code. It was while working some final details of the presentation that I had an epiphany: I was wrong about BLOB command parameters!

The fact that the OLE DB specification limits us to using just one parameter accessor handle to set all the parameters does not mean that we are unable to use storage objects. Why? When I discussed the limitation of the SQL Compact engine that prevents it to create more than one storage object at the same time, it actually applies to when the data flows from the provider to the consumer, like when you are reading data from a base table cursor or from a query cursor. It does not apply to when data flows from the consumer to the provider. In fact, we experienced no such limitation when inserting or updating data through a base table cursor. The consumer application can create as many storage object as it pleases to, and this is the exact same situation with command parameters: the consumer application can only write to them because SQL Compact does not support output parameters.

Right now this is still a theory that seems to make a lot of sense to me and I will focus now on proving it right. Meanwhile, I developed a new set of classes to manage column values in such a way that they would do double duty when used on a rowset and on command parameters. For the latter, these classes would work as intermediate data buffers that would be used as temporary storage for application-provided parameter data. When needed, the CCommand code would bind these to a real data buffer, recreating the accessor handle when needed. In fact, I have just tested the code and it works... But this can't be right. If I can bind the parameters before setting them (just like in the CRowset class) then I will have the best solution.

Back to the drawing board. (If you want to see the code as it is, please drop me a line.)

SQL Compact Command Parameter BLOBs

2008-10-31T11:53:00.003Z

The problem du jour is to make the CCommand class accept BLOB parameters, something that I have been thinking about the last week or so. Let me recap what's at stake.

When executing a SQL command with parameters, the OLE DB standard allows only for a single accessor handle to be provided by the consumer. This accessor handle is created to bind the command parameter data to a data buffer, just like in a normal rowset. By just having one accessor to handle bind the whole set of parameters means that we cannot bind more than one BLOB with SQL Compact, due to its OLE DB provider limitations. In order to have a general solution, we must bind the BLOB parameter as regular columns but we must know how much memory to allocate for each BLOB parameter, and for each instance of the command execution.

After some thought I concluded that there are two acceptable approaches:

Ask the user to provide maximum acceptable size for each BLOB. This is what the ADO .NET provider does and essentially delegates this responsibility to the user. Do you know in advance what is the maximum size of each BLOB that you are going to feed as a parameter? If so this is acceptable, if not... The great advantage of this approach is that you can make an apriori parameter binding (and make it only once).
Be smart about it and recreate the binding when needed. This requires keeping track of the current BLOB binding sizes and increasing the binding array when needed. This approach frees the user from having to specify a maximum size for the BLOB columns but requires that the binding be rebuilt when a larger BLOB is used.

Because the second option seems to be hardest to implement, that is exactly the one I chose. See you in the next post with the code!

SQL Compact Command Parameters - II

2008-10-23T15:29:00.002+01:00

After a bit of work, here's the new version of the OLE DB Client library with basic support for command parameters. The sample code implements a very simple query:

SELECT * FROM Customers WHERE Country = @c

and executes it by setting the parameter value to 'Germany' (using the same sample database).

The first change you will see in the sample code is that I downgraded the solution to VS 2005 format due to some requests to do so. VS 2008 users can easily upgrade the solution, so this is a minor issue. For convenience, I'm also packaging everything in a single ZIP file. Also, all the samples from now on will deploy to the same device directory (typically in \Program Files\OleDbClient).

Executing a SQL Compact command with parameters is quite simple. In a nutshell:

command.SetText(pszCommand);
hr = command.Prepare();
if(FAILED(hr)) ...
command.SetParam(1, _T("Germany"));
hr = command.Execute(rowset);

Remember from my last post that parameterized commands must be prepared before executing. If you look under the hood, you will see that parameters are bound (using code very similar to CRowset's column binding code) in the Prepare method. You can only set command parameter values after the Prepare method executes successfully.

This code still has some limitations, and the most notable one is the inability to handle BLOB parameters. When executing SQL commands, the OLE DB provider only accepts one accessor handle to bind all parameters. As we have seen in previous posts, this prevents us from binding more than one BLOB parameter. The only solution I devised for this when using the ATL OLE DB Consumer Templates was to force the BLOB parameters to have a maximum lenght and bind them as regular columns. Of course, this implies a change to the parameter binding code and some user intervention...

Sample code: OleDbClientLib.zip

SQL Compact Command Parameters

2008-10-19T21:40:00.002+01:00

Command parameters can be thought of as value replacements or variables for SQL commands. You use a command parameter when you need to execute a SQL command multiple times changing one or more values on each execution. Instead of rebuilding the command text every time the command is executed, your code merely changes a variable value. This means that your code changes from this:

SELECT * FROM Customers WHERE Country = 'Germany'

to this:

SELECT * FROM Customers WHERE Country = @p

Note that the command parameter (@p) replaces a literal value that will change every time you execute the command. Command parameters have some interesting advantages over string concatenation:

Command parameters are strongly typed;
Command parameters make the command run faster;
Command parameters are safe because they avoid SQL injection attacks.

Suppose that you need to pass a date value as a parameter to a SQL command. How exactly do you format a date value in order to be correctly interpreted by the query processor? When using command parameters you just don't are about this - you just feed the parameter with the binary value and it atakes care of the rest, no string formatting contortions needed.

The OLE DB provider requires that parameterized commands be prepared (compiled) before executing. Contrary to the managed ADO .NET provider, the OLE DB provider does check the command for errors and enumerates all the given parameters and even infers their data types (your code can then read this informtion and use it). Preparing a command with the ADO .NET provider for CF merely marks the command as requiring preparation before executing. Either way, a prepared command runs faster than an unprepared one.

Finally, with command parameters there is no way to let your user change the syntax of the underlying SQL command. If you use string concatenation to build command strings (with user input), you risk exposing your code to SQL injection attacks that may break your application code or even its security.

Since version 3.0, SQL Compact supports named parameters meaning that you can give your command parameters some meaningful names. All command parameter identifiers must be prepended by a '@' character. If you are still using SQL CE 2.0 then you are out of luck as all command parameters are identified by a single '?' character and distinguishable by their ordinal position only.

After this brief introduction, I will look at the details of implementing a SQL command parameter management code and some of the finer details of binding, such as what to do with BLOBs.

Very fast row counting on SQL Compact

2008-10-16T21:58:00.003+01:00

What's the fastest way to retrieve a table's row count in SQL Compact 3.0 and 3.5? Use the sp_show_statistics stored procedure! Pass your table name as the first parameter (the second is optional) and you will get back a few statistics about the table and its indexes, so more than one row might be returned. The row count is on the ROWS column (ordinal 3), an unsigned 32-bit integer.

Note that this is not supported in versions 2.0 and earlier.

Meet me at Tech Ed EMEA 2008

2008-10-16T09:18:00.002+01:00

Meet me in Tech Ed EMEA 2008 where I will deliver a presentation about "Microsoft SQL Server 2005 Compact Edition Ultimate Performance Tuning" (Windows Mobile track). See you there!

New column binding and commands

2008-10-15T11:40:00.003+01:00

As promised in a previous post, I changed the way columns are bound in the CRowset class so that the class retains all the important column information retrieved from the IColumnsInfo interface, but also the binding offsets. You can see all of this at work in the new version of CRowset in the sample files.

You will also find the first implementation of the CCommand class, an abstraction over the OLE DB Command object. This first implementation is very simple and allows only for execution of a command text (no parameters yet) returning a CRowset and/or an affected row count.

Using a CCommand object is very simple (much like using CTable):

CCommand command(m_session);

command.SetText(_T("SELECT * FROM Employees"));
hr = command.Execute(m_rowset);

Start by declaring a CCommand object passing the open CSession object as parameter. Next, set the command text and then call execute passing a CRowset object as parameter. You can now use the rowset to navigate the rows and retrieve data.

Note that the returning rowset may not have a bookmark (not in this case, anyway) so you must be careful how you index the resulting columns. Remember that CRowset::GetColumnCount returns the total number of columns including the bookmark, if present. Also, columns are referred to by their ordinal position and the first data column is always at ordinal 1, so if you are iterating through all columns using GetColumnCount, make sure you always start at 1 and also add 1 if the rowset does not have bookmarks (see CRowset::HasBookmark).

Finally, if you don't set any specific command properties (as in this case) you get a forward-only cursor, so there is no going back...

Sample files: EditRow5.zip, oledbcli4.zip

P.S.: Don't try to use the EditRow sample to edit rows coming from the command rowset! I just adapted the sample to show how easy it is to display command data. The resulting is read-only.

OLE DB Commands and SQL Compact

2008-10-10T15:52:00.004+01:00

OLE DB Commands are created through the IDBCreateCommand Session object interface. Just find your Session object and call QueryInterface using the IID_IDBCreateCommand GUID. This interface just exposes one method - CreateCommand - that you use to create OLE DB Command objects. As with most COM and OLE DB objects, several interfaces are exposable for a given object subject to provider implementation. For SQL Compact the following interfaces are exposed:

ICommand - Execute the command through this interface. As I noted in a previous post, the Cancel method is not implemented in SQL Compact.
ICommandPrepare - Use this interface to Prepare or Unprepare the command. Command preparation is useful when executing the same command more than once and is required when using command parameters. Contrary to what happens with the ADO .NET provider, the Prepare method actively validates the SQL Command against the existing database, returning errors for incorrect syntax.
ICommandProperties - Gets or sets the properties associated with the Command.
ICommandText - Gets or sets the command text. This interface inherits from ICommand.
ICommandWithParameters - Manages command parameters (we will have to implement a data binding scheme similar to what exists in CRowset). SQL Compact allows only for input parameters, not output.

As you can see, this is enough to run SQL commands against SQL Compact's Query Processor. For now, we leave this here as a reference while I go back to work and start implementing these features in the OLE DB Client library.

BLOB update code

2008-10-10T10:21:00.003+01:00

The sample code I'm posting today illustrates the techniques I described on my last post. Please note that this is really a sample - I will change it in the next incarnation in order to be more effective when binding data. Right now you can see a bit of a mess when binding columns because both the DBCOLUMNINFO and DBBINDING arrays are being used to find the column information addresses on the data buffer. Next version will have a unique array with all the required info to correctly and accurately describe each column. Enough of this and let's look at the code.

First of all, take a look at CRowset::SetValue. Here you see how text strings (only!) are written and how the code decides if the column is a BLOB or not. If we are writing to a BLOB column, the code creates a CBlobStream object and writes the string contents to it. Upon success, a pointer to the CBlobStream object is stored in the rowset data buffer. When this data is written to the provider (by calling IRowsetChange::SetData on the BLOB column's accessor handle), the provider calls the CBlobStream::Read method to read all the string contents into the BLOB storage. When it is done with it, the provider calls the Release method and the CBlobStream deletes itself - neat.

Finally, take a look at the CRowset::Insert, CRowset::Delete and CRowset::SetData methods. These implement the changes I described before. Also, note that these methods will work even if the IRowsetUpdate interface is not exposed.

Next up - After rewriting most of the column binding code, I will start talking about commands.

Sample files: EditRow4.zip, oledbcli3.zip

Updating rows with BLOBs

2008-10-09T11:45:00.002+01:00

As I have previously discussed, we need to make some substantial changes to our code if we want to update rows with BLOB columns. So how exactly do we update these things? The first thing we need is to implement an object that behaves like an ISequentialStream COM object. Our code will create one such object per BLOB column, store the data in this object and then store a pointer to this object in the rowset buffer. When updating data, the OLE DB provider will read it (just like our code reads the storage objects coming from the provider) and then releases it. To make our life easier, our implementation of this object should delete itself when the reference count reaches zero. This way we will not have to worry about deleting it later.

After setting all the row data (BLOB and otherwise), we can insert or update it. If you want to delete a row it is pointless to write data to it. If you remember, I said that when using BLOBs with SQL Compact we must use the deferred update mode by requesting the IRowsetUpdate interface. Here's how the code changes for each operation (insert, update or delete):

Insert

Call IRowsetChange::InsertRow - this sets the non-BLOB data;
Call IRowsetChange::SetData for each BLOB accessor;
Call IRowsetUpdate::Update to commit the changes.

Update

Call IRowsetChange::SetData for each accessor.
Call IRowsetUpdate::Update to commit the row changes.

Delete

Call IRowsetChange::DeleteRow.
Call IRowsetUpdate::Update.

Note that if any of these processes fails before calling Update, you should call Undo to revert any changes made to the row. I'll show you the code in the next post.

ICommand::Cancel is a dummy

2008-10-08T08:37:00.003+01:00

No, I'm not going to talk about the ICommand interface right now, but I just found out that the ICommand::Cancel implementation on the SQL Compact OLE DB provider is a dummy. Good to know!

Reading BLOBs

2008-10-07T18:01:00.003+01:00

My first approach to the BLOB-handling code took a bit longer than expected due to my aging brain. Please understand that the code I'm publishing today is incomplete and will change in the future. I hope that it helps you in understanding what it takes to read BLOB data through the SQL Compact OLE DB provider.

There are two important changes to the code since last post:

Columns are indexed by their ordinal.
Columns are bound using a two-pass approach - On the first pass the non-BLOB columns are bound using the same accessor handle. The second pass binds the BLOB columns allocating an accessor handle for each column.

The bulk of the code that reads the BLOB into memory is on the new CRowset::GetValue method. Note how the column accessor handle is used to retrieve the ISequentialStream pointer and how it is read into a CString. You have to consume all the data immediately because you cannot keep that stream object alive very long, especially if you want to read or write to another BLOB.

Most of the complexity you see on the code (the difference between binding and column indexes) stems from the fact that columns are being moved around in order to make binding simpler. When life was simpler and there were no BLOBs, there was a direct relation between column and binding entries and there was no need to map between the two. Now that we are moving binding entries around, we must make sure that a mapping exists between both arrays (see the GetBindingIndex method).

Next post will handle writing data to the BLOB.

Sample files: EditRow3.zip, oledbcli2.zip

CStdDialogImpl and the OK button

2008-10-01T13:48:00.005+01:00

Are using CStdDialogImpl as the base class for your WTL dialogs? Are you having difficulty hiding the "ok" button? Here's a simple solution:

Derive your dialog class (say CMyDialog) from

CStdDialogImpl<CMyDialog, SHIDIF_SIZEDLGFULLSCREEN | SHIDIF_SIPDOWN>

On the OnInitDialog handler, call:

SHDoneButton(m_hWnd, SHDB_HIDE);
ModifyStyle(0, WS_NONAVDONEBUTTON, SWP_NOSIZE);

Make sure you change all the calls to CStdDialogImpl<CMyDialog> methods with the new base (you can use a typedef for that).

Let's talk about BLOBs

2008-09-30T08:49:00.003+01:00

So far we have seen how to edit table data using the IRowsetChange interface. The provided sample managed to edit a simple table where all columns had limited size. Table edits are performed in a single step, where all the data is read or written at once, and all the data items are sequentially stored on the same data buffer. This scheme works very well but does not support BLOBs.

A BLOB is a different beast because it can store a very large amount of data. In SQL Compact there are only two types of BLOBs: the image (variable length byte array) and the ntext types (variable length character array). These can be really large and the maximum theoretical size of a BLOB field is 4 GB in SQL Compact (the same size limit for the whole database).Contrary to smaller data types, you may not be able to store the whole BLOB data in the device memory, so we need a different approach to handling these types.

The OLE DB standard allows two different ways to bind BLOB data to your accessor: by reference or by value. SQL Compact does not allow by reference binding, so we are limited to binding these by value? What are the differences between both types of data binding?

When binding by reference, the OLE DB provider allocates memory for each bound BLOB field and fills it with data. A pointer to this data is stored in the accessor buffer along with its status and size. Column data is accessed through an extra pointer indirection and our code must make sure that the BLOB data buffer is released when the data is not needed anymore.
When binding by value, we can either specify (force) a fixed column size or bind through a storage interface. The first approach allows your BLOB data to be manipulated just like any other column but forces you to put a limit on how much data you will handle per BLOB column and this defeats the whole purpose of using them. Alternatively (and most commonly) your code will request that the provider creates a storage object to handle the BLOB data (when reading - when writing data your code must do this). When data is read ftom the table, the provider stores a pointer to either an ISequentialStream or an ILockBytes interface and then your code can read the data through these. When writing, your code must create one of these objects per BLOB column.

The last approach - binding by value through storage objects - is the recommended approach, but it becomes harder to implement when your code is handling more than one BLOB per row. In fact, SQL Compact will only manage (read from or write to) one storage object at a time, and if you ask it to instantiate more than one you will get an error. So how do you read from a table (or query) that returns more than one BLOB column? And how do you write the data? Well, you do so one at a time and this means that your code cannot read nor write the whole data in one single step.

The first change we must make to the rowset code is to add more accessor handles. The first accessor handle will bind the non-BLOB columns and these can be read and written to just like we are doing right now. For each BLOB column the rowset must allocate an additional accessor handle and make a note of it.

When data is read, the non-BLOB columns are all read, but none of the BLOB columns are. When your client code requests data from a BLOB column, the rowset must know that this is a special case, retrieve the colsumn's accessor handle and call IRowset::GetData on that handle. On the data buffer, the OLE DB provider will write a pointer to one of the above-mentioned storage objects (it is chosen by the rowset code when the data binding is created) and your client code must read through it and release it when no longer needed.

When you need to write data back to the table, your client code will have to create one storage object per BLOB column, store the respective pointers in the data buffer and engage in a multi-step update (new row insertion or existing row update). This means that your code will store the non-BLOB columns first and then store the BLOB columns one at a time, but to be able to achieve this the rowset must be in delayed update mode. You specify this mode by setting the DBPROP_IRowsetUpdate property to VARIANT_TRUE and the rowset now exposes the IRowsetUpdate interface. When this interface is exposed, you can call IRowsetChange::SetData on the allocated accessor handles without having to worry about having the row in an inconsisten state (due to table constraints, for instace) while you are updating data. When your code is done setting data through all the accessors, just call IRowsetUpdate::Update to update all the changes at once.

This will be easier to understand through a sample, which I will publish on my next post.

Using the OLE DB Client code

2008-09-29T11:12:00.002+01:00

I was recently asked for permission to use the OLE DB Client library code in a third party application. One of the blog readers asked me about a specific license to go with the code and I pointed him to the CPOL, the permissive license used by most Code Project articles. I'm actually planning to convert the material from these posts into a series of articles to be published in Code Project under this license. This is a very simple license: you can use the code for whatever purposes you like (even in a commercial application) and the only thing you are required to do is to acknowledge who wrote the original code (see the CPOL details).

Now that you know this, should you use this code? If you want to use it to learn about OLE DB and the SQL Compact native provider, please go right ahead and use it (and change it, and tweak it and experiment with it). As to using it in a commercial application, I would say "not right now" for two reasons:

The code has not been extensively tested. Although I plan to publish a bug-free (ah ah ah!) library, I cannot take resposibility for any use you make of it. You will be on your own.
This code will change. Don't start writing your own code based on this library expecting that the library interfaces will not change. They will, and your code will fail to compile.

Now that this is clarified, let's have a look at BLOBs and what we need to support them.

Moving to a DLL

2008-09-24T15:46:00.002+01:00

Can I put the OLE DB Client library in a DLL? I tried but got some compiler warnings (C4251) concerning some of the ATL class templates (CComPtr and CString). The reason is that other classes might inherit from these and template definitions are not always exportable.

To solve this problem, I replaced all references of CComPtr in the protected interface with raw COM pointers, which required me to implement some missing destructors. I also removed the protected CString in the CTable class and now the code cleanly compiles into a DLL.

You can download the rewritten sample code (EditRow2.zip and oledbcli.zip) and test them. Make sure that you place the OleDbClient project in a directory with the same name and at the same level of EditRow.

Updating data

2008-09-22T15:26:00.003+01:00

There are two ways to update data using OLE DB: using SQL DML commands (which I will look at in later posts), or through the IRowsetChange interface exposed through a base table IRowset. It supports the basic methods to insert new rows (InsertRow), update existing rows (SetData) and to delete rows (DeleteRows) and is only exposed when the consumer sets the DBPROP_IRowsetChange property to VARIANT_TRUE.

To illustrate this type of data update, I put up a sample that allows you to edit the "Suppliers" table from the Northwind sample database. The list view display code has been changed in order to support a load-on-demand cache of rows, stored in their native (CRowset's) format. The cache items are preloaded when the list view notifies the parent through the LVN_ODCACHEHINT notification message. Row bookmarks are stored along the row data, so we can now very quickly position the rowset cursor in any row so we can edit its data.

Note that when modifying any data, we have to make sure that both the bookmark and the row ID columns are not touched because both are managed by the engine (the row ID is an IDENTITY column that gets automatically filled in upon insert). Apart from this, setting data is just a matter of copying the new row value to the rowset and correctly setting the column status and length (optional for fixed-length types).

This code does not support BLOBs yet. I will look at these in my next post.

Sample project: EditRow.zip (1.29 MB)

Bookmarks

2008-09-17T09:24:00.002+01:00

So what is a bookmark and why should you care? To put it simply, a bookmark is a row unique identifier on an open rowset that allows for very fast random seeks. In order to get a rowset with bookmarks, you must set the DBPROP_BOOKMARKS property to VARIANT_TRUE. The resulting rowset has the bookmark value at column ordinal zero and its size and type varies according to the database engine. On all versions of SQL Compact the bookmark type is a 32-bit integer (DBTYPE_I4 on SQL CE 2.0 and DBTYPE_UI4 on 3.0 and 3.5).

There are two important bookmark restrictions that you must be aware of:

The bookmark value is valid only for the particular rowset that generated it. You should not store this value for later use after the rowset closes because the bookmark values will be rendered invalid. Even if you open the same rowset again, there is no guarantee that the bookmark values will be the same. (Ok, now that I wrote this disclaimer I can tell you that apparently on base table cursors, SQL Compact does seem to reuse the same bookmark values. Nevertheless don't reuse them, please.)
There is no way to know a bookmark's value before loading the row data to memory. This means that if you are using bookmarks to navigate in your rowset, your code must visit the row before knowing its bookmark value, just like in a book.

So why should you care about bookmarks? I have found a very interesting use for them: editing table data without using unique indexes or primary keys. The bookmark gives you an exact and unique identifier to any row and the rowset pointer moves very quickly to that particular row.

Before you can seek to a known bookmark, your rowset must be created with the DBPROP_IRowsetBookmark property set to VARIANT_TRUE. Doing so exposes the optional IRowsetBookmark interface on the IRowset, and you can use it to very quickly position your rowset pointer to any valid bookmark by calling the PositionOnBookmark method.

Before looking at BLOBs, the next post will show how to edit data using a base table cursor and bookmarks.

Sample Databases

2008-09-15T10:10:00.003+01:00

Isn't this nice? I have been publishing the "OpenTable" sample code but forgot to publish the sample databases. My apologies! You can find the three databases (test20.sdf, test30.sdf and test35.sdf) on this zip file: testxx.zip

Please note that the sample has hard-coded the location of the SDF file in the device root. You can change this, of course.