Tuesday, March 31, 2009

The Touch Pro menu

This is not really a menu but a window instead, and it's also not what I want to write about. What really interested me was the effect of showing a single menu item at the right side of the menu bar. This is a very cool effect because you use the same toolbar button to both show and hide the menu list. Achieving this effect is actually quite easy: you must add an empty button on the left. Here's the script I used to mimic this effect:


IDM_MENU_EMPTY_CANCEL RCDATA
BEGIN
IDM_MENU_EMPTY_CANCEL, 2,
I_IMAGENONE, ID_MENU_EMPTY, TBSTATE_ENABLED, TBSTYLE_AUTOSIZE,
IDS_EMPTY, 0, NOMENU,
I_IMAGENONE, ID_MENU_CANCEL, TBSTATE_ENABLED, TBSTYLE_AUTOSIZE,
ID_MENU_CANCEL, 0, NOMENU,
END

The IDS_EMPTY string resource contains one space character only and the ID_MENU_EMPTY command is not handled. This seems to be the same approach that HTC used when writing the user interface: if you click the left soft button or the left menu area, you will see the visual feedback just like it had a real button.

Friday, March 27, 2009

Monday, March 16, 2009

IImage::Draw

Drawing PNG images on the screen has become quite easy since Microsoft released the imaging library for Windows Mobile devices. Christopher Fairbairn illustrated the technique on one of his excellent blog posts where he illustrates the process, from acquiring the bitmap data to rendering it on the screen. As he points out, if you load a PNG file with an alpha layer, the transparency will be correctly painted to the image creatin all sorts of nice visual effects. But...

... this is a very expensive operation! In fact, IImage::Draw is slower than painting a non transparency bitmap using the regular BitBlt API. If you need raw performance (like when you are repeatedly painting lots of these to the screen, even with a double buffer) yo will get a terrible performance hit. I found out about this in my map rendering scenario, while adding KML-encoded layers. As the number of bitmaps grows, the rendering speed degrades very rapidly to the point where it becomes unusable. What can you do?

My solution was to pre-render the PNG file into a regular GDI bitmap (adding a white background) and then using the BitBlt API to render it. The performance was back at the expense of some unsightly white backgrounds on my map icons... Seems like this is going to be the solution if I want to stick with the GDI.

P.S: The white box issue was neatly solved by rendering the PNG against a Magenta (RGB(0xff, 0, 0xff)) background and using the TransparentImage API.

Wednesday, March 04, 2009

Seek and you shall find

I recently found myself involved in a prototype Windows Mobile mapping application using the Virtual Earth tile system. Downloading the 256 x 256 bitmaps to a mobile device through a GPRS connection is slow and a bit painful, so I thought about creating an external cache mechanism. This would temporarily store the bitmaps for future use and, instead of writing the bitmaps to the file system I thought about using SQL Compact. There are a few advantages for using a database instead of the file system: you store all the bitmaps in a single file and you get the added benefit of easier maintenance (especially when deleting outdated bitmaps to clear the cache). So I looked back to the OLE DB library I have been working with for some help but I found a major whole (yes, there are more, I know!): there was no support for IRowsetIndex::Seek. Why do I need to use this method when I can equally retrieve individual records using a plain SELECT command? The answer is speed. When you go through the SQL Compact Query Processor your application pays a performance penalty that may be avoided (with varying degrees of effort). In this case I wanted to retrieve a single row from a table using an index (no fancy JOIN or GROUP BY clauses) so using a base table is very appropriate, performance-wise.

The idea is very simple: open a base table cursor and specify the index to use; seek using the required key values and retrieve the row data (if it's there). As I said, you can do this using SQL but there is performance penalty that I did not want to pay, especially when fast painting is required. My major problem was that there was no support for this in the previous version of the OLD DB library, so I had to implement it.

Implementing the IRowsetIndex::Seek supporting code implies adding an extra column binding code (similar to the one I wrote for the columns). In fact, you need an extra IAccessor object to bind the key columns as well as all the other supporting data structures. After a few hours worth of work, I managed to develop a working version of the code that I'm publishing here. If you look at the CRowset class implementation you will see how its complexity increased to a point where some new abstractions are just crying out to be created...

When using this code, remember to:
  • Add the DBPROP_IRowsetIndex property (set to true) to the rowset property set;
  • Call CRowset::SetKeyValue for each one of the index key values: ordinals start at one;
  • To seek to the requested row, call CRowset::Seek specifying the number of valid key columns (the number of valid values set with CRowset::SetKeyValue).
If you get an S_OK as return value, the row was found and the data loaded into the row and ready to use.

Sample code: OleDbClientLib3.zip

EDIT: The link has been fixed.

Friday, February 27, 2009

TouchBrowser - III


Here's the latest update of the TouchBrowser code. This time I added a gradient to the selection cursor and also added the file size and last access date. The file size needs to be pretty-printed to make reading easier.

On touch-screen devices, the list now supports a HitTest function that, by delegating to the underlying list item, reports what part of the item was clicked by the user and what action is associated with it. In this case I made the icon area sensitive to the selection action so you now navigate into a directory by clicking the icon. If you just click the file name, nothing will happen beyond scrolling the list and setting the selection cursor.

Sample: TouchBrowser3.zip

SQL Compact Insert Performance

I've just posted the SQL Compact Insert Performance demo on my managed code blog. It compares various approaches for inserting large volumes of data in a SQL Compact database and, although written for a managed code audience, is also relevant for native code and the OLE DB stack (the basic reality is the same, only slightly faster...).

Thursday, February 19, 2009

TouchBrowser - II

Here are a few more tweaks to the touch browser sample I published in my last post. One of the issues that nagged me was the item's height - too small for finger use. This new sample uses the large system image list which makes for bigger items that can be more easily selected with a finger.

As you can see from the image, directories and regular files are rendered differently. File items will also have the file size and last change date printed in the list for the next release.

In this version I also fixed the bub in the EnsureVisible function. When you use the direction keys to navigate the list the items are now correctly shown and the list scrolls with a small animation.

My major concern with this code is finger-friendliness. There are still a lot of rough edges and it works best with a stylus - not my intended purpose. I'm still wondering how to solve this issue and make the code consistently recognize a drag gesture. Finally, I will also have to change the way the code accepts an action on an item. One of the ideas I'm exploring for this is to define a "hot" area in the item that will be used to take the action (in this case to open a file or a directory). Each item will know where its hot area is and will report it back to the container in a HitTest - type of function.

Sample: TouchBrowser2.zip

Wednesday, February 11, 2009

TouchBrowser - I


My planning for the first half of 2009 includes writing a SQL Compact explorer, something like the old isqlw mobile application. One of the components this application needs is a friendly way to select the database files - something along the lines of the open file dialog, only a bit better.

Towards this end, I picked up the "touch" window code used in the property list and adapted it to a very simple file browser. The sample application that I'm publishing includes a revised version of the atltouch.h file and implements a very simple-minded "file explorer" window.

You navigate either by scrolling the list or by using the arrow keys. To list a sub folder, just click it or select it and press the enter key. To back up to the previous level, use the main menu "Up" command.

There are still some issues to be sorted out in this code, like the apparent sensitivity when you start dragging a directory entry up or down - you may end up expanding that directory instead of dragging the list. I have struggled with this issue ever since I started to write this code and I'm starting to believe that a different approach may be needed, like the addition of an action button to the right of each file: only by pressing it will the user get the associated action (opening a sub-folder, for instance). This solution implies having larger list items so that a finger can be effectively used (but this is also an issue with WM devices that are expecting a stylus, not a bulky finger). I will be investigating these issues and posting the code in the next few posts.

Sample code: TouchBrowser.zip (27 KB)

Thursday, February 05, 2009

Minor bug correction

There is a bug in the last version of atltouch.h that causes a debug assertion when the user clicks the empty area of the list. To solve it open the atltouch.h file and look for the ActivateItem method on line 910. Before the last call to NotifyItemChange, insert the following line:

if(IsValidItem(iItem))

Thanks, Vincent!

Wednesday, February 04, 2009

Why does CeRapiInitEx return E_INVALIDARG?

This was a very hard debugging session. Suddenly I found that while using exactly the same DLLs, Wizard was failing to open a RAPI connection while Console was working without any issues. Yes, the code and the binaries were the same. I tried everything, from file and directory rights to the .NET trust, not forgetting the application threading model. Nothing worked: Wizard consistently reported an E_INVALIDARG error when calling CeRapiInitEx (both static and dynamically linked). What to do?

Out of desperation I called the RAPI initialization code on the main form constructor. Surprise: it works! If I move the code to the button event handler it failed. Then I tried the code on the form load event, only to find that the code also failed. What could be happening between the main form constructor and the form event handling code?

Wait, let's try the code at the end of the main form constructor. Voilá, it failed. At the start of the constructor I get an S_OK and at the end I get an E_INVALIDARG. Why? Some time ago I tried to be really smart (lesson: don't try to be smart) and wrote some code that would test the presence of a particular database engine OLE DB CLSID so that the user would see a list of the available engines on her PC. This actually worked badly and instead of dynamically building the UI labels, I went back to static text but I forgot to remove the calls. They look like this:


DPTRACK_API int IsLocalEngineInstalled(GUID guid)
{
CComPtr<IDBInitialize> spInitialize;
HRESULT hr;

CoInitializeEx(NULL, COINIT_MULTITHREADED);
hr = CoCreateInstance(guid, NULL, CLSCTX_INPROC_SERVER,
IID_IDBInitialize,
(void**)&spInitialize);
CoUninitialize();

return SUCCEEDED(hr);
}

After removing the calls to this toxic code, Wizard works again! Why is CeRapiInitEx sensitive to COM initialization?

Tuesday, February 03, 2009

Property List Notifications

In my last post I presented a custom control based on the previously published CTouchWindow WTL class. In this post I'm updating the code to support individual item enabling and disabling and also to illustrate how the client code can manage item notifications.

Each CTouchListItem (the base class for CPropertyListItem) now supports two new functions to control and report the enabled state: SetEnabled and IsEnabled. After changing the item state your code will have to update the item in order to reflect its updated status (this is very likely to change in the near future).

The new sample code now implements a very simple notification handling mechanism that tests the check state of the check box. This is done in the dialog's OnItemActivated method, called after a particular item is activated or deactivated. In this case, I test if the activated item is the check box item and enable or disable the second group (collapsing it when disabling).

As always, comments are welcome!

Sample: PropList2.zip (46 KB)

Wednesday, January 21, 2009

The Property List

Before I can go any further in the development of the SQL Explorer device application, I must make sure that I have some components ready for use. These will include (but will not be limited to):

  • A tree component to display the database schema;
  • A property list control to edit/display object properties such as connection, table and column properties;
  • A grid or table-like display for table and query results;
  • A splitter control for compound views;
  • A decent file open dialog box.

It surely would be easy to just use the controls that Windows Mobile has in store for us: the tree view and the list view. These are undoubtedly the most used controls but are somewhat déja-vu and are arguably hard to use (especially the list view). I decided to (guess what) write these controls from scratch using something the WTL CTouchWindow class that I published here a some months ago.

This class is a nice candidate to be a base class for the new tree, grid and property list controls because it enables a touch-like interface. My first approach was to implement the property list control taking some of my previous work as a model: A list-based form for Windows Mobile. There are two nagging issues with this code: it uses MFC and subclasses the list view control. As I've learned by implementing the WTL version I'm presenting here, subclassing and custom-painting the list view is more work than you need to implement the control.

The property list is shown here contains a list of editable items and groups. Groups are shown as collapsed (the + sign indicates that the item can be expanded) and, like the original MFC implementation, each group can contain only editable items, not other goups (I may change this later on).

A group should be used as a means to group related data items and to ease the list navigation. When clicked the group expands to display the contained editable items. By clicking again, the group collapses and hides the contained items. This is the same principle that you can see at work with a tree, so extending this concept to a "touch tree" should not be very difficult.

Each editable item may be in either of two states: focused or activated. An item gets the focus when it is clicked and when either the up or down keys are used to change the selection. A focused item shows the focus rectangle around it. When the enter key is pressed (or when the user clicks the item) it is activated (note that activating implies setting the focus). The notable exception to this rule is the text editor item that activates itself when it receives the focus.

When an item is activated, it generally creates a Windows control to perform the editing. The exceptions to this rule are the check box (changes its state) and the group item (expands / collapses the contained items).

As of this implementation, the supported data item types are:

  • Text editor
  • Date time editor
  • Check box
  • Combo box
Here is the expanding property list:



The scroll bar to the right is custom-painted and is not yet the final version (I'm still thinking about allowing the use of old style scroll bars). The advantage of this type of scroll bar is its transparency - you can actually read what's beneath it so you do get a few more pixels of screen real estate.

I will make lots of changes and additions to this code but, in the meantime, please do take a look at it and be so kind as to use it and criticize it.

Sample code: PropList.zip (45 KB)

Friday, January 09, 2009

Enumerating Unique Constraints

There's no big deal about getting the list of UNIQUE constraints - they get listed as indexes by the CIndexesRowset class. When using this schema rowset you get everything that is implemented as an index by the database engine:
  • Indexes
  • Primary Key constraints
  • Unique constraints
While the schema rowset can distinguish between a regular index and a PRIMARY KEY, there's no telling if the listed index is a UNIQUE constraint or not. To correct this, you must first list all the UNIQUE constraints using the new CTableDefinition::FillUniqueArray method (see the updated sample below). As you can see, a unique index is also represented by a CIndex object, but stored in a different collection in the CTableSchema class. When the indexes are finally enumerated, the list is filtered for known UNIQUE constraints.

Another addition to the sample code is the ability to open any SQL Compact database on your device by using the Menu / Open sequence. The code that opens the database uses the installed SQL Compact OLE DB engines to "sense" the correct database format (see the updated CSchemaTreeFrame::OpenDataSource method).

On the next post I will start building the SQL Compact Explorer application by using this sample as a starting point. I will also use this application to bring back some of the code that I have been working on, namely the touch list and all the derivatives that have been developed but left unpublished.

Sample code: SchemaTree4.zip (155 KB)

Tuesday, January 06, 2009

Windows Mobile VM Article

A very interesting article about Windows Mobile Virtual Memory has just turned up on Code Project:

Visualizing the Windows Mobile Virtual Memory Monster

Not only this is a worthwhile read, you will also get a couple of links into the Windows Mobile RSS (Reed and Steve Stuff) Feed blog where the Windows Mobile virtual memory manager is discussed.

On my next post I will publish the revised version of the SchemaTree sample with a few additions, like the ability to open any SQL Compact database on the device. From then on, the project will be renamed and I will start building an on-device SQL Compact explorer application. In native code, of course!

Friday, December 19, 2008

Enumerating Foreign Key Constraints

After enumerating columns and indexes, now it's time to work on the first of the three types of constraints that are supported by SQL Compact: foreign key constraints. Constraint definitions are also retrieved by calling ITableCreation:.GetTableDefinition. These are returned in an array of DBCONSTRAINTDESC structures that contain all the supported constraints. Each record contains one constraint whose type is stored on the ConstraintType member. The SQL Compact header file defines the following values:
  • DBCONSTRAINTTYPE_UNIQUE - A unique constraint.
  • DBCONSTRAINTTYPE_FOREIGNKEY - A foreign key constraint.
  • DBCONSTRAINTTYPE_PRIMARYKEY - A primary key constraint.
  • DBCONSTRAINTTYPE_CHECK - A check constraint (not supported by the current SQL Compact versions)
  • DBCONSTRAINTTYPE_SSCE_DEFAULT - Apparently this is a default value constraint. I have found no documentation about this constant and will investigate it later.
For now, we concentrate on the foreign key constraints and how to show them on the SchemaTree sample aplpication. In this version of tha sample, foreign keys will be displayed in a folder of their own, but for future versions this folder will be renamed "Constraints" and will contain all table contraints, not just foreign keys.

A foreign key definition is stored in the new CForeignKey class. You can see these classes being built from the array of DBCONSTRAINTDESC structures in the new CTableDefinition::FillForeignKeyArray function, where each array item is tested for the correct type and then fed to the CForeignKey class constructor, where all thework takes place. Each foreign key contains a list of matched columns from the reference table and the base table (each pair is stored in an instance of the CForeignKeyPair class).

Displaying these in the schema tree is quite straightforward and follows the same rules I have used for columns and indexes. Please note that there are new accessors for foreign keys on the CTableSchema class.

On my next post I will also enumerate primary keys and unique constraints, put them all under the same folder and start moving towards an on-device SQL Compact editor. I already got a name suggestion from Alberto Silva: SQL Explorer. More names, anyone?

Sample code: SchemaTree3.zip (1.2 MB)

Saturday, December 13, 2008

Native Pointers

No, I'm not talking about memory addresses, just interesting bits of information for the Native Mobile developer.

MSDN forums have recently moved to a new platform and got new URLs. Here are my favorites:
Visual Studio Smart Device Development - Native C++ Project
SQL Server Compact

On another note, Christopher Fairbairn just wrote another great post in his blog where he discusses some very cool techniques for Native Mobile developers, such as playing sound, using COM and displaying PNG images. A must read!

Thursday, December 11, 2008

Code changes

After a second look at the array CAtlArray implementation, I'm planning to change all the containers from value to pointer (instead of CAtlArray, I will change this to CAtlArray). The reason is quite simple: when growing the buffer through Add, the existing items are moved to the new buffer with a simple Checked::memmove_s call. The contained class copy constructor is not called which may lead to memory leaks (depending on the contained class implementation). If the array merely contains pointers, everything works as expected and all memory relocations will be much faster (only the pointers are copied, not the objects themselves). The downsidr to this approach is that I will have to manually implement the code to release all the objects pointed to by the array.

Finally, don't miss this great post on the Windows Mobile Team Blog: Uninstalling Applications Programmatically in Windows Mobile.

Wednesday, December 10, 2008

The perfect time-waster

Reinventing the wheel. That's what I have been doing for the last few days implementing object arrays in C++... Why? Apparently some template classes are not very easily exportable on a DLL so you cannot use useful stuff like CString or CAtlArray as exported class members. After having implemented a very simple string class and while working on object array allocation I decided it was enough: from now on the OLE DB Client library will not live in a DLL but either on a LIB or embedded in the client project. Enough is enough - I want to move ahead and implement the cool stuff, not arrays and strings so that the damned thing can be neatly exported as a DLL.

The code I'm presenting today has some additions to the OLE DB Client library, namely some new schema-related classes:
  • CSchema - Contains an array of table CTableSchema objects and a reference to a CSession.
  • CTableSchema - Contains a table schema information. This class is prepared to load this information on demand in order to avoid a performance penalty when enumerating the database schema (all tables are loaded and each table schema is loaded on demand).
  • CTableDefinition - Helper class that loads the table definition in a single OLE DB call. This populates the columns and constraints collections. Indexes are loaded separately through a specialized schema rowset (see the LoadIndexes method).
  • CColumn - Contains column schema information.
  • CIndex - Contains index schema information and a list of index columns.
  • CIndexColumn - An individual index column.
Please note that some of these classes will change in content and placement (I mean the .cpp file).

The sample project is the same - it enumerates in a tree the schema of the sample database placed on the device root. As you can see from the code, the tree lazily loads the table schema information (when the user expands either the "Columns" or the "Indexes" folder).

After loking a bit at this sample and to how the OLE DB Client has evolved, I started wondering about writing a "Query Analyzer" type of application. This would mimick most of the "old" SQL Compact Query Analyzer application and would add a few more features. Writing this application would require development in other areas such as the user interface, but I think that it will be a very interesting challenge.

What features would you like to see in the open source QA? What name would you give such app?

Sample: SchemaTree2.zip (1.21 MB)

Tuesday, December 02, 2008

Enumerating Columns with IDBSchemaRowset

While it's freezing out there (for Lisbon standards, of course) I have been busy working on the schema enumeration code for the OLE DB Client library. My first approach is to enumerate database tables and columns using the IDBSchemaRowset interface. Access to this interface is encapsulated in the CRowsetInterface class that is derived in CTablesRowset and CColumnsRowset classes to enumerate tables and columns respectively. The IDBSchemaRowset interface works by accepting a set of "restrictions" and generates a schema rowset containing the requested data. Schema rowsets are formally defined on the Appendix B of the OLE DB Programmer's Guide, where you can see the returned columns and the accepted "restrictions". SQL Compact implements a subset of these, as described in the BOL.

Using the CTablesRowset is quite easy (and you have seen it at work in a previous sample where it was used to test for the presence of a given table):

CTablesRowset tables(m_session);
CRowset rowset;
HRESULT hr;
hr = tables.Open(NULL, NULL, NULL, NULL, rowset);

The first four NULL parameters mean that you don't want to filter on any of the supported restrictions, so you will get a list of all the tables in the database. SQL Compact does not support table catalogs nor table schemas so the first two parameters are always NULL for this provider. The third restriction is the table name and the fourth is the table type (see the possible values for this in the TABLES schema rowset reference).

When scrolling through the tables rowset, you retrieve the current table name like this:

CString strTable;
rowset.GetValue(3, strTable);

Now that we have the table name, we can enumerate its columns by using the CColumnsRowset and imposing a restriction on the table name (see the COLUMNS schema rowset reference):

CColumnsRowset columns(m_session);
CRowset rsColumns;

hr = columns.Open(NULL, NULL, strTable, NULL, rsColumns);

This retrieves all columns from the given table and you can get the column name from ordinal 4. As you can see from the COLUMNS schema rowset, there are lots of additional schema information about a column, like its OLE DB type, "nullability", size and more.

Although this is a quite convenient way to retrieve a table schema, we can use another OLE DB interface that in a single call returns column and constraint information. On the next post, I will look at ITableCreation.

Sample code:
SchemaTree.zip
OleDbClientLib2.zip

Monday, November 24, 2008

Connection Manager Article

The latest issue of MSDN Magazine is out and it features a great article from Marcus Perryman: Going Places: How Connection Manager Connects. The reason why I'm also referring to this article here is that its code samples are in C, a rarity these days.

Sunday, November 23, 2008

SQL Compact Schemas

Now that I have looked into the most basic principles of using a SQL Compact database from native code (base table cursors and command execution), I will now look at the available tools to enumerate and manipulate the database schema. There are essentially two ways to go about this:
  1. SQL commands
  2. OLE DB interfaces
Using SQL commands
This may be the easiest approach and is also language-independent - you can use these techniques from both native and managed code. To determine what's in your database you can execute SELECT commands against the INFORMATION_SCHEMA system views. These work like regular SELECT commands and return specific rowsets:
  • COLUMNS
  • INDEXES
  • KEY_COLUMN_USAGE
  • PROVIDER_TYPES
  • TABLES
  • TABLE_CONSTRAINTS
  • REFERENTIAL_CONSTRAINTS (not documented for 2.0)
To retrieve information from one of these views, you just execute a simple query like:

SELECT * FROM INFORMATION_SCHEMA.TABLES

This will return all tables in your database. The advantage of using a SELECT command is that you can add a WHERE clause and filter the result. If you need to make changes to your database schema you can use all the supported DDL commands. Since version 3.0 came out, Microsoft even provided us with a stored procedure to rename tables: sp_rename. Unfortunately you cannot rename columns which sometimes is needed.

Using OLE DB interfaces
All of this (and a bit more) can also be done through some of the exposed OLE DB interfaces, namely the following Session interfaces:
  • IDBSchemaRowset - Generates rowsets that are similar to the ones returned by the INFORMATION_SCHEMA views.
  • IIndexDefinition - Creates and drops indexes.
  • ITableCreation - Retrieves the table definition (full schema).
  • ITableDefinition - Creates drops tables, adds and drops columns.
  • ITableDefinitionWithConstraints - Same as above, but also manages table constraints.
  • IAlterIndex - Only renames indexes in SQL Compact.
  • IAlterTable - Allows changing table properties like the table name, column defaults and identity properties. You can also use this interface to rename columns (not available in managed code).
Note that when using these interfaces, you are accessing the storage engine directly, circumventing the Query Processor.

On my next post I'm planning to publish a sample that illustrates how to use these interfaces.

Thursday, November 20, 2008

Logo Certification Article

Here's a very nice article for native mobile developers that addresses the required coding for some of the strict Windows Mobile Logo Certification. A worthwhile read.

Tuesday, November 18, 2008

Implementing SQL Compact Command Parameter BLOBs

After much ramblings, tests and misconceptions, I finally closed the chapter on SQL Compact command parameters. The sample code (see below) illustrates how to use a BLOB command parameter using the "variable length" binding buffer. Whenever the total size of the columns exceeds the allocated binding buffer size, the binding process is repeated increasing the binding buffer size. Although this is not an optimal solution (being able to use storage objects for this purpose would be much better), it frees you from having to specify the maximum size for BLOB parameters - the code calculates this for you.

Before you can execute a command with parameters, you must prepare it by calling the Prepare method (failing to do so will cause a run-time error). Here the code determines the number of existing parameters and allocates memory for a "user buffer". This user buffer is an array of CDbValue objects, a VARIANT-like class whose sole purpose is to store parameter values. When the user sets a parameter it is first written to one of these objects and upon execution it is copied to the data buffer. Meanwhile, the code determines if the new set of parameters on the CDbValue array requires rebinding, does so if needed and finally copies the values to the data buffer. This way you just have to set the parameters and execute the command (see the sample code).

As a side-effect of all these travails, I ended up creating a new class - CDbValueRef - from which CDbValue derives that I use as the "engine" that sets and gets data from various types, performing the necessary data type conversions. As you can now see, the column data access on the CRowset class is performed through an array of such objects that directly reference the bound data buffer, and they know how to handle storage object BLOBs. Now you can access your CRowset data using a type-safe interface.

Sample: InsertBlob.zip (152 KB)

Monday, November 17, 2008

I was right...

... for the wrong reasons. In fact, you cannot use storage objects to set BLOB command parameters. I found that the usual mechanism does not work because:
  1. When retrieving command parameter information through GetParameterInfo, you don't get the DBPARAMFLAGS_ISLONG bit set for BLOBs;
  2. Binding a BLOB parameter through a storage object causes the Execute method to fail reporting a DBSTATUS_E_CANTCONVERTVALUE on the BLOB column.
So I had to revert to the old code that buffers the parameter values in a buffer and forces a rebind when the new set of parameters is larger than the old parameter buffer (used by the accessor). Now that I got this straight, it's time to finish the new demo that inserts BLOBs using a parametrized INSERT command. Please stay tuned for the next post.

Thursday, November 13, 2008

I was wrong

Tomorrow I will deliver (using the English language) my second presentation in an international Microsoft conference. My first time ever was in 2005 in Nice for the now defunct MEDC (Mobile and Embedded Developers Conference). I have been splitting my time in attending interesting presentations, preparing my own presentation and completing the OLE DB Client library code. It was while working some final details of the presentation that I had an epiphany: I was wrong about BLOB command parameters!

The fact that the OLE DB specification limits us to using just one parameter accessor handle to set all the parameters does not mean that we are unable to use storage objects. Why? When I discussed the limitation of the SQL Compact engine that prevents it to create more than one storage object at the same time, it actually applies to when the data flows from the provider to the consumer, like when you are reading data from a base table cursor or from a query cursor. It does not apply to when data flows from the consumer to the provider. In fact, we experienced no such limitation when inserting or updating data through a base table cursor. The consumer application can create as many storage object as it pleases to, and this is the exact same situation with command parameters: the consumer application can only write to them because SQL Compact does not support output parameters.

Right now this is still a theory that seems to make a lot of sense to me and I will focus now on proving it right. Meanwhile, I developed a new set of classes to manage column values in such a way that they would do double duty when used on a rowset and on command parameters. For the latter, these classes would work as intermediate data buffers that would be used as temporary storage for application-provided parameter data. When needed, the CCommand code would bind these to a real data buffer, recreating the accessor handle when needed. In fact, I have just tested the code and it works... But this can't be right. If I can bind the parameters before setting them (just like in the CRowset class) then I will have the best solution.

Back to the drawing board. (If you want to see the code as it is, please drop me a line.)

Friday, October 31, 2008

SQL Compact Command Parameter BLOBs

The problem du jour is to make the CCommand class accept BLOB parameters, something that I have been thinking about the last week or so. Let me recap what's at stake.

When executing a SQL command with parameters, the OLE DB standard allows only for a single accessor handle to be provided by the consumer. This accessor handle is created to bind the command parameter data to a data buffer, just like in a normal rowset. By just having one accessor to handle bind the whole set of parameters means that we cannot bind more than one BLOB with SQL Compact, due to its OLE DB provider limitations. In order to have a general solution, we must bind the BLOB parameter as regular columns but we must know how much memory to allocate for each BLOB parameter, and for each instance of the command execution.

After some thought I concluded that there are two acceptable approaches:
  1. Ask the user to provide maximum acceptable size for each BLOB. This is what the ADO .NET provider does and essentially delegates this responsibility to the user. Do you know in advance what is the maximum size of each BLOB that you are going to feed as a parameter? If so this is acceptable, if not... The great advantage of this approach is that you can make an apriori parameter binding (and make it only once).
  2. Be smart about it and recreate the binding when needed. This requires keeping track of the current BLOB binding sizes and increasing the binding array when needed. This approach frees the user from having to specify a maximum size for the BLOB columns but requires that the binding be rebuilt when a larger BLOB is used.
Because the second option seems to be hardest to implement, that is exactly the one I chose. See you in the next post with the code!

Thursday, October 23, 2008

SQL Compact Command Parameters - II

After a bit of work, here's the new version of the OLE DB Client library with basic support for command parameters. The sample code implements a very simple query:

SELECT * FROM Customers WHERE Country = @c

and executes it by setting the parameter value to 'Germany' (using the same sample database).

The first change you will see in the sample code is that I downgraded the solution to VS 2005 format due to some requests to do so. VS 2008 users can easily upgrade the solution, so this is a minor issue. For convenience, I'm also packaging everything in a single ZIP file. Also, all the samples from now on will deploy to the same device directory (typically in \Program Files\OleDbClient).

Executing a SQL Compact command with parameters is quite simple. In a nutshell:

command.SetText(pszCommand);
hr = command.Prepare();
if(FAILED(hr)) ...
command.SetParam(1, _T("Germany"));
hr = command.Execute(rowset);

Remember from my last post that parameterized commands must be prepared before executing. If you look under the hood, you will see that parameters are bound (using code very similar to CRowset's column binding code) in the Prepare method. You can only set command parameter values after the Prepare method executes successfully.

This code still has some limitations, and the most notable one is the inability to handle BLOB parameters. When executing SQL commands, the OLE DB provider only accepts one accessor handle to bind all parameters. As we have seen in previous posts, this prevents us from binding more than one BLOB parameter. The only solution I devised for this when using the ATL OLE DB Consumer Templates was to force the BLOB parameters to have a maximum lenght and bind them as regular columns. Of course, this implies a change to the parameter binding code and some user intervention...

Sample code: OleDbClientLib.zip

Sunday, October 19, 2008

SQL Compact Command Parameters

Command parameters can be thought of as value replacements or variables for SQL commands. You use a command parameter when you need to execute a SQL command multiple times changing one or more values on each execution. Instead of rebuilding the command text every time the command is executed, your code merely changes a variable value. This means that your code changes from this:

SELECT * FROM Customers WHERE Country = 'Germany'

to this:

SELECT * FROM Customers WHERE Country = @p

Note that the command parameter (@p) replaces a literal value that will change every time you execute the command. Command parameters have some interesting advantages over string concatenation:
  • Command parameters are strongly typed;
  • Command parameters make the command run faster;
  • Command parameters are safe because they avoid SQL injection attacks.
Suppose that you need to pass a date value as a parameter to a SQL command. How exactly do you format a date value in order to be correctly interpreted by the query processor? When using command parameters you just don't are about this - you just feed the parameter with the binary value and it atakes care of the rest, no string formatting contortions needed.

The OLE DB provider requires that parameterized commands be prepared (compiled) before executing. Contrary to the managed ADO .NET provider, the OLE DB provider does check the command for errors and enumerates all the given parameters and even infers their data types (your code can then read this informtion and use it). Preparing a command with the ADO .NET provider for CF merely marks the command as requiring preparation before executing. Either way, a prepared command runs faster than an unprepared one.

Finally, with command parameters there is no way to let your user change the syntax of the underlying SQL command. If you use string concatenation to build command strings (with user input), you risk exposing your code to SQL injection attacks that may break your application code or even its security.

Since version 3.0, SQL Compact supports named parameters meaning that you can give your command parameters some meaningful names. All command parameter identifiers must be prepended by a '@' character. If you are still using SQL CE 2.0 then you are out of luck as all command parameters are identified by a single '?' character and distinguishable by their ordinal position only.

After this brief introduction, I will look at the details of implementing a SQL command parameter management code and some of the finer details of binding, such as what to do with BLOBs.

Thursday, October 16, 2008

Very fast row counting on SQL Compact

What's the fastest way to retrieve a table's row count in SQL Compact 3.0 and 3.5? Use the sp_show_statistics stored procedure! Pass your table name as the first parameter (the second is optional) and you will get back a few statistics about the table and its indexes, so more than one row might be returned. The row count is on the ROWS column (ordinal 3), an unsigned 32-bit integer.

Note that this is not supported in versions 2.0 and earlier.

Meet me at Tech Ed EMEA 2008

Meet me in Tech Ed EMEA 2008 where I will deliver a presentation about "Microsoft SQL Server 2005 Compact Edition Ultimate Performance Tuning" (Windows Mobile track). See you there!

Wednesday, October 15, 2008

New column binding and commands

As promised in a previous post, I changed the way columns are bound in the CRowset class so that the class retains all the important column information retrieved from the IColumnsInfo interface, but also the binding offsets. You can see all of this at work in the new version of CRowset in the sample files.

You will also find the first implementation of the CCommand class, an abstraction over the OLE DB Command object. This first implementation is very simple and allows only for execution of a command text (no parameters yet) returning a CRowset and/or an affected row count.

Using a CCommand object is very simple (much like using CTable):

CCommand command(m_session);

command.SetText(_T("SELECT * FROM Employees"));
hr = command.Execute(m_rowset);

Start by declaring a CCommand object passing the open CSession object as parameter. Next, set the command text and then call execute passing a CRowset object as parameter. You can now use the rowset to navigate the rows and retrieve data.

Note that the returning rowset may not have a bookmark (not in this case, anyway) so you must be careful how you index the resulting columns. Remember that CRowset::GetColumnCount returns the total number of columns including the bookmark, if present. Also, columns are referred to by their ordinal position and the first data column is always at ordinal 1, so if you are iterating through all columns using GetColumnCount, make sure you always start at 1 and also add 1 if the rowset does not have bookmarks (see CRowset::HasBookmark).

Finally, if you don't set any specific command properties (as in this case) you get a forward-only cursor, so there is no going back...

Sample files: EditRow5.zip, oledbcli4.zip

P.S.: Don't try to use the EditRow sample to edit rows coming from the command rowset! I just adapted the sample to show how easy it is to display command data. The resulting is read-only.

Friday, October 10, 2008

OLE DB Commands and SQL Compact

OLE DB Commands are created through the IDBCreateCommand Session object interface. Just find your Session object and call QueryInterface using the IID_IDBCreateCommand GUID. This interface just exposes one method - CreateCommand - that you use to create OLE DB Command objects. As with most COM and OLE DB objects, several interfaces are exposable for a given object subject to provider implementation. For SQL Compact the following interfaces are exposed:
  • ICommand - Execute the command through this interface. As I noted in a previous post, the Cancel method is not implemented in SQL Compact.
  • ICommandPrepare - Use this interface to Prepare or Unprepare the command. Command preparation is useful when executing the same command more than once and is required when using command parameters. Contrary to what happens with the ADO .NET provider, the Prepare method actively validates the SQL Command against the existing database, returning errors for incorrect syntax.
  • ICommandProperties - Gets or sets the properties associated with the Command.
  • ICommandText - Gets or sets the command text. This interface inherits from ICommand.
  • ICommandWithParameters - Manages command parameters (we will have to implement a data binding scheme similar to what exists in CRowset). SQL Compact allows only for input parameters, not output.
As you can see, this is enough to run SQL commands against SQL Compact's Query Processor. For now, we leave this here as a reference while I go back to work and start implementing these features in the OLE DB Client library.

BLOB update code

The sample code I'm posting today illustrates the techniques I described on my last post. Please note that this is really a sample - I will change it in the next incarnation in order to be more effective when binding data. Right now you can see a bit of a mess when binding columns because both the DBCOLUMNINFO and DBBINDING arrays are being used to find the column information addresses on the data buffer. Next version will have a unique array with all the required info to correctly and accurately describe each column. Enough of this and let's look at the code.

First of all, take a look at CRowset::SetValue. Here you see how text strings (only!) are written and how the code decides if the column is a BLOB or not. If we are writing to a BLOB column, the code creates a CBlobStream object and writes the string contents to it. Upon success, a pointer to the CBlobStream object is stored in the rowset data buffer. When this data is written to the provider (by calling IRowsetChange::SetData on the BLOB column's accessor handle), the provider calls the CBlobStream::Read method to read all the string contents into the BLOB storage. When it is done with it, the provider calls the Release method and the CBlobStream deletes itself - neat.

Finally, take a look at the CRowset::Insert, CRowset::Delete and CRowset::SetData methods. These implement the changes I described before. Also, note that these methods will work even if the IRowsetUpdate interface is not exposed.

Next up - After rewriting most of the column binding code, I will start talking about commands.

Sample files: EditRow4.zip, oledbcli3.zip

Thursday, October 09, 2008

Updating rows with BLOBs

As I have previously discussed, we need to make some substantial changes to our code if we want to update rows with BLOB columns. So how exactly do we update these things? The first thing we need is to implement an object that behaves like an ISequentialStream COM object. Our code will create one such object per BLOB column, store the data in this object and then store a pointer to this object in the rowset buffer. When updating data, the OLE DB provider will read it (just like our code reads the storage objects coming from the provider) and then releases it. To make our life easier, our implementation of this object should delete itself when the reference count reaches zero. This way we will not have to worry about deleting it later.

After setting all the row data (BLOB and otherwise), we can insert or update it. If you want to delete a row it is pointless to write data to it. If you remember, I said that when using BLOBs with SQL Compact we must use the deferred update mode by requesting the IRowsetUpdate interface. Here's how the code changes for each operation (insert, update or delete):

Insert
  1. Call IRowsetChange::InsertRow - this sets the non-BLOB data;
  2. Call IRowsetChange::SetData for each BLOB accessor;
  3. Call IRowsetUpdate::Update to commit the changes.
Update
  1. Call IRowsetChange::SetData for each accessor.
  2. Call IRowsetUpdate::Update to commit the row changes.

Delete
  1. Call IRowsetChange::DeleteRow.
  2. Call IRowsetUpdate::Update.
Note that if any of these processes fails before calling Update, you should call Undo to revert any changes made to the row. I'll show you the code in the next post.

Wednesday, October 08, 2008

ICommand::Cancel is a dummy

No, I'm not going to talk about the ICommand interface right now, but I just found out that the ICommand::Cancel implementation on the SQL Compact OLE DB provider is a dummy. Good to know!

Tuesday, October 07, 2008

Reading BLOBs

My first approach to the BLOB-handling code took a bit longer than expected due to my aging brain. Please understand that the code I'm publishing today is incomplete and will change in the future. I hope that it helps you in understanding what it takes to read BLOB data through the SQL Compact OLE DB provider.

There are two important changes to the code since last post:
  1. Columns are indexed by their ordinal.
  2. Columns are bound using a two-pass approach - On the first pass the non-BLOB columns are bound using the same accessor handle. The second pass binds the BLOB columns allocating an accessor handle for each column.
The bulk of the code that reads the BLOB into memory is on the new CRowset::GetValue method. Note how the column accessor handle is used to retrieve the ISequentialStream pointer and how it is read into a CString. You have to consume all the data immediately because you cannot keep that stream object alive very long, especially if you want to read or write to another BLOB.

Most of the complexity you see on the code (the difference between binding and column indexes) stems from the fact that columns are being moved around in order to make binding simpler. When life was simpler and there were no BLOBs, there was a direct relation between column and binding entries and there was no need to map between the two. Now that we are moving binding entries around, we must make sure that a mapping exists between both arrays (see the GetBindingIndex method).

Next post will handle writing data to the BLOB.

Sample files: EditRow3.zip, oledbcli2.zip

Wednesday, October 01, 2008

CStdDialogImpl and the OK button

Are using CStdDialogImpl as the base class for your WTL dialogs? Are you having difficulty hiding the "ok" button? Here's a simple solution:

Derive your dialog class (say CMyDialog) from

CStdDialogImpl<CMyDialog, SHIDIF_SIZEDLGFULLSCREEN | SHIDIF_SIPDOWN>

On the OnInitDialog handler, call:

SHDoneButton(m_hWnd, SHDB_HIDE);
ModifyStyle(0, WS_NONAVDONEBUTTON, SWP_NOSIZE);

Make sure you change all the calls to CStdDialogImpl<CMyDialog> methods with the new base (you can use a typedef for that).

Tuesday, September 30, 2008

Let's talk about BLOBs

So far we have seen how to edit table data using the IRowsetChange interface. The provided sample managed to edit a simple table where all columns had limited size. Table edits are performed in a single step, where all the data is read or written at once, and all the data items are sequentially stored on the same data buffer. This scheme works very well but does not support BLOBs.

A BLOB is a different beast because it can store a very large amount of data. In SQL Compact there are only two types of BLOBs: the image (variable length byte array) and the ntext types (variable length character array). These can be really large and the maximum theoretical size of a BLOB field is 4 GB in SQL Compact (the same size limit for the whole database).Contrary to smaller data types, you may not be able to store the whole BLOB data in the device memory, so we need a different approach to handling these types.

The OLE DB standard allows two different ways to bind BLOB data to your accessor: by reference or by value. SQL Compact does not allow by reference binding, so we are limited to binding these by value? What are the differences between both types of data binding?
  • When binding by reference, the OLE DB provider allocates memory for each bound BLOB field and fills it with data. A pointer to this data is stored in the accessor buffer along with its status and size. Column data is accessed through an extra pointer indirection and our code must make sure that the BLOB data buffer is released when the data is not needed anymore.
  • When binding by value, we can either specify (force) a fixed column size or bind through a storage interface. The first approach allows your BLOB data to be manipulated just like any other column but forces you to put a limit on how much data you will handle per BLOB column and this defeats the whole purpose of using them. Alternatively (and most commonly) your code will request that the provider creates a storage object to handle the BLOB data (when reading - when writing data your code must do this). When data is read ftom the table, the provider stores a pointer to either an ISequentialStream or an ILockBytes interface and then your code can read the data through these. When writing, your code must create one of these objects per BLOB column.
The last approach - binding by value through storage objects - is the recommended approach, but it becomes harder to implement when your code is handling more than one BLOB per row. In fact, SQL Compact will only manage (read from or write to) one storage object at a time, and if you ask it to instantiate more than one you will get an error. So how do you read from a table (or query) that returns more than one BLOB column? And how do you write the data? Well, you do so one at a time and this means that your code cannot read nor write the whole data in one single step.

The first change we must make to the rowset code is to add more accessor handles. The first accessor handle will bind the non-BLOB columns and these can be read and written to just like we are doing right now. For each BLOB column the rowset must allocate an additional accessor handle and make a note of it.

When data is read, the non-BLOB columns are all read, but none of the BLOB columns are. When your client code requests data from a BLOB column, the rowset must know that this is a special case, retrieve the colsumn's accessor handle and call IRowset::GetData on that handle. On the data buffer, the OLE DB provider will write a pointer to one of the above-mentioned storage objects (it is chosen by the rowset code when the data binding is created) and your client code must read through it and release it when no longer needed.

When you need to write data back to the table, your client code will have to create one storage object per BLOB column, store the respective pointers in the data buffer and engage in a multi-step update (new row insertion or existing row update). This means that your code will store the non-BLOB columns first and then store the BLOB columns one at a time, but to be able to achieve this the rowset must be in delayed update mode. You specify this mode by setting the DBPROP_IRowsetUpdate property to VARIANT_TRUE and the rowset now exposes the IRowsetUpdate interface. When this interface is exposed, you can call IRowsetChange::SetData on the allocated accessor handles without having to worry about having the row in an inconsisten state (due to table constraints, for instace) while you are updating data. When your code is done setting data through all the accessors, just call IRowsetUpdate::Update to update all the changes at once.

This will be easier to understand through a sample, which I will publish on my next post.

Monday, September 29, 2008

Using the OLE DB Client code

I was recently asked for permission to use the OLE DB Client library code in a third party application. One of the blog readers asked me about a specific license to go with the code and I pointed him to the CPOL, the permissive license used by most Code Project articles. I'm actually planning to convert the material from these posts into a series of articles to be published in Code Project under this license. This is a very simple license: you can use the code for whatever purposes you like (even in a commercial application) and the only thing you are required to do is to acknowledge who wrote the original code (see the CPOL details).

Now that you know this, should you use this code? If you want to use it to learn about OLE DB and the SQL Compact native provider, please go right ahead and use it (and change it, and tweak it and experiment with it). As to using it in a commercial application, I would say "not right now" for two reasons:
  1. The code has not been extensively tested. Although I plan to publish a bug-free (ah ah ah!) library, I cannot take resposibility for any use you make of it. You will be on your own.
  2. This code will change. Don't start writing your own code based on this library expecting that the library interfaces will not change. They will, and your code will fail to compile.
Now that this is clarified, let's have a look at BLOBs and what we need to support them.

Wednesday, September 24, 2008

Moving to a DLL

Can I put the OLE DB Client library in a DLL? I tried but got some compiler warnings (C4251) concerning some of the ATL class templates (CComPtr and CString). The reason is that other classes might inherit from these and template definitions are not always exportable.

To solve this problem, I replaced all references of CComPtr in the protected interface with raw COM pointers, which required me to implement some missing destructors. I also removed the protected CString in the CTable class and now the code cleanly compiles into a DLL.

You can download the rewritten sample code (EditRow2.zip and oledbcli.zip) and test them. Make sure that you place the OleDbClient project in a directory with the same name and at the same level of EditRow.

Monday, September 22, 2008

Updating data

There are two ways to update data using OLE DB: using SQL DML commands (which I will look at in later posts), or through the IRowsetChange interface exposed through a base table IRowset. It supports the basic methods to insert new rows (InsertRow), update existing rows (SetData) and to delete rows (DeleteRows) and is only exposed when the consumer sets the DBPROP_IRowsetChange property to VARIANT_TRUE.

To illustrate this type of data update, I put up a sample that allows you to edit the "Suppliers" table from the Northwind sample database. The list view display code has been changed in order to support a load-on-demand cache of rows, stored in their native (CRowset's) format. The cache items are preloaded when the list view notifies the parent through the LVN_ODCACHEHINT notification message. Row bookmarks are stored along the row data, so we can now very quickly position the rowset cursor in any row so we can edit its data.

Note that when modifying any data, we have to make sure that both the bookmark and the row ID columns are not touched because both are managed by the engine (the row ID is an IDENTITY column that gets automatically filled in upon insert). Apart from this, setting data is just a matter of copying the new row value to the rowset and correctly setting the column status and length (optional for fixed-length types).

This code does not support BLOBs yet. I will look at these in my next post.

Sample project: EditRow.zip (1.29 MB)

Wednesday, September 17, 2008

Bookmarks

So what is a bookmark and why should you care? To put it simply, a bookmark is a row unique identifier on an open rowset that allows for very fast random seeks. In order to get a rowset with bookmarks, you must set the DBPROP_BOOKMARKS property to VARIANT_TRUE. The resulting rowset has the bookmark value at column ordinal zero and its size and type varies according to the database engine. On all versions of SQL Compact the bookmark type is a 32-bit integer (DBTYPE_I4 on SQL CE 2.0 and DBTYPE_UI4 on 3.0 and 3.5).

There are two important bookmark restrictions that you must be aware of:
  1. The bookmark value is valid only for the particular rowset that generated it. You should not store this value for later use after the rowset closes because the bookmark values will be rendered invalid. Even if you open the same rowset again, there is no guarantee that the bookmark values will be the same. (Ok, now that I wrote this disclaimer I can tell you that apparently on base table cursors, SQL Compact does seem to reuse the same bookmark values. Nevertheless don't reuse them, please.)
  2. There is no way to know a bookmark's value before loading the row data to memory. This means that if you are using bookmarks to navigate in your rowset, your code must visit the row before knowing its bookmark value, just like in a book.
So why should you care about bookmarks? I have found a very interesting use for them: editing table data without using unique indexes or primary keys. The bookmark gives you an exact and unique identifier to any row and the rowset pointer moves very quickly to that particular row.

Before you can seek to a known bookmark, your rowset must be created with the DBPROP_IRowsetBookmark property set to VARIANT_TRUE. Doing so exposes the optional IRowsetBookmark interface on the IRowset, and you can use it to very quickly position your rowset pointer to any valid bookmark by calling the PositionOnBookmark method.

Before looking at BLOBs, the next post will show how to edit data using a base table cursor and bookmarks.

Monday, September 15, 2008

Sample Databases

Isn't this nice? I have been publishing the "OpenTable" sample code but forgot to publish the sample databases. My apologies! You can find the three databases (test20.sdf, test30.sdf and test35.sdf) on this zip file: testxx.zip

Please note that the sample has hard-coded the location of the SDF file in the device root. You can change this, of course.

Database Schema Rowsets

Let's make a small detour here, now that we have mastered the IRowset, and take a brief look at database schema rowsets. The OLE DB specification requires that providers expose a set of rowsets that contain information about the database schema. You can find information about the existing tables, columns, constraints and more (depending on your provider).

Rowset schemas are created by IDBSchemaRowset, an optional Session interface that is supported by SQL Compact, through the GetRowset method. When calling this method, you must specify the rowset schema GUID and an array of VARIANTs containing restrictions that will filter the output of the rowset (each rowset has its own set of restrictions). Each restriction value corresponds to a column in the schema rowset and is either empty (VT_EMPTY), meaning no filtering will occur, or has a value of a type compatible with the column's (see here) by which the rowset will be filtered.

To illustrate this, I adapted the last sample code so that it displays the TABLES schema rowset. The very simple implementation is on the CSchemaRowset (base class) and the CTablesRowset classes.

Sample code: OpenTable4.zip

Saturday, September 13, 2008

Moving around

A little change of plans is in order as instead of discussing bookmarks, I will take a deeper look into the IRowset::GetNextRows method. The reason behind this little change of plans is that I got fed up waiting for the sample's list view to populate. I told you before that this was the least-effective way of displaying large amounts of data in a list view: by filling it all up we have to wait for all the internal memory allocations to run and worse, we essentially duplicate the rowset data (much like reading a table into a .NET's DataSet). In fact, list views work much faster when they don't own the data but rely on the parent window to provide it for them. To do this, you must specify the LVS_OWNERDATA style bit when creating the list view control, thereby making it a virtual list view. Virtual list views query each item and sub-item data through the LVN_GETDISPINFO notification message (more info on this here). If you want to cache the data in advance, the virtual list view notifies the parent through an LVN_ODCACHEHINT notification, where the indexes of the rows about to be displayed are reported. This gives the parent a chance to cache the rows ahead. So how does this affect our sample code?

The first major difference os that we have to keep the CRowset object open thrughout the list view's lifetime so that ic can move back and forth to get the requested row. This means that we must find a way to map the zero-based list view item indexes and the table rows, and the most obvious one is to use the number of skipped rows from the rowset start. When the rowset is open, we must make sure that the internal row handle is initialized by calling MoveFirst. This will read the first row into memory and set the next reading position to "before the second row". What does this mean?

If you look at the IRowset::GetNextRows documentation, you see that the second (lRowsOffset) and third (cRows) parameters are signed numbers. We already know that the cRows parameter can only have the value of 1 in SQL Compact, because this engine only manages one row at a time. Interestingly it can also be -1, meaning that the cursor is reading backwards (the DBPROP_CANFETCHBACKWARDS is VARIANT_TRUE). This value is also by how much the "next read position" is incremented for next read. So if you open a brand-new rowset, call RestartPosition to position it "before the first row" and then consecutively call GetNextRows(NULL, 0, 1, ...) until the return value is different from S_OK, you will have visited all the rows from the rowset.

Moving back on the rowset would entail feeding a negative lRowsOffset value to the call but there are two gotchas:
  1. What value would you use to read the previous row? If you think -1 you are wrong (if you keep cRows = 1) and the reason is simple: you would go back one row from the current reading position (after the current row) and put it before the current row, so you would effectively re-read the current row! To move back one row (with cRows = 1) you must set lRowsOffset to -2.
  2. Can you really scroll back? If you look at all the available SQL Compact BOLs you will see that DBPROP_CANSCROLLBACKWARDS is read-only and VARIANT_FALSE, meaning that lRowsOffset cannot be negative. (I'm still trying to figure out if this is true for 3.0 and 3.5 becase elsewhere these BOLs say that you can have a negative offset.)
Just to be on the safe side, if you need to move back just restart the rowset position and move forwards from there (sigh).

Now go and have a look at the updated sample code. You will see some differences in the code, especially how fast the list view is displayed.


Download sample code: OpenTable3.zip

Friday, September 12, 2008

Date, time and money

The sample from my last post did not show all the table column data because it only understood integers and strings. If you look at the "Orders" table schema you will see datetime and currency columns. Let's look at these types in a bit more detail

datetime
The datetime type is identified by the DBTYPE_DBTIMESTAMP data type identifier and is implemented by the DBTIMESTAMP structure. As you can see from its definition, this is a very simple structure and the revised sample application merely uses it to display the date part as a string. If you want to manipulate the date and time values, I advise you to use the MFC/ATL shared COleDateTime class (it's very easy to convert between the two).

currency
The currency type is identified by the DBTYPE_CY data type identifier and is implemented as the well-known CY or CURRENCY structure. It's essentially an __int64 that represents decimal values scaled by 10,000 so that it has four decimal digits. To manage this data type, and especially to render it as a string, I created the new CCurrency class (not under the OLEDBCLI namespace because this represents a more general Windows data type). The string formatting method (named Format) takes two parameters:
  1. The numeric format of the value: None, Number or Currency. The first format returns the number as it is internally formatted, with trailing zeores and a decimal point (locale-insensitive). The second format returns the value as a number in the given locale and the third does the same but as a currency.
  2. The locale identifier (see above).
You can also use this class to perform some basic arithmetic with currency values (although I have not tested it extensively). 

I will be addressing other OLE DB types as we meet them, but for the next post I will show you a very interesting and useful feature of base table (and scrollable) cursors: bookmarks.