Showing posts with label database. Show all posts
Showing posts with label database. Show all posts

Wednesday, March 27, 2013

Using sqlite.net with POCO classes



SQLite has become one of the most pervasive embedded databases around.  It's built into Android, iOS and OSX and is a part of many applications.  It has also become the recommended client side database for WinRT applications

The go-to solution for using SQLite in .net is sqlite.net.  It’s a simple ORM that comes as one (or two if you want async support) source files.  You can get the full source from github or just the main files from Nuget

Sqlite.net lets you store nearly any object in the database without needing to descend from a specific type.  However to make it work well, you need to decorate your objects with data attributes denoting primary keys, indexes and so on.

For example:

       public class Valuation
       {
             [PrimaryKey, AutoIncrement]
             public int Id { get; set; }
             [Indexed]
             public int StockId { get; set; }
             [Indexed]
             public DateTime Time { get; set; }
             public decimal Price { get; set; }
       }

On startup, you register your class with SQLite as follows:

       var myConnection = new SQLiteConnection ("Stocks.db");
       myConnection.CreateTable<Valuation> ();

This will check to see if the table exists in the database and create or update it if required. 
You can then start using the database in your code.

       var valuations = myConnection.Table<Valuation>().Where(...);

However for my use case (cross platform application with data objects defined in a portable class library and used in WPF, WinRT and MVC applications), the attribute approach didn’t work.  The nice thing about open source though is that you can always change things…

The latest github version of sqlite.net now has an optional argument for CreateTable that allows defining indexes by convention.   

Acceptable values are:
    [Flags]
    public enum CreateFlags
    {
        None = 0,
        ImplicitPK = 1,    // create a primary key for field called 'Id'
        ImplicitIndex = 2, // create an index for fields ending in 'Id'  
        AllImplicit = 3,   // do both above

        AutoIncPK = 4      // force PK field to be auto inc
    }

So to define and register a class, we can now use something like the following:

    public class Valuation
    {
        public int Id { get; set; }
        public int StockId { get; set; }
        public DateTime Time { get; set; }
        public decimal Price { get; set; }
    }

And then 

       myConnection.CreateTable<Valuation>(CreateFlags.AllImplicit | CreateFlags.AutoIncPK);

This will create the table, make Id into an auto incrementing primary key and add an index for StockId.  To explicitly add additional indexes, use the CreateIndex methods.  E.g. 

       myConnection.CreateIndex<Valuation>(v => v.Time);


The main advantage of this approach is that it lets you separate storage details of the object from it's definition.  This is great for cases when you don't want to, or can't, use the sqlite attributes in the main class.  

The main disadvantages are that you have now separated storage from definition, and currently there is no way to set the field size.

Sample applications demonstrating both approaches can be found on github.  The sqlite.dll binaries can be downloaded from here.

Saturday, June 21, 2008

ti Object persistance framework updated

What's up?

v2.50 of tiOPF is now available at http://tiopf.sourceforge.net/

So what's tiOPF?

tiOPF is a Object Persistence Framework. That is, it is a framework based around saving your objects to, and loading them from, databases and/or flat files. See the overview for more details.

In a nutshell it lets you do things like:

var
user: TMyuser;
userList: TUserList;
...

user:= Tuser.CreateNew;
user.FirstName:= 'Sean';
user.LastName:= 'cross';

user.Save;

...
userList:= TUserList.Create;
userList.Load;

for user in userList do
...


tiOPF handles the saving and loading of objects to databases and flat files. You can swap between databases by initialising a new persistence layer.


Why should I care?

  • tiOPF lets you code in objects rather than datasets (you can still use data aware controls though). This gives you more object orientated code
  • tiOPF provides database independance
  • Better code reuse. Because the persistance is separated out, you can use the same code across different databases and structures
  • Easier unit testing (in my experience anyway). I find it much easier to setup objects and test them that to setup databases and test.

Links

Home page: http://tiopf.sourceforge.net/
Overview: http://tiopf.sourceforge.net/Doc/overview/index.shtml
Newsgroups: http://tiopf.sourceforge.net/Support.shtml

Thursday, June 12, 2008

" Connection is busy with results for another hstmt"

One of the problems with working on the same program for years, is that you always end up paying for your sins. In my case the sin in question is still using the bde and odbc.

"Connection is busy with results for another hstmt" is a common error when connecting to ODBC databases. It occurs because an OBDC connection can only have one active cursor at a time. By default, odbc only retrieves the first 20 records for a query. If the query contains more than 20 records, the rest are retrieved on demand. That's all very well, but when you open a second query using same connection/session you get the dreaded hstmt error.

I thought I had beaten it years ago, but my latest set of changes have resulted in the error reappearing.

There are a few possible solutions to this error:

  1. Use TTable components. These don't have the error as they open a new connection each time. Not a very pretty solution :(
  2. Put a FetchAll after each Open. This forces the retrieval of all records. Not practical in my case as I have around 250 tquery components to check.
  3. Use multiple connections. Also not practical for me.
  4. Replace the BDE with something else that doesn't have the problem. In progress but...
  5. Cheat. Set the odbc rowset size to a larger number. If you set it to 200, then the first 200 records will be returned. If you set it to -1 then ALL records will be returned. This could have dire effects on performance if your queries return a lot of records.
This time round I discovered, and went with option 5.

To set the rowset size with the BDE, do the following:
  1. Click on your TDatabase component
  2. In the object inspector, expand Params
  3. Put in a Key of "ROWSET SIZE"
  4. Put in the desired value
Changing the rowset size should work with other odbc connection components as well.

Update: As Otto has pointed out, the error can also be resolved, if you are using sql server 2005 or later, by using the SQL Native Client drivers. See Multiple Active Record Sets for more details. Doesn't help me though :(

Tuesday, April 22, 2008

Database versioning part 2

In part 1, I discussed database versioning with MS SQL Server. SQL Server has robust tools available that simplifies the process of differencing and creating upgrade scripts. DBISAM doesn't have these tools, but using datamodules and table components makes versioning close to painless.

Situation 2
Delphi app
DBISAM database
1 developer

Rental Property Manager uses a DBISAM database (DBISAM v3 in RPM 1, DBISAM v4 in RPM 2). In the 4 years since it was released, it has been through about 20 changes.

Unlike sql server, there is no database comparison utility to generate upgrade scripts. That means things have to be done a bit differently.

I still use the same general process as for sql server:
  1. Record the database version number in the database
  2. Define the expected db version in the application
  3. Keep a master db for comparison
  4. Only make one set of changes at a time
  5. Automate differencing as much as possible
  6. Unit test, test and test again. The tests should fail the moment a table is modified.
Because dbisam doesn't support views, the version number is stored (along with a bunch of other info) in an ini file in the database directory.

I have a datamodule, TdmodCheckDatabase. This has a TdbisamTable component for every table in the database. The table component contains all fields in the table and is updated whenever the table is changed.

To make database changes, the following process was used:

  1. Increase the version number in the application
  2. Make and test DB changes.
  3. Update the affected tables in TdmodCheckDatabase
  4. If necessary (rarely) add further upgrade queries to TdmodCheckDatabase. E.g. to set the values of new fields, or to add new data rows.
  5. Generate a CreateDatabase unit script using the supplied database tools.
  6. Update unit tests to suit the new db
When the application is run, it goes through the following process
  1. If no database is found, then run CreateDatabase unit and then do step 3
  2. Get the current version number from the database ini file
  3. If it is less than the expected version number then
    Run CreateDatabase (to create any new tables)
    Check every table component in TdmodCheckDatabase
    Apply any table changes
    run any manual upgrade scripts
  4. Update the version number in the database ini file.
In code this is:


class procedure TdmodCheckDatabase.UpgradeDatabase(databasePath: string; currentVersion, newVersion: integer);
var
module: TdmodCheckDatabase;
f: integer;
begin
module:= TdmodCheckDatabase.create(nil);
try
module.OpenDatabase( databasePath );

for f:= 0 to module.ComponentCount -1 do
begin
if module.Components[f] is TDBISAMTable then
begin
try
// if we need to upgrade table to dbisam 4
if currentVersion <= DB_VERSION_FOR_DBISAM4 then
TDBISAMTable(module.Components[f]).UpgradeTable;

module.UpgradeTable(TDBISAMTable(module.Components[f]));
except
// logging and error stuff removed
end;
end;
end;

for f:= currentVersion + 1 to newVersion do
module.RunUpgradeScripts(f);

module.sqlMakeIndexes.ExecSQL;
// have to create additional indexes manually
finally
module.DBISAMDatabase1.Close;
module.free;
end;
end;

procedure TdmodCheckDatabase.UpgradeTable(table: TDBISAMTable);
var
fieldIndex: integer;
needsRestructure: boolean;
canonical: TField;
begin
needsRestructure:= false;

table.FieldDefs.Update;

// add any new fields to the FieldDefs
if table.FieldDefs.Count < table.FieldCount then
begin
for fieldIndex := table.FieldDefs.Count to table.Fields.Count -1 do
begin
table.FieldDefs.Add(fieldIndex + 1, table.Fields[fieldIndex].FieldName, table.Fields[fieldIndex].DataType, table.Fields[fieldIndex].Size, table.Fields[fieldIndex].Required);
end;
needsRestructure:= true;
end;

// make sure we have correct size for string fields
for fieldIndex := 0 to table.FieldDefs.Count -1 do
begin
if (table.FieldDefs[fieldIndex].DataType = ftString) then
begin
canonical:= table.FindField(table.FieldDefs[fieldIndex].Name);
if assigned(canonical) and (table.FieldDefs[fieldIndex].Size <> canonical.Size) then
begin
// field size has changed
needsRestructure:= true;
table.FieldDefs[fieldIndex].Size:= canonical.Size;
end;
end;
end;

if needsRestructure then
table.AlterTable(); // upgrades table using the new FieldDef values
end;

procedure TdmodCheckDatabase.RunUpgradeScripts(newVersion: integer);
begin
case newVersion of
3: sqlVersion3.ExecSQL;
9: sqlVersion9.ExecSQL;
11: begin // change to DBISAM 4
sqlVersion11a.ExecSQL;
sqlVersion11b.ExecSQL;
sqlVersion11c.ExecSQL;
sqlVersion11d.ExecSQL;
sqlVersion11e.ExecSQL;
end;
19: sqlVersion19.ExecSQL;
20: sqlVersion20.ExecSQL;
end;
end;

Unit tests included:
  • Make sure the current version is correct
  • Make sure that every table and every field exists
  • Create a new blank database (for a number of different versions) and work though the upgrade process to make sure the final database is correct.
  • Restore an existing older database with data and upgrade to the latest version
With this process, altering the database structure is trivial for most changes. Adding fields and table usually requires no more work than updating the table components and generating a new creation script .

The current implementation does have a couple of restrictions in that it won't remove tables or fields. However if that is required, it won't take long to add.

Thursday, April 17, 2008

Database versioning part 1

Versioning databases is one of those ongoing problems that has no one-size-fits-all solution. There are 2 solutions I have developed and used successfully.

The general process I use each time is:
  1. Record the database version number in the database
  2. Define the expected db version the application
  3. Keep a master db for comparison
  4. Only make one set of changes at a time
  5. Automate differencing as much as possible
  6. Unit test, test and test again. The tests should fail the moment a table is modified.

Situation 1
MSDE (= SQL server) database
8 developers (pair programming)
C# application

Every developer had 2 databases, Unit Test db and Application Test DB (UnitDb and AppDb from now on). With msde (now sql express) there are no licensing costs to worry about.
There was also a Master database stored on a central server that served as the canonical reference.

The database version number was stored in a view ('select xx as VersionNumber).

To make database changes, the following process was used:
  1. Check out the latest version of the app
  2. Increase the version number in the application
  3. Make and test DB changes. Usually this was done in UnitDB. AppDb was kept synchronised using SQL Compare
  4. Update the version number in UnitDB
  5. Generate an update script using Sql Compare. Scripts were named UpgradeXXX.sql where XXX is the version that was being upgraded from.
  6. Generate a CreateDatabase.sql script (for shipping with the app) and a CreateDatabaseXXX.sql (for unit tests only) script. In this case XXX is the version that will be created. The 2 scripts are the same except for the name.
  7. If necessary (rarely) append further queries to the scripts. E.g. to set the values of new fields, or to add new data rows.
  8. Update unit tests to suit the new db
  9. Check in changes
When the application is run, it goes through the following process
  1. If no database is found, then run CreateDatabase.sql
  2. Get the current version number from the database
  3. If it is less than the expected version number then
    run UpgradeXXX.sql and go to 2
When the unit tests are run, the first step is to upgrade UnitDb to the current version using the same process as the main application. That means that other peoples changes are automatically applied. The unit tests included:
  • Make sure the current version is correct
  • Make sure that every table and every field exist (this doesn't always need to be explicit as the persistence unit tests should pick up any problems here).
  • Create a new blank database (for a number of different versions) and work though the upgrade process to make sure the final database is correct.
The secret weapon in all this was sql compare which makes generating the scripts quite straight forward.
Also the upgrade scripts can do more than one set of version changes. ie Upgrade001.sql could upgrade the version to v10 so that scripts 002 - 009 don't need to be run.

Situation 2 (delphi and dbisam) will follow in a later post.

Links
Sql Compare