Coding with the enemy

Delphi for iOS!

2011-08-04T16:50:00.004+12:00

I attended the Rad Studio World Tour today in Auckland. Delphi XE2 has some nice features (x64, OSX, FireMonkey) but the standout for me was the iOS support.

Developing for the iPhone et al is mostly a pain in the proverbial. XCode is somewhat of a mess and Objective C was designed by someone with an unholy fetish for square brackets. The last time I did iPhone dev, I did most of my coding in c++ on Windows and only booted into OSX for deployment and testing on the iPhone.

Embarcadero are looking to fix that with Delphi XE2. You can write and test your code in Delphi on Windows. When you need to try it on iOS, you create a xcode project (1 mouse click, only needed once) and then boot into OSX and open the xcode project there. From xcode you can edit, compile, run and debug your 100% Delphi code. If you have either Windows or OSX in a virtual machine you can flick from one tother as you wish. Yor app can be compiled and run in both Windows and iOS.

Awesome!

It's not all perfect, xcode is still there, OSX is a must and the whole code signing is probably as irritating as before, but it's much better than the objective c alternative. It only works with new apps written using FireMonkey but you will be able to pull in older code.

The iOS app is full native code, with access to hardware such as gps, accelerometer and camera.

Note: Accessing the phone hardware means that your app will no longer run under windows due to either the hardware or the support units not being there. I suspect that this is resolvable with some conditional defines and a bit of hacking.

Much to my supprise I am now excited again; both about delphi programming and about iOS programming.

Delphi image components, ImageEn has a new home

2011-06-11T14:44:00.002+12:00

If you have used image editing and manipulation in delphi, there is a good change you have used or heard of ImageEn components. I have used them in my photo printing program, Pics Print.

The ImageEn component suite has now moved to delphi shareware company Xequte. You can find them at www.xequte.com/imageen or at www.imageen.com. If you want to see the components in action, download demos from here.

Helpful tips for US based developers AKA Short date time is evil

2011-03-09T12:26:00.001+13:00

English is a bit like html. There is a standard, but not everyone understands it, and there are a number of slightly incompatible implementations.

This is most obvious in spoken English where accents, and idioms vary wildly causing mutual incomprehension.

The last time I was in the USA, I had the following conversation in a restaurant:

Me: Can I have a coke please

Waitress: I’m sorry, what did you want?

Me: A coca cola please

Waitress: I didn’t quite get that. What did you want?

Me (using the British technique of speaking slowly and loudly so the natives understand): A COCA COLA PLEASE

Waitress (getting quite testy): What?

Me: Can I have a Pepsi?

Waitress: I’m sorry, we don’t have Pepsi. Would you like a coke?

English as she is wrote.

The problem with the World Wide Web is the worldwide part of it. Whatever you write, will probably be read by people from a different country. Even if your audience is English speaking, there are enough differences between countries for confusion and irritation to creep in.

A few examples:

Last year, Steam had a pre release offer for a game due to be released on 10/12/10(or similar). When is that? My natural reading was Dec 10, but it may have been Oct 12. There was no way of telling without trying to divine the writer’s intentions and country of residence.
Last month, I evaluated several rss readers for Win Phone 7. The first one I looked at had the dates hardcoded as mm/dd. If the author had just used the standard system formatting, it would have been correct and less work.
I also looked at The Weather Channel for Win Phone 7. There was a setting to use metric units. However even with this selected some (but not all) temperatures were shown in Fahrenheit. As an added bonus, the tile would sometimes show a temperature of -18. Hawkes Bay hasn't seen -18 since the last ice age.
In Blogger post options, the Post date is in m/d/yy format

So what do (mostly)US websites and applications frequently get wrong?

Dates
Short date time is evil! Repeat that and make it a mantra.

Most of the countries I have visited use dd/mm/yy. The USA uses mm/dd/yy. I believe some countries use yy/mm/dd, but don't quote me on that.

The upshot is that a date such as 09/10/11 is entirely ambiguous without context. Even if the date has some hints, such as 09/17/11, I still have to stop and parse the date rather than reading it. When reading, I devote less than a 1/10 of a second per word. Having to stop, figure out the date, and then start up again seriously screws up my reading.

It's even worse when entering dates, now I have even more opportunity for screwing up. Fortunately most sites have moved to date time pickers or separate day/month/year boxes. Not all have though, one site I was on today used a single edit box with a helpful "dd/mm/yyyy" label beside it.

Blogs and articles frequently refer to seasons. I.e. the new gadget is due in Summer 2011.
This is still problematic; our seasons are about 5.5 months out from the USA so that requires a certain amount of adjustment.

As for the holiday season, WTF? I had to ask an American friend for clarification.

"in general US holiday season means from Thanksgiving (3rd Thursday in November) to the day after new year’s day".

That's not what I would have guessed. If we had a holiday season here, it would start just before Christmas and extend till late Jan. No-one would release anything then, we are all on holiday.

Measurement units
When it comes to measures, there is the US still soldiering on with the Imperial system, and pretty much everyone else is on metric. Unfortunately many sites and applications use just the Imperial measures which is at best an aggravation and at worst requires a quick unit conversion with Google, or more likely, a move onto a different site/app.

Now I understand feet and inches in small quantities provided I don't need to be accurate. Miles require a mental conversion so it has to be something important before I will bother with the maths. After that, I get lost quite quickly.

Ounces are awkward. To my uncertain knowledge, the only people around here who still use ounces are those engaged in buying and selling small amounts of dope. This is not at all useful in performing conversions. Update: I have been reliably informed that the local drug dealers have moved on and now sell in metric units and/or dollar units (e.g. a $250 bag). Pounds are the same as ounces, only more so.

Don't get me started on fluid ounces. I have absolutely no idea how big a fluid ounce is, and no desire to know. This has caused me some aggravation in the past; "How big a drink should I have? Well 60 oz sounds small..."

Number formats
Most countries use "," as the thousand separator and "." as the decimal separator. A few however have it the other way round. Some use spaces for a separator, and the Romans don't even use Arabic numerals.

What to do?

Don't hard code date formats. I use 'dd mmm yyyy' in written text. In my web applications, I mostly use 'dd mmm yyyy'. When I use the system short date time then I make sure that I also use asp.net globalisation settings.
Don't hard code number formats either.
Test with multiple cultures. I generally test with NZ and US cultures for dates. I also use Norway or Sweden for testing number formatting.

Why am I picking on the Americans?
American based programmers seem to the be worst offenders. There are 2 main reasons for this:

The USA is a large part of the English speaking market, too large to be ignored for developers outside it. Anyone wanting to sell software into there needs to take account of the US language and units. The reverse is not necessarily true.
Observational bias. If someone was to hardcode metric and dd/mm/yyyy in their applications, I would never notice.

Update
I didn't put my location in the original post; my point is that you should write and program so that others can understand regardless of your, and their, location. Nonetheless, for those that are interested, I live and work in Hawkes Bay, New Zealand.

Developing on the Win phone 7 hardware without a developer account

2010-12-17T14:28:00.004+13:00

Microsoft have released a free toolset for developing Windows Phone 7 applications. However you cannot debug or deploy onto actual hardware until you create a developer account @ $99 per year.

In my case, I only have a demo phone for 2 weeks so a developer account seems like a waste of money.

To work around this restriction, you need to do the following:

1 Unlock the phone:
Download and install the Chevron unlocker and certificate from here.

2 Stop the phone from relocking every time you reboot
Edit your hosts file (C:\Windows\System32\drivers\etc\hosts)
Add the line
127.0.0.1 developerservices.windowsphone.com

You can now deploy and debug on your phone.

Note: the phone will still relock itself every couple of weeks, giving the following error

“[application name] has been revoked by Microsoft. Please uninstall it.”

Just rerun Chevron and unlock again.

2nd note: You can deploy your .xap file to others and they can run them on unlocked phones. However the continual relocking may get annoying.

3rd note: You can't use Chevron to pirate applications. It will only allow you to run unsigned applications.

Developing for Windows Phone 7 in a virtual machine

2010-12-13T09:48:00.004+13:00

According to Microsoft, installing the windows phone developer tools into a virtual machine is not supported. This is because the phone emulator is itself a virtual machine and, as Inception has shown, running a virtual machine inside a virtual machine gets really slow. The emulator also requires DirectX 10 for XNA develoment, and current virtual machines only offer DirectX9.

However I do all my development in virtual machines. I already have a VM setup with VS2010, version control, database etc. I don't fancy setting that up all over again just for a phone, particularly one that I only have for 2 weeks.

Time to do some testing...

Virtual PC
Running the dev tools in a Windows 7 Virtual PC works (or rather fails to work) as advertised. While I could create and compile a phone project, I couldn't actually run it in the emulator.

VMWare
Using a VMware vm was much better. I could compile and run a silverlight project on the emulator. On my laptop, the emulator performance was dire. On my desktop however, emulator performance was adequate but not stunning.

On either machine, XNA projects wouldn't run on the emulator due to the lack of DirectX 10. They would compile but trying to deploy would fail with "The current display adapter does not meet the emulator requirements to run XNA Framework applications."

However deploying and running on a real phone worked fine. Both Silverlight and XNA deployed and ran without any issues.

Booting from a virtual hard drive (thanks Paul)
Should work but requires Windows 7. See here.

TL;DR
You can develop Silverlight applications in a VMWare virtual machine, testing against the emulator (slow) or actual hardware (fast). You can develop XNA applications in a VMWare vm but you need to deploy to actual hardware.

Useful links
Microsoft's free tools: Create App hub
Charles Petzold giant ebook: Programming Windows Phone 7
Reddit Win Phone 7 section: http://www.reddit.com/r/wp7dev/

.net Unit Testing 101

2010-09-10T19:25:00.003+12:00

Links, slides and sample code from my .net Unit Testing 101 presentation.

NUnit - unit testing library
TestDriven.net - Visual Studio integration
Lightspeed ORM Free or full version is needed to run the sample code

Pragmatic unit testing ebook

Presentation and code

Windows Phone 7 - A party pack of sweets for developers

2010-03-22T21:41:00.011+13:00

They are playing advertisements for Pascall's Party Pack on TV at the moment. The tag line is "All sweets you love, and one you can't stand." That pretty much sums up Windows Phone 7 for developers. Lots of goodness with a few things that may leave a bad taste in your mouth.

Despite the name, Windows Phone 7 Series is not a successor to Windows Mobile 6.x. Instead the name is a marketing nod towards Windows 7. Although both WP7 and WM6 are based on Windows CE, there is no resemblance at a higher level. From a user point of view, WP7 is closer to the iPhone than to WM6. A more accurate name would be Windows Phone 1.

After far too much time reading, watching and playing, here are my thoughts on WP7 as relates to development:

Tasty:

Development is done in c# and Silverlight/XNA. While native code would be nice, C# is easier to get into than Objective C. I haven't used either Silverlight or XNA previously but they look a lot more promising than the XP era Windows Forms used by Windows Mobile.
The tools are much better than those for the iPhone, and free
Push notifications look at lot easier to do than on the iPhone
Apparently xbox live integration is really good (I wouldn't know)
There are metric shitloads of tutorials, walk throughs and documentation (Twitter apps are the new hello world)
Games programming looks really, really nice

Meh:

The tools don't work well in virtual machines. You can make it work in a Win 7 32bit VM but it will run like a dog. The tools don't work at all in Win 7 x64. If you don't want to install onto your main system, you can try booting from a virtual hard drive.

Yuck:

No database support (apparently it's not necessary coz you have xml and the cloud). SQL Server CE is built into the rom, but no access is provided.
No multitasking (push notifications only)
No access to the file system. Application files all go into Isolated Storage in the app folder.
No "sideloading" of applications. Apps can only be installed from the Marketplace, or by Visual Studio
No built in file synchronisation (it's this cloud thing again).
No Silverlight support in Windows Mobile 6.x (but there is Silverlight for Symbian wtf) and no Win forms support in WP7. I.e. even if your win mobile app was written in c# it needs to be rewritten for WP7.
Limited APIs (no access to contacts, no sockets, ...)
No native code - no c, c++, pascal etc (and thus no firefox :()

I haven't felt this conflicted since the vfr 1200. Some nice things balanced by some not so nice.

As mentioned, so this is essentially version 1. Some of the issues are planned to be fixed after the initial release (copy/paste, database access).

Links

Microsoft
Windows Phone 7 tools
Code samples
Developer network
Mix10 videos (check out the keynote)
UI Guide

Silverlight
Get started in Silverlight
Get started with Silverlight in WM7

Example apps
Twitter client
Another twitter client (Scott Gu)
Labyrinth
Sqlite database

Blogs
Artificial Ignorance
10rem
Mobile development

Solving a mid-life crisis with a Solid State Drive

2010-02-13T13:20:00.006+13:00

My development laptop is now 18 months old, which makes it older than me in computer years. As part of a mid-life refresh, I replaced the boot drive, (250gb 7200rpm) with an Intel X25M 160gb ssd.

The result of this is that things are much snappier. How much snappier? I timed a few things of importance to me, booting my laptop and virtual machines, starting delphi and rebuilding a medium sized project (600,000 loc).

Overall, using an ssd cut the times by an average of 39%. Not too bad an improvement. To put it another way, my "getting started" time (boot laptop, boot vm, start delphi) went from 6:05 to 3:37.

Edit:
The Delphi 2007 install is screwed up somewhere. It certainly never used to take 3 minutes to start up:( but sometime last year it started getting really slow. I don't use that install or vm much so I haven't bothered trying to sort it out.

[.net] Using sql server full text searching with Mindscape's LightSpeed

2010-01-14T21:18:00.005+13:00

Advance warning: This post is only of use to people using Lightspeed or suffering from insomnia.

The problem
For my current c# project, I am using Mindscape's Lightspeed O/R framework. Lightspeed is a mighty fine product that makes object persistence a breeze. I would kill to have this in delphi.

However the only full text searching that LightSpeed supports out of the box is Lucene.net. Sql Server full text searching is not provided and no-one else seems to have created it.

The good news is that Lightspeed has a pluggable search engine framework so it is possible to add a new search engine. The bad news is that it's not overly well documented and there aren't many examples. The good news is that their online support is pretty good.

The resolution
Mindscape have a blog post outlining the process of adding a new search engine. However there are a number of traps for young players that aren't covered there, or indeed anywhere. The following should shed some light on the process.

Creating a new search engine requires implimenting ISearchEngine

  public interface ISearchEngine
 {
   IList<SearchResult> Search(string query, params string[] scopes);
   void Add(IndexKey indexKey, string data);
   void Update(IndexKey indexKey, string data);
   void Remove(IndexKey indexKey);
   void Optimize();
   void Clear();
   LightSpeedContext Context { get; set; }
   void BeginBulkAdd();
   void EndBulkAdd();
 }

For sql server, and for most databases with inbuilt FTS, most of these methods can just be left as empty method bodies. All that needs to be implemented is Context (stores lightspeed configuration info such as the database connection string) and Search.

All that is needed to implement Context is:

    public LightSpeedContext Context { get; set; }

The fun comes in implementing Search:).

Example class
LightSpeed takes a class and maps it to a database table. In my case, the class is CmsModels.Entities.AccClaim. It is stored in the table AccData and has an integer primary key called ClaimId.

To perform a search, say finding all of my claims, I need to run the following query:

    select *
   from FreeTextTable(AccData, *, 'Sean Cross Prevshort')
   order by RANK desc

returning something like

    KEY       RANK
   175671    27
   175673    23
   175646    15
   175657    7

Key is the pk field (ClaimId) and Rank is how well the record matched the query.

Implimenting search
The Search method takes a query ("Sean Cross Hastings") and an array of type names (["CmsModels.Entities.AccClaim", ...]). It returns a list of SearchResult objects. SearchResult has the following constructor:

    public SearchResult(
        string key,
        string scope,
        string entityId,
        float score
   )

In short, we take the query and the class name, construct a query, run it and use the results to create a list of SearchResult objects. Easy, in theory.

The devilish details
A Getting the table name
Given an object name, Lightspeed has a number of ways of determining the table name. Which option is chosen depends on the model and class settings.
The possibilities are

Table name = class name (eg AccClaim)
Table name = pluralised class name (eg AccClaims)
Table name is explicitly set in class attributes (eg [Table("AccData", IdColumnName="ClaimID")])
Table name is set programmaticly using a user defined naming strategy object (eg TBL_ACCCLAIM)

AFAIAA It is not possible to cover all these cases without access to LightSpeed internals. I have suggested that they provide a GetMyBloodyTableName method but that hasn't happened yet.

I have opted to only support cases 1 and 3 as this covers most of my classes, and it's easy. Any table name that falls into a different case can still be searched by specifying the table name explicitly.

To get the table name, I look for a TableAttribute and use it. If I can't find one, I use the type name.

    string tableName = "";
   Type t = Type.GetType(scope);
   object[] tableAttrib = t.GetCustomAttributes(typeof(TableAttribute), false);
   if (tableAttrib.Length > 0)
   {
       tableName = ((TableAttribute)tableAttrib[0]).Name;
   }
   else          
   {
        tableName = t.Name.ToUpper();
   }

B Running the query
LightSpeed doesn't support running random sql so it's back to ado.net. The Context object can be used to create a SqlCommand object so it's pretty straightforward:

    IDbCommand command = Context.DataProviderObjectFactory.CreateCommand();           
   Context.CreateUnitOfWork().PrepareCommand(command)           
   command.CommandText = string.Format("select * from FreeTextTable([{0}], *, @query) order by RANK desc", tableName);           
   ((SqlCommand)command).Parameters.AddWithValue("@query", query);           
   IDataReader reader = command.ExecuteReader();               

   while (reader.Read())               
   {
       ...
   }

C Creating the SearchResult objects
The SearchResult constructor is as follows:

    public SearchResult(
        string key,
        string scope,
        string entityId,
        float score
   )

scope is the full class name passed into the Search method (eg "CmsModels.Entities.AccClaim")
entityId is the pk returned from the query in the Key field (eg 175671)
score is the rank returned from the query in the Rank field (eg 27)

key is the problem child. It is not properly explained anywhere in the documentation. I only found out how it is created by creating a dummy Add method, putting a breakpoint in there and examining the values passed in. It is a string in the format "{type name}]{entityId}" (eg "[AccClaim]175671").

So creating the results is

    while (reader.Read())
   {
       string entityId = reader[0].ToString();
       int rank = reader.GetInt32(1);
       string srKey = string.Format("[{0}]{1}", t.Name, entityId);
       var sr = new SearchResult(srKey, scope, entityId, rank);
       result.Add(sr);
   }

Use the source Luke
The full source code is here. This is "works on my machine" quality code, so use with care. In particular:

It doesn't handle pluralisation or naming strategies
I cast directly to SqlCommand without testing it. Works for me but..
It's not solidly tested. I only use it for one table so it may do funny things with other table structures.
While running the code for the first time, the office power failed and the entire building went dark. I'm not saying it's related, but if it happens to you, you are on your own.

Things for MindScape to do
Provide a GetMyBloodyTableName and GetMyBloodyKey helper methods. Pretty please.

Solving Excel's "DBCC TRACEON" error with brute force and ignorance

2009-12-09T16:07:00.002+13:00

We use excel reports extensively for reporting. They are fast to create and everyone can read and manipulate them. However there are the occasional gotchas. In the past, I have had to muck about with macros to fix connection string errors.

We upgraded to sql server 2008 on the weekend. Following that, nearly every data aware spreadsheet started giving "User 'public' does not have permission to run DBCC TRACEON" errors.

The problem, and a resolution is described here. Older versions of Excel/MS query identified themselves by including APP=Microsoft® Query;" in the connection string. MS also hard coded a check for "Microsoft® Query" in sql server which then runs DBCC TRACEON for no apparent reason.

Under sql server 2000, this in fine, but under sql server 2008, this fails as DBCC requires admin permissions.

The correct approach is change the connection string as described in the linked posts. Previously I have done this with macros. However this time I had 200+ spreadsheets to change :(.

I used a free search and replace utility (ReplaceText) to replace all instances of "APP=Microsoft® Query;" with "APP=Microsoft® WTFIT;". 2 important things here, the search and the replacement text are the same size, and the strings include "App=". Failing to do either will give you a corrupt file.

Problem solved in about an hour. It's not a pretty solution but it's better than spending a week at it.

Why I am moving from delphi to .net

2009-11-16T22:24:00.003+13:00

This post is somewhat of a conterpoint to Why we didn’t convert to .Net. And perhaps we never will… I would have writen a post earlier, but I have spent most of the last week writing a tender proposal. The tender was for a claims management application with 70 users scattered across the country. My proposal was for a web application, using asp.net mvc and writen in c#.

My main objection to the WoW post is not his (or her) conclusion, but the subtext that .net has nothing to offer. Delphi programmers (or at least the vocal ones hanging out in non-technical) seem to have a strong view on what "real' programming is, and .net fails on several points (not native code, garbage collected, large installation footprint) for these guys. My view is that "real" programming is solving problems, and using the best tools for the job. In some cases that is delphi, but increasingly for me it is c#.

I haven't suddenly jumped ship, I reevaluated our (Catalyst's) needs earlier this year. Two things came out of that; most of our futher development will be in web apps, and .net is a better fit for us.

My and Catalyst's needs are not univeral. I don't pretend that my reason for moving apply to everybody. None the less, here are my reasons for moving:

Performance:
When it comes to performance, my most important benchmark is developer time. As a (mostly) solo developer, I don't have enough time to do all the things that I want. Anything that improves my productivity is good. I find c# faster to develop in.

From an runtime speed, it's hard to find good benchmarks. My feeling is that Delphi is faster, but not enough to make a noticable difference in my kind of apps. .net is let down by the winforms implimentation which (the last time I used it) had painfully slow drawing. The rest of it is plenty fast enough.

Language features:
Garbage collection and Linq. You ever love them or hate them. I love them. Delphi has mostly caught up with the other big ticket items but not these.

The standard library is another .net plus. It may have an enormous footprint but it's certainly extensive.

Staffing:
I work in a small city. Finding delphi developers here is nearly impossible. I only know of dev company using delphi and, for historical reasons, I can't use them.
.net developers is a different story. I could throw a stone out the window and hit one (seriously, they're just across the road).

Finally
I am not abandoning delphi completely, I have a number of delphi apps to maintain so I will be using it for a while yet. .net is not a silve bullet either. For web apps I find .net better, but for desktop apps it's pretty much even.

D2010 suppport in tiOPF

2009-09-22T16:43:00.002+12:00

tiOPF is a Object Persistence Framework. That is, it is a framework based around saving your objects to, and loading them from, databases and/or flat files.

Delphi 2010 support has now been added to the latest repository version.

No release has been made as of yet, so you need to to retrieve it from the subversion repository. See here for instructions.

Notes

Running unit tests requires copying the xdom.pas unit from Delphi 2009as this is no longer supplied
There are a large number of failures with BDE paradox tests. I haven't investigated this yetas no-one uses paradox any more
tiVirtualTree and related components have been removed from the gui controls as they don't work under D2009/D2010.
Unicode is still not supported :(

Links

C# for the iPhone: Monotouch released

2009-09-15T15:56:00.004+12:00

Novell's MonoTouch product has been released. MonoTouch is mono for the iPhone.
This allows compiling .net applications to native code and deploying to the iPhone.

There are akready 200 apps in the app store using Mono via Unity so the underlying technology is fairly well tested. MonoTouch adds Cocoa support and integration into MonoDevelop.

MonoTouch has 2 main issues for some users, the $399US price point, and the logo.
The price has disappointed a number of beta testers and the logo is the second worst I have seen recently (worst is the Toastmasters goatse).

When you install Delphi 2010, put it in a virtual machine

2009-08-28T14:44:00.004+12:00

I recently compared the performance of virtual machines with the real hardware. The figures were suprisingly good.

Shortly after that, disaster struck. My laptop wouldn't turn on. It took Dell 2 weeks to fix, replacing the motherboard and the video card.

If I had delphi installed directly, I would have had 2 weeks of very limited productivity. As it was, I ripped the hard drive out, stuck it in a usb caddy and continued working on another machine. I was back up to speed the next day.
Edit: It didn't take a full day to get productive again, I did other work until it became apparent that it would be a while before my computer was fixed. It only took about 30 minutes to transfer the data over and get going.

If it had been a hard drive problem, I would have restored the latest vm backup off a dvd, pulled the latest changes from the version control or the source backup and been back up to speed with limited data loss.

I get a new laptop next month. Installing delphi is going to be as easy as installing vmware and copying the vm files over. 30 minutes work, most of which is surfing the internet waiting for the files to copy. The last time I actually had to install Delphi, it took hours.

There are other advantages as well, my development backups fit on a single dual layer dvd, I get to run and test on multiple OSs and disaster recovery plan is much shorter.

So make Delphi 2010 your starting point. Download Virtual Box, VMWare Workstation ($$$) or Virtual PC and use that for development. If you are running Windows 7, you could look into Windows XP Mode and use that.

Changing Excel query connection strings

2009-08-14T09:56:00.002+12:00

We use Excel and database queries extensively for reporting purposes. It's quick and easy to set-up, and provides reports that our clients can manipulate.

However I have recently run into a rather painful excel quirk with ODBC connections: Excel stores the database connection string internally. Even if you change the ODBC connection on the computer, excel still uses the original connection from when the query was created.

This bit us when moving the reports to a different machine. Despite having the same ODBC connection set up, excel wouldn't refresh the query and gave the error "[Microsoft][ODBC SQL Server Driver][TCP/IP Sockets]SQL Server does not exist or access denied."

Aside
A similar problem can happen if you update the database from sql server 2000 to sql server 2005 or 2008. In that case you may get the error "[Microsoft][SQL Native Client][SQL Server]User 'DOMAIN\username' does not have permission to run DBCC TRACEON." In that case, you need to alter the connection string to change "APP=Microsoft® Query" to something else. Apparently MS hard coded a check for "Microsoft® Query" in sql server which then runs DBCC TRACEON for no apparent reason. I suggest "APP=WTFWYT"

Changing one query
(from here)
"Open the worksheet and place the cursor on a cell within the cell range of
the query. Press Alt-F11 to open the VBEditor. Press Ctrl-G to open the
Intermediate window. Type the command: ? ActiveCell.QueryTable.Connection.
The embedded connection string will be echoed back to the screen. Put double
quotes around the string and update the connection information with the new
server info. Move the cursor to the beginning of the connection string and
insert the following in front of the string:
ActiveCell.QueryTable.Connection =
"

Changing multiple queries in a spreadsheet
(modified from here)
You need to create the following macro (change connection string to suit, see above) and run it in each spreadsheet requiring change. (See the steps below). Once the macro has run successfully, delete it before saving the spreadsheet.

Sub ChangeConnections()
Dim sh As Worksheet, qy As QueryTable
For Each ws In ActiveWorkbook.Sheets
For Each qy In ws.QueryTables
qy.Connection = "..."
On Error Resume Next
qy.Refresh
If Err.Number <> 0 Then MsgBox "Problem refreshing QueryTable: " & Err.Description
Next qy
Next ws
End Sub

Getting the new connection string
The easiest way to find the new connections string is to create a new query and do the ?ActiveCell.QueryTable.Connection trick.

Windows Performance Index - VMware workstation v Virtual Box

2009-07-10T16:44:00.004+12:00

Virtual Box 3 has been released. I haven't used previous versions of VB, but it has a good rep. Version 3 now supports 3d acceleration. I have been wanting to use Aero and Glass in my virtual machines so that I could develop Vista specific applications so I gave it a try.

First the bad news, VB doesn't support enough 3d features to run Aero (neither does VMware). I did discover a work around though, connect to your virtual machine using Remote Desktop Connection from a Vista machine, turn all graphics options to high and volia.

While I had VB installed, I did a quick and dirty comparision with VMware workstation using Windows Performance Index. Take these results with a grain of salt, I didn't spend much time playing with the setup and tweaking. I couldn't get dual core to work on VB, it just blue-screened, and I didn't have the time to spend sorting it out.

Windows Performance Index

	Host	VMware Single proc	VMware Dual proc	Virtual Box Single Proc
Processor	5.4	4.4	5.2	4.4
Memory	5.1	4.9	5.2	5.9
Primary Hard disk	5.4	5.3	5.4	4.7

The memory figures are surprising, I am not sure if the VB figures of 5.9 are real or just a sign of something wrong somewhere. Draw your own conclusions on that.

VB is slower in hard drive, but otherwise up with the play processor wise. Getting dual core working would probably give similar processor figures to VMware dual core.

I was impressed with how well VMware matches up to the host system. While VB doesn't seem to be quite there yet, it is still doing pretty well, and you can't beat the price.

iPhone programming might be getting a whole lot easier

2009-06-30T22:42:00.005+12:00

The mono project has announced MonoTouch. MonoTouch is Mono for the iPhone and the iPod touch. Applications writen with MonoTouch work and non-jailbroken phones, and can be submitted to the app store.

See Miguel's blog for additional details. According to the comments, a beta should be released in August.

Time to start brushing up my C# skills.

Guido just doesn't get windows programming

2009-06-29T18:06:00.005+12:00

Among the blogs I follow is that of Guido van Rossum, the creator of Python. Mostly they are interesting but his latest one (IronPython in Action and the Decline of Windows) is a large pile of wtf. Apparently programming windows apps is tedious and gui creation requires many lines of code. Even more surprising to me, web apps are far easier to create.

Wow, apparently he has never heard of delphi, visual studio or any of the other gui editors that have been about since at least the mid 80s. The last time I laid a gui out in code, other than for eductational purposes, would have been about 1986. I have no desire to do it again either.

iPhone Dev: Zombie Mansion post mortem

2009-05-13T15:41:00.003+12:00

Overview
I released Zombie Mansion in mid December 08. It is a 1st person shooter, the 2nd available on the iPhone and the first one to be designed for mobile devices rather than being a pc port. ZM was a port and extension of the work I had done previously on the Yeti engine.

Seeing as how every other iPhone developer is sharing their tales of mega success, or crushing failure, I though I would add my story.

Sales
ZM has sold steadily, if not spectacularly. Unfortunately I have not made enough to pay off my house and buy a fast car. On the other hand, it sold well enough to pay my mortgage payments for the 1st 3 months. Sales are as you would expect: hundreds sold in the first couple of days, rapidly dropping off as ZM moved off the recent releases page. My first update provided a brief blip but my second update was unnoticeable in terms of sales.

I took 2 weeks leave to finish ZM, and I estimate that I spent the equivalent of another 2 weeks previously. This doesn't include the original porting work I did moving the yeti engine to the pocket pc.

As a return on time spent, ZM performed slightly better than working at my day job. It was however considerably more fun. It would have performed even better but I made a couple of bad design decisions.

Programming
Unusually, I did the bulk of the development in windows using visual studio rather than in xcode on OSX. I already had an OpenGL ES port running on Windows Mobile and Windows so an iPhone port wasn't much of a problem.

Coding was done in a mixture of C (yeti code) and C++ (my code).
On the iPhone, I used the Oolong engine to handle the sound, input and 3d setup. Oolong provides c++ wrapper classes for all this (and more).

On the pc and windows mobile side, I used the PowerVr sdk to handle 3d setup. Keyboard handling is trivial, and I used Hekkus to provide sound support.

Once I had this all set-up,I did most coding in visual studio. At irregular intervals, I would reboot into OSX and compile with xcode. I mostly used xcode for iPhone specific stuff and performance testing. All my support tools (map editor, 3d importer etc.) are windows based (in c++ builder and Delphi) so that necessitated mostly using windows. I prefer visual studio to xcode, so it wasn't much of a chore.

Mistakes
I made a couple of mistakes. The first, and most major was simplifying the weapons too much. I went with one weapon, a staff, with upgrades rather than with multiple weapons. In retrospect, a bad call.

The second mistake was not putting some help tips around controls. The controls work well if you leave your thumb on the d pad and slide it to control. If you pick your thumb up and move it, the controls don't work as well. Some easy way of explaining this to the player would have made their initial impressions better.

The underlying problem is that perennial development favourite, I underestimated the time needed to write the software. I ran out of annual leave and thus ran out of time. While I enjoyed the game that I wrote, it would have been better if I had more time to add extra features and polish.

Dealing with Apple
There are 3 major drawbacks to iPhone development: Apple, Apple and Apple. I don't know if they are overworked, incompetent or just don't care. Either way, developer support seems to be missing in action.

Since I have had children, I have made a concious effort to improve my language, removing the more robust terms I picked up working in the abbottiors. Three months of dealing with Apple has made a mockery of my self-betterment attempts, leaving me swearing like Gordon Ramsey on a bad day.

Simply getting a paid developer contract took six weeks. My first payment, for December, didn't get paid until late March. Emails asking when money was going to arrive were ignored, or fobbed off with stock responses or outright lies. Whenever I phoned, I was told they couldn't help and to email another department.

Even when I did get paid, there was no notification. Apple can email me every week, trying to sell me a second rate album farted out by a talentless boy band, but they can't be bothered sending me an email saying "We have now condescended to pay you".

As a comparison, RegNow emailed me on Jan 4th letting me know my December payment had been released, and the money was in my account on Jan 6th. Even now that Apple has decided to make my monthly payments on a monthly basis, they still take nearly 40 days longer than Regnow.

Random thoughts
There is clearly still money to be made in iPhone dev. ZM hasn't disturbed the sales charts, but it has still sold well enough to cover it's costs and my time. I imagine that there are many developers in a similar position, not making riches, but making enough to get by and keep developing. A small stable of good games/applications could provide a nice living without requiring a hit.

It used to be that you could get a large amount of exposure and thus sales by releasing an update every couple of weeks. However the volume of new applications has changed that. There are so many apps being released each day that the initial rush of sales from being in the "what's new" charts is much smaller. You need to get sales the hard way, with marketing; promotion and advertising.

I also have a couple of windows shareware applications, Pics Print and Rental Property Manager. While it is not easy to compare windows utilities and business applications with an iPhone, the return on time invested is far greater for these 2, particularly RPM, than for ZM.

All this brings me full circle from where I started out six months ago. There is money to be made in iPhone apps, but there is also money to be made in windows shareware as well. If you are a windows developer, particularly if you are a Delphi developer, then you are probably better off developing windows shareware than iPhone apps. I know a lot more developers making money off windows than off iPhones.

Oh, and developing for iPhones still sucks! Not as badly as I first thought, but still noticeably.

And Now for Something Completely Different

2009-04-17T20:39:00.006+12:00

One of the pragmatic programmer tenets is to learn a new programming language every year. I'm a bit behind on that, I haven't learn a new language since 2002 (I don't count objective C because I hate it and have only learnt the bare minimum necessary to get by). Unfortunately the need to earn a living has gotten in the way of random learning.

However I have now been programming in Delphi for 15 years. It's starting to get a bit boring and it's time to kick-start the brain again. While Delphi will be my main language for the near future, I am going to make the effort to learn something different.

The main things I am looking for in my language de jour are

Dynamically typed (I have done statically typed languages for decades)
Popular (general rule of thumb, should be at least as popular as delphi)
Object Orientated
Cross platform
Batteries included

This pretty much narrows it down to Perl, Php, Python and Ruby. I was initially tempted by Ruby as I don't yet have an R on my alphabetical list of programming languages. However I have settled on Python instead.

Part of the reason for going with Python is the presence of the Python for Delphi components. These promise to let you use python as a scripting language inside delphi apps. These components are used in the PyScripter ide.

The easiest way I have found to get started is to use Portable Python. This will put Python and a few tools (inc. PyScripter) into a portable drive along with a few tools and useful modules such as Django. This is considerably easier for a python newbie than sourcing and installing the packages separately.

Some useful links to get started with:
Official website
Official python tutorial (v2.6)
Dive onto python online book (useful but a bit old)
Pycon US 2009 videos
Reddit python feed

Filtering generic collections with anonymous methods

2009-03-09T12:52:00.009+13:00

I have recently been adding generics to the ti Object Persistence Framework. As part of that I was asked to add enumerator filtering. I did this using a similar technique to that shown here by Malcolm Groves.

I ended up with code used like this:


for item in Flist.FilteredEnumerator(function (TestObject: TtiOPFTestIntegerProp): Boolean
                                begin
                                  result:= TestObject.IntField mod 2 = 1;
                                end) do
begin
 inc(intCount);
 intSum:= intSum + item.IntField;
end;

I was happy enough with the result, but I wasn't happy with the implementation. I wanted something that was, well, more generic. What I ended up doing was wrapping the existing enumerator into one containing a filter. This will work with any generic collection that descends from TEnumerable. I.e. TList, TQueue, TStack, TDictionary and the TObjectXXX variations

To implement an enumerator, a class must have a GetEnumerator function. This returns an object (or a record) that has the Current property and the MoveNext function. Delphi does a fair amount of work behind the scenes to wrap this all up nicely. See The Delphi Geek's series on enumerators here for more background.

Wrapping an existing enumerator meant I could use the existing GetCurrent and MoveNext for accessing the collection. Filtering then becomes as simple as:

function TFilteredEnumerator<T>.MoveNext: Boolean;
begin
  while  FEnumerator.MoveNext do
  begin
    if FPredicate(FEnumerator.Current) then
     exit(true);
  end;
 result:= false;
end;

The full code is as follows:

unit FilteredEnumeratorU;

interface

uses Sysutils, generics.collections;

type
 TFilteredEnumerator = class
  private
   FEnumerator: TEnumerator;
   FPredicate: TPredicate;
  protected
    function DoGetCurrent: T;
  public
    constructor Create(AEnumerable: TEnumerable; APredicate: TPredicate);
    destructor  Destroy;
    function GetEnumerator: TFilteredEnumerator;
    property Current: T read DoGetCurrent;
    function MoveNext: Boolean;
  end;

implementation

{ TFilteredEnumerator }

constructor TFilteredEnumerator.Create(AEnumerable: TEnumerable; APredicate: TPredicate);
begin
  inherited create;
 FEnumerator:= AEnumerable.GetEnumerator;
 FPredicate:= APredicate;
end;

destructor TFilteredEnumerator.Destroy;
begin
 FEnumerator.Free;
  inherited Destroy;
end;

function TFilteredEnumerator.DoGetCurrent: T;
begin
 result:= FEnumerator.Current;
end;

function TFilteredEnumerator.GetEnumerator: TFilteredEnumerator;
begin
 result:= self;
end;

function TFilteredEnumerator.MoveNext: Boolean;
begin
  while  FEnumerator.MoveNext do
  begin
    if FPredicate(FEnumerator.Current) then
     exit(true);
  end;
 result:= false;
end;

end.

To use filtering in action, simply do something like:


for xxx in TFilteredEnumerator<T>.Create(queue, function (Arg1: T): Boolean
                                                        begin
                                                          result:= ...;
                                                        end)
do
  ...


var
 queue: TQueue<string>;
 cur, combined: string;
 filter: TFilteredEnumerator<TTestObject>;
begin
 queue:= TQueue<string>.Create;
  try
...

   for cur in TFilteredEnumerator<string>.Create(queue, function (Arg1: string): Boolean
                                                         begin
                                                           result:= Arg1 < 'A';
                                                         end)
   do
   begin
     combined:= combined + cur;
   end;

If you are deriving from a collection, you could also wrap this into a method:


function TFilterableList.Filter(APredicate: TPredicate<ttestobject>): TFilteredEnumerator<ttestobject>;
begin
 result:= TFilteredEnumerator<ttestobject>.Create(self, APredicate);
end;

Source can be downloaded from here.

Microsoft Study: TDD reduced defects by 40%-90%, increased dev time by 15% - 35%

2009-02-10T13:18:00.001+13:00

Found via Reddit.

See here (pdf).

Garbage collection - it's not about lazyness

2009-01-19T10:02:00.004+13:00

There seems to be a common belief amongst a number of Delphi programmers (and c++ programmers etc) that garbage collection is all about being too lazy to do your own clean-up.

E.g. Babnik asks "What is it about Garbage Collection?" (kudos for actually asking the question) and his post has the implicit assumption that advocates are trying to avoid doing a bit of minor work.

Others have posted far more stronger comments (check the responses to Babnik's post) straight out stating that gc = lazyness, bad coding and a slack attitude to life. I find this attitude condescending and offensive. The reason I want gc, is not because I write bad code, but because i want to write better code.

I would like to see garbage collection in native Delphi, not as a mandatory feature, but as an option. It's not something I am holding my breath for, but if there was one thing I could add, that would be it.

I know I am repeating myself, but here are my main reasons for wanting a gc (expanded from an earlier post on the subject):

Better code: In a non gc language, you end up with a number of idioms and practices to guard against memory leaks. Delphi has several of these.
E.g.:
It is rare to return an object from a function. Typically you would create an object and then pass it to a procedure to be modified.
The use of assign rather than :=
The use of Owner, Owned and the like to solve object destruction problems)
Less code: I performed a naive analysis on my most recent project by removing most .Free calls and the supporting destructors and try … finally blocks. The result was about 4% fewer lines of code. More importantly, the code I removed was boring, boilerplate which solved no business problems and added no value (other than preventing leaks)
Faster development: There are some minor time savings to be had simply by typing less code. However when I was regularly programming in c# I found my productivity improved due to the change of coding style that gc allowed.
Fewer memory leaks: By reducing the need for manually freeing memory, a gc significantly reduces the scope for memory leaks. It is still possible to leak memory, but much harder.

There are some problems that it is difficult to solve without a GC. Linq is often given as an example. Class operators (as opposed to record operators) is another

One of the main reasons I see for Delphi to have a gc, is that it's competitors have it. If we are to persuade people to write applications in Delphi as opposed to C#, Java etc then requiring manual memory is at best a speed bump.

Finally garbage collection definitely falls into the "Your kilometreage may vary" category. If you are working in memory constrained environments then manual management is the way to go. If you are writing database driven business apps or web apps (as I usually am) then manual management offers little or no advantage over a garbage collector.

Zombie Mansion released for the iPhone

2008-12-12T08:21:00.002+13:00

Zombie Mansion is first person shooter game I have been developing for the past few months (and the cause of my receding hairline). It got finally got approved yesterday.

See here for screenshots and more info.
See here for the App store page.

iPhone Dev: "The binary you uploaded was invalid. The signature was invalid, or it was not signed with an Apple submission certificate."

2008-11-23T16:35:00.003+13:00

I finally finished my iPhone game this week (Zombie Mansion). All I had to do on Saturday was upload the it into the App store and spend the rest of the weekend relaxing at the beach.

Not so fast. I have spent all weekend battling the dreaded "The binary you uploaded was invalid. The signature was invalid, or it was not signed with an Apple submission certificate." error. As is often the case with error messages, the message itself was on no use what so ever in solving the problem. It was in fact a positive hindrance as I spent several hours redoing my distribution certificates, provisioning profiles and waving dead chickens (frozen) over my keyboard.

In the end I resolved it by moving my dev folder off the thumb drive (fat 32 formatted) and onto the main Mac partition (Mac OS ext format) and rebuilding.

For the benefit of any other poor buggers in the same straits, here is a list of tips I picked up while investigating this:

Build on a Max OS Est formated drive (you only need to do this for the App store build)
Make sure you have a 57*57 icon called (case sensitive) "Icon.png"
Check the spelling of the code signing identity
make sure your bundle app id is correct
Make sure the app bundle contains the following:
CodeResources
_CodeSignature
embedded.mobileprovision

One think I found helpful was having an ad-hoc version. That let me sort out problems with certificates much more easily than doing continuous uploads.

Links:
iPhone Dev forums discussion: https://devforums.apple.com/message/12311#12311
Zombie Mansion: http://sourceitsoftware.com/zombie.html

Getting line numbers in dunit test

2008-10-13T09:46:00.003+13:00

I forget this every time:(

By default, dunit gives the the address of where your unit tests failed.
To get the line numbers instead, do the following:

Install the JCL from http://sourceforge.net/projects/jcl
In your test project settings, add the conditional define USE_JEDI_JCL (Directories/Conditionals page)
In your test project settings, set Map File to detailed (Linker page)
Rebuild your project

Delphi 2009 = A good test of my backup strategy

2008-09-16T16:44:00.005+12:00

I installed Delphi 2009 today. In a fit of enthusiasm, I installed it onto my Delphi 2007 virtual machine...

That turned out not to be such a smart move. Not only did d09 fail to install (stopping 1/2 way through with a 'failed to find setup.msi' error), but it trashed my d07 install as well. I don't know how bad the trashing was, I killed delphi after the 10th "package not found error" on start up.

I am not blaming CodeGear for this, my D07 install was "customised" to fit on a small virtual machine, with several, possibly important, folders deleted, and with little free space remaining.

One of the many nice things about using a virtual machine is the ease of backups and recovery. In this case, it came down to a 2 step process

Copy fridays backup onto the computer
Use subversion to retrieve all the files changed since friday

Total elapsed time to recover was about 10 minutes, most of which was spent making coffee while waiting for the files to copy.

Moral of the story: If you are installing D2009 (or any other version), put it in a virtual machine.
And use version control
And backup regularly

Programming for the iphone really sucks

2008-08-18T09:45:00.005+12:00

Ogrampray, ergo sum. I program, therefore I am.

I have wanted to program nearly every device that I own (except for video recorders).
Now that I have an iPhone, I want to program that. Unfortunately there are a number of roadblocks in the way...

NDA
The iPhone nda is ridiculously draconian. There are enough posts on the subject that i won't do into details. See here if you want to know how developers feel about it. Basically, you need to figure everything out yourself coz you can't ask anyone else. There are still discussion groups, but they may be gone tomorrow. There is also a series of tutorials at IphoneSdkArticles.com but that may also disappear.

Objective C
ATM, your choice of development language is Objective C or Objective C.
If you are a windows developer, you first question is probably "Wtf is Objective C?". The short answer is that it is yet another version of C with objects, designed by someone with an unholy fascination for square brackets. The average line of code contains slightly more text than symbols, but only just.

ObjC is primarily used on Apple machines, and sits at #42 in the tiobe list, just below Erlang.

Because it is Apple only, the development tools only run on OS X (you can use gcc on windows, see here, but it's not easy). The tools may be free, but you need a $1000 OS X machine to run it on.

The sdk license expressly forbids interpretors, JITers and iPhone based compilers. So the only way to get java or mono is if they develop an ahead of time complier. Here's hoping.

Multitasking
Nope, sorry, you don't need it. Applications run full screen, single window. When the user presses the home button, your app exits. To get back to your app, the user needs to start it up all other again. This immediately rules out a large number of interesting applications, and adds a certain amount of complication to development. As an aside, Windows Mobile does exactly the opposite and minimises applications rather than closing them, so they can reopen more quickly. This approach is also arse.

Application sandbox
Each application is stored in a single folder. All files, settings and related documents are stored within that folder. The application can only access the contents of their folder. There is no concept of a user documents folder. There is also no simple way to get documents onto the phone for use by your app.

Example: I have an ebook on my computer. I would like to read it on my phone (I have kids, I spend a lot of time sitting in the car waiting). On a windows mobile machine, a palm, or even my old Psion S5, you copy the file over onto your machine, and open it. On a iPhone, it's not so easy, Using ereader, I need to upload the ebook to their website, and then download it again on the iPhone. Alternately, they helpfully suggest, I can run a web-server on my PC.
On the other hand, the Stylus and Bookshelf book readers provide desktop software (50meg download, written in java) that will let you copy documents over using wi-fi. If I ever need to transfer documents when I don't have a wi-fi router, I am in for a large amount of aggravation.

So, if you are writing a app that needs to work with documents, you also need to write a client/server file transfer application, in a different programming language, just to get you document where you can use it!

Distribution
As a means of getting your application to a large number of paying punters, App Store is not too bad. The 30% commission is high compared to Paypal, Regnow, SwReg etc, but low compared to phone/pda specific sellers such as Handango. However as a means of distributing your app to a specific group (ie company wide as opposed to world wide) it is less useful.

Positives
There are some good points about iPhone development though.

The hardware is essentially the same (+/- 3g, gps) on every device. This is a significant contrast to Windows Mobile where the screen size and orientation can change, there may or may not be a touch screen, camera, gps, wi-fi, internet etc).
Appstore makes purchasing applications dead easy. I suspect that iPhone users will end up with more applications than windows mobile users.
The development tools are quite polished, and cheap if you already have an Apple machine.
The phone has lots of useful functionality. GPS, wi-fi, accelerometers, camera etc. I can't think of any other phone that has sold as well and has as many toys to play with.
The xcode development tools can also deal with C++ and C code, although the sdk is in Objective C.

If you are a Apple developer, developing for the iPhone is a no-brainer. For a windows developer, it is a much harder decision.

Links
NDA comments
SDK Discussion group
iPhone SDK Articles
Wikipedia on objective C
iPhone Development on Windows

Subversion add-ins for Delphi

2008-08-15T07:42:00.004+12:00

Part 4 of a n part series on Subversion and delphi

Subversion add-ins
As we have already established, TortoiseSvn adds version control functionality to windows explorer. This is all good, and the result is quite functional. Many people need nothing more. However for the rest of us, there are a number of ways to integrate subversion into delphi.

File Browser
If you have a recent version of delphi, and have TortoiseSvn installed then you get integration for free. The File Browser window (View menu -> File Browser) uses the Windows Explorer context menu so all TortoiseSvn commands also work with the File Browser

JCL Version
The jedi control library (jcl) includes a Tortoise (Svn and CVS) add-in that adds a JCL Version menu. Most functionality is available, although I have never managed to get Diff working. It requires Tortoise to be installed. If you are already using Tortoise and the JCL, it's a bit of a no-brainer.

Others

There are a few others that I haven't used.

Delphi Svn is available from SourceForge. It looks like it doesn't need Tortoise to be installed.
The gloriously named delphiaddinfortortoisesvn is available from tigris.org, home of TortoiseSvn. As you might expect, it requires Tortoise to be installed.

Links
Tortoise
Jedi Control Library
Delphi Svn
Delphi Addin (Tigris)

Common tasks with Subversion

2008-07-28T10:00:00.004+12:00

Part 3 of an n part series.

General workflow for version control

The following descriptions all use TortoiseSVN and Explorer. The steps can also be done using other Subversion clients such as DelphiSvn or JCL.

The general workflow for changing code is:

Update. Right click on your base development folder and choose TortoiseSvn -> Update.
Resolve conflicts
code like a demon...
Test, test and test
Update (again). As 1, just in case someone else has changed things
Resolve conflicts (again)
Add (also Delete and Ignore as required)
Commit

Notes:

Solo developers can normally leave out steps 1,2,5,6
Steps 7 and 8 can be combined (check the "Show unversioned files" box in the commit dialog)

For a solo developer, the process essentially comes down to

Commit. Do Adds, Deletes etc from commit dialog

Now isn't that easier than zipping up all your source code and archiving it.

Traps for young players

Deletes and (particular) Renames should be done from within Explorer/Tortoise.
Do commits from the base directory where possible. If you commit further down the tree, then you need to do an update before you can commit from the base.

(Edit) Team Version Control
One thing I didn't point out first time round (thanks Lars).
The golden rule of team version control:
Only check in working code to the trunk.
If you are working it a team environment, and you want to commit non working code then create a branch. That will let you have all the benefits of version control without annoying everyone you work with. Once you have completed your changes, merge your changes back into the trunk. See the manual for details.
Also, Sean's rule number 2 (for team programming):
Always update before you commit ()
If you commit without an update first, the code in the repository may not be the same as your code. That means that you don't know that the tests pass, and the code still compiles.

E.g. Bob changes the signature of DoSomething to take an extra parameter. It's in a file that you haven't touched, so there are no conflicts. All of your code uses the old signature so when you commit, it won't compile any more.

If you update first, you can catch the problem and avoid making a fool of yourself. This sort of scenario can be quite common with larger teams. Don't ask me how I know.

Manual
All Tortoise commands are available from the Explorer context menu. See here for the manual.

Subversion server options

2008-07-26T15:22:00.003+12:00

I didn't cover subversion servers in my last post as I don't use one any more. However there have been some queries so here are a few ideas:

Windows installers
VisualSVN Server
official Svn 1-click setup
Other OS

VMWare appliances
Subversion and WebSVN on Ubuntu Server
SVN VM
vmTrac

Online hosting
Cvs Dude
Google search

I am not endorsing any of these. However, if I needed a server, I would probably go with one of the online solutions.

Starting out with Delphi and subversion

2008-07-07T09:22:00.013+12:00

I am always disturbed by the number of programmers I run into who don't use version control. A version control system (VCS) is one of the fundamental tools of a programmer, up there with the ide/compiler, bug tracking and backups.

Subversion, together with the TortoiseSvn plugin for Explorer, is one of the better free VCS solutions available. What follows is a quick guide to setting it up the easy way.

Note: This setup is not suitable if you have a large number of users, or need to provide access across the internet.

Tortoise and Subversion

Subversion is the actual VCS. Typically you would run it as a service and then connect to it with client software such as Tortoise.

Tortoise adds context menus and icon overlays to Explorer allowing control of the VCS. These menus and overlays are also available in programs that use the Explorer menus and icons such as the Delphi File Explorer.

Other clients for Subversion are also available.

If you only have a limited number of users, and do not need remote access, you can use Tortoise without a subversion server. This makes set-up much easier. I am not going to cover setting up the server here, primarily because I haven't used one for 3 years.

My file structure

I have 2 base folders, D:\dev and d:\devother.

D:\devother\ is used for files that are use different version control settings (ie open source projects that I update from their repository such as tiopf).
D:\dev\ is used for everything else; my code, third party components, images etc. Having everything under one folder makes version control easier.

I recommend moving all source under a single base folder, if it is not already. If you have source on a network share, move it into a base folder on a local drive. If you need to share source, place the repository on a shared folder.

Repository

The repository is where the versions are stored. It has a directory structure similar to a file structure.

The subversion book recommends having the following directories at the repository root

/branches/
/trunk/
/tags/

With this layout, your main development takes place under /truck/. Branches and tags go where you would expect. I don't do a lot of branching and tagging, when I do I place the branch directories along side the main project directories instead. Therefore I don't bother with the initial directories and create branch directories as required.

My layout is more like

/components/
...
/projects/
/projects/project 1 v1/
/projects/project 1 v2/
...

This maps to my file structure of

d:\dev\
d:\dev\components\
...
d:\dev\projects
d:\dev\projects\project 1 v1\
d:\dev\projects\project 1 v2\
...
etc

It is reasonably easy to change from one repository layout to another if you change your mind, so choose whichever structure makes sense to you..

Setup

Download and install Tortoise from here.
Open Windows Explorer
Create an empty folder to serve as your repository. This can be local or on a network drive. The location needs to be reasonably safe and easy to back up.
Right click on the folder and click on "Create repository here"
Choose "Native filesystem"
If you want to set up /branches/, /tags/ and /truck/ directories, right click on your repository folder and choose TortoiseSvn -> Repo-browser. Right click on the root directory and choose "Create Folder" for each of the initial directories.
Right click on any folder and choose TortoiseSvn -> Settings
In the Global ignore patterns, enter in the file types that should be ignored by default. Ie things like map files that will (almost) never need to be under version control. I use "*.dcu *.~* dcu temp *.exe *.zip *.bkm *.ddp *.cfg *.dof *.dsk *.ini *.hlp *.gid *.bmp *.png *.gif ~* *.log bin debug release *.map *.chm *.bkf Thumbs.db *.mdb .obj *.elf *.stat *.ddp *.bpl *.map *.GID *.hlp *.opt *.dll *.raw *.BIN *.obj *.pdb *.scc Debug Release *.xml obj *.~* *.backup *.INI *.ArmLog *.KeyLog *.NanoLog *.Stats *.PreARM *.old *.drc *.*~ *.doc *.pdf *.bmp *.jpg *.MRW *.NEF *.ORF *.psd *.X3F __history *.local *.identcache *.bak Thumbs.db *.ldb *.dex *.rar DllDcu *.lck CVS cvs *.txt *.TXT *.jdbg *.HLP *.KWF *.xls *.cnt *.dsm *.dti *.tmp *.lnk *.cbk *.mes"
Note that the patterns are case sensitive.

Initial Import

Back up your source code!
Right click on your base development folder (I use d:\dev\) and click on "SVN Checkout"
Click on the browse button beside URL and navigate to the repository directory (if you have set up a /trunk/ directory, navigate to this).
Right click on your base development folder (d:\dev\ in my case) and choose TortoiseSVN->Add.
Wait for the Add dialog to populate. The first time it is used, it can take a while to populate. Work your way down the file/folder list.
If there is something you don't want to add now, uncheck the selection box.
If you never want to add it, right click on the item and choose "Add to ignore list"
Once you are happy with the selected files and folders, click OK.

The files are now all added to version control. However they have not yet been saved (committed).
Right click on the base development folder again and choose Commit.
Provide a message.
Uncheck any files you don't want to commit at the moment and click OK.

Your files are now under version control. You can revert to any version, perform diffs, see what files are changed and perform all other sorts of good things.

Version controlled files are marked to indicate their status. A full list is given here. The main ones are

Committed: Check mark
Added: +
Changed: !
Deleted: x

To come
I will do followup posts explaining

How to work with files on an ogoing basis
Various delphi integration options

Links
Subversion http://subversion.tigris.org/
Subversion book http://svnbook.red-bean.com/
Tortoise http://tortoisesvn.tigris.org/

ti Object persistance framework updated

2008-06-21T09:29:00.003+12:00

What's up?

v2.50 of tiOPF is now available at http://tiopf.sourceforge.net/

So what's tiOPF?

tiOPF is a Object Persistence Framework. That is, it is a framework based around saving your objects to, and loading them from, databases and/or flat files. See the overview for more details.

In a nutshell it lets you do things like:

var
user: TMyuser;
userList: TUserList;
...

user:= Tuser.CreateNew;
user.FirstName:= 'Sean';
user.LastName:= 'cross';

user.Save;

...
userList:= TUserList.Create;
userList.Load;

for user in userList do
...

tiOPF handles the saving and loading of objects to databases and flat files. You can swap between databases by initialising a new persistence layer.

Why should I care?

tiOPF lets you code in objects rather than datasets (you can still use data aware controls though). This gives you more object orientated code
tiOPF provides database independance
Better code reuse. Because the persistance is separated out, you can use the same code across different databases and structures
Easier unit testing (in my experience anyway). I find it much easier to setup objects and test them that to setup databases and test.

Links

Home page: http://tiopf.sourceforge.net/
Overview: http://tiopf.sourceforge.net/Doc/overview/index.shtml
Newsgroups: http://tiopf.sourceforge.net/Support.shtml

" Connection is busy with results for another hstmt"

2008-06-12T14:35:00.004+12:00

One of the problems with working on the same program for years, is that you always end up paying for your sins. In my case the sin in question is still using the bde and odbc.

"Connection is busy with results for another hstmt" is a common error when connecting to ODBC databases. It occurs because an OBDC connection can only have one active cursor at a time. By default, odbc only retrieves the first 20 records for a query. If the query contains more than 20 records, the rest are retrieved on demand. That's all very well, but when you open a second query using same connection/session you get the dreaded hstmt error.

I thought I had beaten it years ago, but my latest set of changes have resulted in the error reappearing.

There are a few possible solutions to this error:

Use TTable components. These don't have the error as they open a new connection each time. Not a very pretty solution :(
Put a FetchAll after each Open. This forces the retrieval of all records. Not practical in my case as I have around 250 tquery components to check.
Use multiple connections. Also not practical for me.
Replace the BDE with something else that doesn't have the problem. In progress but...
Cheat. Set the odbc rowset size to a larger number. If you set it to 200, then the first 200 records will be returned. If you set it to -1 then ALL records will be returned. This could have dire effects on performance if your queries return a lot of records.

This time round I discovered, and went with option 5.

To set the rowset size with the BDE, do the following:

Click on your TDatabase component
In the object inspector, expand Params
Put in a Key of "ROWSET SIZE"
Put in the desired value

Changing the rowset size should work with other odbc connection components as well.

Update: As Otto has pointed out, the error can also be resolved, if you are using sql server 2005 or later, by using the SQL Native Client drivers. See Multiple Active Record Sets for more details. Doesn't help me though :(

Coverflow update

2008-06-08T20:41:00.006+12:00

TMS GUIMotions

TMS Software have released their 3d animation component, GUIMotions. It includes a number of animations including a coverflow effect. More info is available here.

My GLScene example

I have updated my coverflow example. I have included mirroring, multiple rows, vertical layout and a few other changes making easier to be reused. I have also shown how to use standard vcl controls.

You can download the latest version from here.

Using an editable TMemo control

2x2, Transitioning from horizontal to vertical layout

Vertical layout, 2 across

Horizontal layout 2 across

Links
GuiMotions
GLScene coverflow demo
GLScene

Coverflow example using Delphi and GLScene

2008-05-16T09:24:00.010+12:00

Problem
I am beginning work on Pics Print 4. One of the things I want to do with this release is make it significantly more slick. To that end, I thought "Coverflow". See here for an example

The trouble is that there is not that much in the way of delphi examples. I did find "Flying Cow", which is a Delphi implementation using OpenGl.

Flying Cow
Flying Cow is a clone of Cover Flow written by Matías Andrés Moreno (http://www.matiasmoreno.com.ar/ website currently down)

Changes to make Flying Cow compile under D2007, and download links, are listed in http://www.aqua-soft.org/board/showthread.php?t=46566 (website currently down as well)

Flying Cow looks quite nice. However it has one huge drawback from my point of view, it uses the GPL and is therefore unsuitable for my use.

Flying Cow implementation

GLScene implimentation
I have created my own version using GLScene. Considering that I haven't done any 3d work since 2005 (on a pda using C++ at that), it was remarkably easy to get going without cracking open my OpenGL books.

Here is the result:

Basic layout

Showing 2 images across, and using transparency on mouseover to show hidden images.

Now with added mirroring goodness

Now with vertical layout, showing 2 across and 2 down

You can download my source and the executable from here. It is written in D2007 but should be easily portable to earlier version (there may be a for ... in loop or two but that is about it).

This is prototype code, not production code so be warned. I take no responsibility if it eats your homework, backchats your mother-in-law or transfers all your money into my bank account. On the plus side, it has a promiscuous license so you can use it in your own apps without problems.

If you make any improvements, let me know and I will update source accordingly.

Update
Mirroring added

Update 2 - May 30
Code changed to use an object list instead of an array, removed Graphics32, given optional vertical layout, allows multiple rows, and demonstrates removing pages.

Links
Javascript implimentation
http://flashloaded.com/flashcomponents/photoflow/example1.html

Flying cow discussion thread
http://www.aqua-soft.org/board/showthread.php?t=46566

GLScene
http://glscene.sourceforge.net/wikka/HomePage

My version
http://www.sourceitsoftware.com/download/delphi/CoverTest.zip

Database versioning part 2

2008-04-22T19:45:00.003+12:00

In part 1, I discussed database versioning with MS SQL Server. SQL Server has robust tools available that simplifies the process of differencing and creating upgrade scripts. DBISAM doesn't have these tools, but using datamodules and table components makes versioning close to painless.

Situation 2
Delphi app
DBISAM database
1 developer

Rental Property Manager uses a DBISAM database (DBISAM v3 in RPM 1, DBISAM v4 in RPM 2). In the 4 years since it was released, it has been through about 20 changes.

Unlike sql server, there is no database comparison utility to generate upgrade scripts. That means things have to be done a bit differently.

I still use the same general process as for sql server:

Record the database version number in the database
Define the expected db version in the application
Keep a master db for comparison
Only make one set of changes at a time
Automate differencing as much as possible
Unit test, test and test again. The tests should fail the moment a table is modified.

Because dbisam doesn't support views, the version number is stored (along with a bunch of other info) in an ini file in the database directory.

I have a datamodule, TdmodCheckDatabase. This has a TdbisamTable component for every table in the database. The table component contains all fields in the table and is updated whenever the table is changed.

To make database changes, the following process was used:

Increase the version number in the application
Make and test DB changes.
Update the affected tables in TdmodCheckDatabase
If necessary (rarely) add further upgrade queries to TdmodCheckDatabase. E.g. to set the values of new fields, or to add new data rows.
Generate a CreateDatabase unit script using the supplied database tools.
Update unit tests to suit the new db

When the application is run, it goes through the following process

If no database is found, then run CreateDatabase unit and then do step 3
Get the current version number from the database ini file
If it is less than the expected version number then
Run CreateDatabase (to create any new tables)
Check every table component in TdmodCheckDatabase
Apply any table changes
run any manual upgrade scripts
Update the version number in the database ini file.

In code this is:


class procedure TdmodCheckDatabase.UpgradeDatabase(databasePath: string; currentVersion, newVersion: integer);
var
module: TdmodCheckDatabase;
f: integer;
begin
module:= TdmodCheckDatabase.create(nil);
try
  module.OpenDatabase( databasePath );

  for f:= 0 to module.ComponentCount -1  do
  begin
    if module.Components[f] is TDBISAMTable then
    begin
      try
        // if we need to upgrade table to dbisam 4
        if currentVersion <= DB_VERSION_FOR_DBISAM4 then
          TDBISAMTable(module.Components[f]).UpgradeTable;

        module.UpgradeTable(TDBISAMTable(module.Components[f]));
      except
       // logging and error stuff removed
      end;
    end;
  end;

  for f:= currentVersion + 1 to newVersion do
    module.RunUpgradeScripts(f);

  module.sqlMakeIndexes.ExecSQL; // have to create additional indexes manually
 finally
  module.DBISAMDatabase1.Close;
  module.free;
end;
end;


procedure TdmodCheckDatabase.UpgradeTable(table: TDBISAMTable);
var
 fieldIndex: integer;
 needsRestructure: boolean;
 canonical: TField;
begin
 needsRestructure:= false;

 table.FieldDefs.Update;

 // add any new fields to the FieldDefs
 if table.FieldDefs.Count < table.FieldCount then
 begin
   for fieldIndex := table.FieldDefs.Count to table.Fields.Count -1 do
   begin
     table.FieldDefs.Add(fieldIndex + 1, table.Fields[fieldIndex].FieldName, table.Fields[fieldIndex].DataType, table.Fields[fieldIndex].Size, table.Fields[fieldIndex].Required);
   end;
   needsRestructure:= true;
 end;

 // make sure we have correct size for string fields
 for fieldIndex := 0 to table.FieldDefs.Count -1 do
 begin
   if (table.FieldDefs[fieldIndex].DataType = ftString) then
   begin
     canonical:= table.FindField(table.FieldDefs[fieldIndex].Name);
     if assigned(canonical) and (table.FieldDefs[fieldIndex].Size <> canonical.Size) then
   begin
     // field size has changed
     needsRestructure:= true;
     table.FieldDefs[fieldIndex].Size:= canonical.Size;
   end;
   end;
 end;

 if needsRestructure then
   table.AlterTable(); // upgrades table using the new FieldDef values
end;

procedure TdmodCheckDatabase.RunUpgradeScripts(newVersion: integer);
begin
 case newVersion of
   3: sqlVersion3.ExecSQL;
   9: sqlVersion9.ExecSQL;
   11: begin  // change to DBISAM 4
         sqlVersion11a.ExecSQL;
         sqlVersion11b.ExecSQL;
         sqlVersion11c.ExecSQL;
         sqlVersion11d.ExecSQL;
         sqlVersion11e.ExecSQL;
       end;
   19: sqlVersion19.ExecSQL;
   20: sqlVersion20.ExecSQL;
 end;
end;

Unit tests included:

Make sure the current version is correct
Make sure that every table and every field exists
Create a new blank database (for a number of different versions) and work though the upgrade process to make sure the final database is correct.
Restore an existing older database with data and upgrade to the latest version

With this process, altering the database structure is trivial for most changes. Adding fields and table usually requires no more work than updating the table components and generating a new creation script .

The current implementation does have a couple of restrictions in that it won't remove tables or fields. However if that is required, it won't take long to add.

Database versioning part 1

2008-04-17T14:19:00.004+12:00

Versioning databases is one of those ongoing problems that has no one-size-fits-all solution. There are 2 solutions I have developed and used successfully.

The general process I use each time is:

Record the database version number in the database
Define the expected db version the application
Keep a master db for comparison
Only make one set of changes at a time
Automate differencing as much as possible
Unit test, test and test again. The tests should fail the moment a table is modified.

Situation 1
MSDE (= SQL server) database
8 developers (pair programming)
C# application

Every developer had 2 databases, Unit Test db and Application Test DB (UnitDb and AppDb from now on). With msde (now sql express) there are no licensing costs to worry about.
There was also a Master database stored on a central server that served as the canonical reference.

The database version number was stored in a view ('select xx as VersionNumber).

To make database changes, the following process was used:

Check out the latest version of the app
Increase the version number in the application
Make and test DB changes. Usually this was done in UnitDB. AppDb was kept synchronised using SQL Compare
Update the version number in UnitDB
Generate an update script using Sql Compare. Scripts were named UpgradeXXX.sql where XXX is the version that was being upgraded from.
Generate a CreateDatabase.sql script (for shipping with the app) and a CreateDatabaseXXX.sql (for unit tests only) script. In this case XXX is the version that will be created. The 2 scripts are the same except for the name.
If necessary (rarely) append further queries to the scripts. E.g. to set the values of new fields, or to add new data rows.
Update unit tests to suit the new db
Check in changes

When the application is run, it goes through the following process

If no database is found, then run CreateDatabase.sql
Get the current version number from the database
If it is less than the expected version number then
run UpgradeXXX.sql and go to 2

When the unit tests are run, the first step is to upgrade UnitDb to the current version using the same process as the main application. That means that other peoples changes are automatically applied. The unit tests included:

Make sure the current version is correct
Make sure that every table and every field exist (this doesn't always need to be explicit as the persistence unit tests should pick up any problems here).
Create a new blank database (for a number of different versions) and work though the upgrade process to make sure the final database is correct.

The secret weapon in all this was sql compare which makes generating the scripts quite straight forward.
Also the upgrade scripts can do more than one set of version changes. ie Upgrade001.sql could upgrade the version to v10 so that scripts 002 - 009 don't need to be run.

Situation 2 (delphi and dbisam) will follow in a later post.

Links
Sql Compare

VMWare tips

2008-03-02T20:42:00.004+13:00

Wireless networking
By default vmware networks are set up as bridged. However this doesn't work with Vista and wireless networks.

The easy answer is to change the network setup to NAT. However if you are doing server development this may not be much use. In that case, you can try network bridging as described here.

Resizing the vm image.
The easy way is to download the vm converter from here. Amoungst other things (creating an image from an existing computer) this can let you resize an image.

Performance
If you have sufficent system memory, change VMWare settings to fit all virtual machine memory into reserved host ram (Edit menu -> Preferences, memory tab). This will give a performance boost.

Also, preallocate the hard drive space.

Links
VMware Knowledgebase article - http://kb.vmware.com/selfservice/dynamickc.do?cmd=show&forward=nonthreadedKC&docType=kc&externalId=1212&sliceId=2&stateId=0%200%209345265
Bridging Process http://mingstert.blogspot.com/2007/12/vmware-wireless-network-adapter-and.html

VM Converter http://vmware.com/products/converter/

TIOPF - New persistance layer walkthrough

2008-02-26T10:50:00.007+13:00

TIOPF

I use tiOPF for most new Delphi database applications.
tiOPF is a Object Persistence Framework. That is, it is a framework based around saving your objects to, and loading them from, databases and/or flat files.

See overview for more details on tiOPF.

Persistence Layers (PL)
TIOPF uses persistence layers to save/load objects. If you want to use an unsupported database, you just write another PL, and the required unit tests. The following is a walkthrough the process of creating and testing a new PL.

Corelab - SDAC
I use Corelab's Sql Server Data Access Components for talking to MS Sql Server databases. This is faster than the BDE and ADO components I have used previously. However there is currently no PL for this...

Step 1 - Base point
Writing a PL involves the oldest form of code reuse - copy/paste.
Find an existing PL similar to what you want. In this case, I will use ADOSQLServer. The ado sql server layer is slightly more complicated than most as it uses an abstract class shared with ado access. However this shouldn't be a problem.

Because I am using a very similar persistance layer, the required changes are quite limited. In many cases, I would need to alter the Connection string handling as well, amd perhaps the multithreading support.

I need to copy the *ADOSQLServer files to *CrLabSDAC files, and then make the following substitutions:

TMSConnection replaces TADOConnection
TMSQuery replaces TADOQuery
CrSDAC replaces ADOSQLServer
CrAbs replaces ADOAbs

+ the various uses replacements

Using delphi's 'Find it Files', I searched for ADOSQLServer in the ..\tiOPF2\Trunk\ directory and subdirectories. It pulls up the following files:

tiQueryADOSQLServer (which uses tiQueryADOAbs)
TTestTIPersistenceLayersADOSQLServer
tiTestDependencies
tiConstants
tiOPFManager

tiQueryADOSQLServer.pas is saved as tiQueryCrSdac.pas
tiQueryADOAbs.pas is saved as tiQueryCrAbs.pas
Both of these are saved in the \Options\ directory

tiOPFADOSQLServer_TST.pas is saved as tiOPFCrSdac_TST.pas
This is saved in the \UnitTests\Tests directory.

Step 2 Persistence Layer Changes
In tiQueryCrAbs, I made the following changes:

In the uses I replace ,ADODb with ,DBAccess, MSAccess
replace TADOConnection with TMSConnection
replace FADOConnection with FMSConnection
replace TADOQuery with TMSQuery
replace FADOQuery with FMSQuery
replace TtiQueryADO with TtiQueryCrSdac
replace TtiDatabaseADOAbs with TtiDatabaseCrAbs
replace cErrorADOCoInitialize with cErrorCrCoInitialize
replace EADOError with EDAError
replace cTIPersistADOSQLServer with cTIPersistCrSdac
delete cDelphi5ADOErrorString = '...';

In tiQueryCrSdac, I made the following changes:

In the uses replace ,ADODb with ,DBAccess, MSAccess and tiQueryADOAbs with tiQueryCrAbs
replace ADOSQLServer with CrSdac
replace TtiDatabaseADOAbs with TtiDatabaseCrAbs
replace TADOTable with TMSTable
Delete the line TtiQueryCrSdac = class(TtiQueryADO);

In tiConstants, I added the line
cTIPersistCrSdac = 'CrSdac';

In tiOPFManager, I added the line
{$IFDEF LINK_CRSDAC} ,tiQueryCrSdac {$ENDIF}
below
$IFDEF LINK_BDEPARADOX} ,tiQueryBDEParadox {$ENDIF}

This means that the CR PL can be compiled in to applications by using LINK_CRSDAC in the conditional defines instead of adding the unit tiQueryCrSdac to the application. This makes running the unit tests much easier as my local copy of the standard tests just need the define added

Finally, I added the LINK_CRSDAC define to the DUNINTTIOPFGui appliciation and did a build.

Some errors turned up in constructor TtiQueryCrSdac.Create; so I commented them out for now. I found a few more compile errors as well due to the differences between SDAC and Ado components

replace ExecSQL with Execute
replace Parameters with Params
replace TParameter with TMSParam
replace CommitTrans with Commit
replace RollbackTrans with Rollback
replace BeginTrans with StartTransaction
replace ConnectionString with ConnectString

Finally it all compiles. I ran the unit tests to make sure I haven't broken anything yet. I shouldn't have (yet) but ...

8 minutes later, 1729 tests are run and passed.

Step 3 Unit Test changes

In tiOPFCrSdac_TST, I made the following changes:

replace ADOSQLServer with CrSdac

In tiTestDependencies, I made the following changes:

added ,tiOPFCrSdac_TST after ,tiOPFADOSQLServer_TST
added tiOPFCrSdac_TST.RegisterTests; after tiOPFADOSQLServer_TST.RegisterTests;

Update: Add the following define to the unit test properties: LINK_CRSDAC

This now adds unit tests for the CrSdac PL. It will run the same tests as the AdoSqlServer layer. If necessary, I could override and alter the tests to accommodate database changes.

Compile and run again.

This time there are numerous errors. In part, this is because this layer doesn't implement CreateDatabase and no default database has been defined.

I ran the tests again. In the Setup dialog, I clicked on the [Local Settings] button. I added the following lines to the ini.
[DB_CrSdac]
DBName=localhost:tiopf
UserName=NULL
Password=

This resolves most errors, leaving only 5. I won't step through the process of fixing them. They came down to:

SDAC truncates long strings to 8000. This needs further investigation
SDAC GetTableNames returns the owner as part of the name (eg 'dbo.MyTable') which needs to be removed.

Step 4 Real data test
In the unit tests for my real application, I added then tiQueryCrSdac unit. I can now swap persistence layers by altering the connection string used in my application.

Running the tests raises a couple more errors which are resolved by changing some of the SDAC query options. Once that is done, and the tests all pass, I ran my application against some test data, and real data. All works well.

Step 5 Build a patch file

Links
TIOPF http://tiopf.sourceforge.net/
Overview http://tiopf.sourceforge.net/Doc/overview/index.shtml
Corelabs http://crlab.com/

Remote access to computers

2008-02-01T10:05:00.001+13:00

I spent a fair amount of time using one computer to look at/control another across the internet. Over the past 5 years I have evaluated a number of products. The following are some of my favourites.

Remote Desktop
This is the best option across a wan, giving good performance. However it is a pain in the proverbial to set up for use across the internet, as the ports are frequently blocked by firewalls.

LogMeIn.com
LogMeIn is the one I use must frequently, for controlling my personal computers. A free account will let you control a reasonable number of computers (5 I think). The paying version has additional features such as file access, sound and printing. I have used the free version for about 5 years.

Install is straight forward, go to the web site, log in, install software and go. The software takes care of firewall and NAT issues in nearly every case. In 5 years, I only found one location where I couldn't get connectivity.

There is software available for nearly everything, I have even used my windows mobile cellphone to control my pc.

It's account based, so it is good for computers I own. It is not so good for other computers where I don't have physical access beforehand as I obviously don't want to pass my account details around. Once installed, the client pc can be unattended.

GotoMyPc is an alternative, but they don't have a free version.

CrossLoop
I use crossloop (no relation) for remote user support. They download and install crossloop and I do the same. They click on the share button, send me the access code, and then I connect using the same access code. Performance is not as good as LogMeIn, but it is usually adequate.

It does require a user on the client machine to run the software, hit the share button and provide the access code. It's free to use.

CoPilot is an alternative which is probably better for Grandma and technically challenged users. It's free on weekends, and $5 a day during the week.

Links
LogMeIn
GotoMyPC
CrossLoop
Copilot

Garbage collection: Performance test

2008-01-21T12:38:00.001+13:00

Following my initial GC post, I received feedback regarding my comment "A well written and tuned garbage collector can be faster than manual allocation.". These comments can be summarised as "show us the proof".

I asked on the Boehm GC mailing list (if in doubt, ask for help). The conversation starts here.

They provided the following (my summary):

One benchmark is here, showing that speeds are comparable given sufficient memory (a gc will require more memory) .
Another is here from Hans Boehm's presentation. See pages 50 onwards. He comments that it is a toy benchmark, on linux.
Malloc implementations have improved
Code that favours manual allocation
Simple create, do something, free
Large objects
Code that favours gc
Complicated lifetime management
try ... finally, free
multi threading

Well that kinda helps. But what about in delphi?

I have done some quick tests using my modified version of the delphi wrapper for the Boehm GC (Delphi GC for short). The modifications shouldn't make any major difference to the result.

Delphi benchmark 1:
This is a simple, trivial, benchmark. It creates 60,000,000 small objects and assigns a value.

The object is simply:


TTestObject = class
public
Lines: TStrings;
constructor Create();
destructor  Destroy; override;
end;

and the test is simply

for f := 1 to TestCount do
begin
testObj:= TTestObject.Create;
{$ifdef USE_GC}
testObj.Lines.Add('aaa');
{$else}
try
testObj.Lines.Add('aaa');
finally
testObj.Free;
end;
{$endif}

The try ... finally free section is not required by the GC version as we don't have to worry about memory leaks.

The GC tests were repeated with a range of initial heap sizes and on different computers. The FastMem test was also tried without the try finally. The source code is available if anyone wants it.

The results are

	Old laptop, 512mb	Core 2, 2gig	Single core 2 gig	Quad core 3 gig
FMM (no try finally)		approx 31.5
FMM try finally	81.281	33.306	37.875	48.046
GC 0mb		73.181	59.047	46.25
GC 5mb		39.499	32.906	29.656
GC 10mb	60.891	30.857	29.422	27.984
GC 20mb	58.328	26.926	27.437	27.062

Given a large enough initial heap, the gc version ends up faster than the FastMM version.

This is not a serious benchmark, but it does indicate that a gc can be faster than manual allocation.

Delphi benchmark 2

For this, I added the gc to 2 of my existing unit tests. It was a 2 line conversion, I just added the gc and set the initial heap size.

Enable is a work injury management system. It is heavy on database access and single threaded.
Envisage is a document management system. Database access is done via the tiopf object persistence framework. It reads pdf files, checks for bar codes and creates new ones. It is multi-threaded. It uses a large amount of memory.

Here are the results:

	Envisage, no threads	Envisage, threaded	Enable
FMM	70	114.4	16.4
GC 20	74	119.0
GC 40	71	117.5	14.1
GC 100	71	115.6

Conclusion
Based on these results, I would have to say that my comment "A well written and tuned garbage collector can be faster than manual allocation." is correct. Given sufficient heap space, the gc version is faster in some tests, and 1% slower in others.

Le me restate my conclusion as the initial one is not well worded in terms of what I intended to say. A better conclusion would be "It is possible for a garbage collected application to run at a speed similar to that of an application using manual deallocation". Or alternately, "adding a gc to an application doesn't automatically make it incredibly slow".

The gc performance could probably be improved further by surfacing the gc tuning options, improving the delphi wrapper and using a later version of the GC. The unit tests could also be sped up by removing the now redundant frees, destructors and try ... finally blocks

The Boehm GC used is an early version 6 (6.2 or so). Version 7 is available from cvs. V7.1 should be released soon.

There are downsides to using a gc, such as increased memory use. It is not appropriate for all applications, especially those with memory constraints. However speed does not appear to be one of those downsides.

Update

In response to a query, yes the garbage collector is running, and collecting the objects. After the initial run (which may increase the heap), the heap size remains static no matter how many times I repeat the test (any of the tests).

I also repeated the FastMM test removing the testObj.Free; line.

It "completed" in 35 seconds. By completed, I mean "used up 1.3gig of free mem, all my 4gig page file and then threw an "out of memory" exception.

Reference
GC Mailing list: Are there any benchmarks comparing the speed of gc v non gc
Garbage collector for Delphi
Boehm GC
Wikipedia article

Garbage collection: Follow up

2008-01-18T12:21:00.000+13:00

Given some of the feedback on my previous post, I thought a follow up would be in order

Performance
One of the most contentious points was my comment that "A well written and tuned garbage collector can be faster than manual allocation". I will cover this in a separate post as it needs more than a couple of lines.

Why would you want to use a GC in delphi
I will cover this in a separate post as well. There is probably little gain in just adding a gc to an existing delphi app (unless it's leaky, but we don't write apps like that). If you are writing a new app based around having a gc in place, then you can do things differently.

Clarifications
Once of the original quotes referred to objects referencing each other not being released. I read this as talking about cyclic references. That is a problem with simple reference counting, but not with a tracing (ie mark and sweep) gc such as used by Boehm, .net and java (1).

A gc is not a silver bullet, nor will it catch all memory leaks. I am not suggesting that it will.

Corrections
One point I forgot to mention. Some gc algorithms will allocate extra memory for flags, counts etc (1). This can push up the memory use compared to manual allocation. However Fast mm 4 (2) also allocations a 32 bit flag ahead of every memory block so it is probably a wash.

Fast mm
Fast mm will not catch all memory leaks. It will catch memory that hasn't been freed when the application exits (2) which is not the same thing.

If you have poor testing coverage, then the untested code can have memory leaks.
Fast mm will not catch objects freed on application shutdown (ie forms owned by the application). A gc won't catch this either.

Fast mm will help with double frees, but it won't help with a/v errors (unless I am missing something, it certainly hasn't helped me). A gc will help with both of those (1).

References
1 The wikipedia article on Garbage collection provides a lot of the background.
2 Fast mm details are taken from http://dn.codegear.com/article/33416

"However most Delphi memory managers request large chucks of memory from windows and then parcel it out to the app on request," See (2) and Nexus MM

The quotes in the original article are taken from the newsgroup thread "Garbage collection"

Garbage collection in Delphi Win 32: Why and how to do it

2008-01-17T12:33:00.000+13:00

I got drawn into a newsgroup thread last month (Delphi and XCode) regarding Delphi and x code. I posted on the existence of a Garbage collector for Delphi (http://codecentral.codegear.com/Item/21646) and spent several days in ongoing discussion. Another recent thread is at Garbage Collection.

This posting is a summary of my views of the subject. I spent 3 years as a fulltime C# programmer (and about 14 as a Delphi programmer) so I have a reasonable amount of experience and without a GC. Based on some of the comments I have been reading, there are a lot of people with more opinion than experience in the newsgroups (no surprises there).

Note: When I refer to Delphi, I mean Delphi win 32. When referring to Delphi.net, I do explicitly.

In general, the debate over garbage collection is already over. I am unaware of any recent language that doesn’t have gc built in. There is even a proposal before the c++ standards board to have a gc in c++. In other words, get used to the idea.

There seems to be a certain amount of dislike/distrust of garbage collection amongst a number of people. So far I have seen comments about laziness and incompetence, spurious analogies to automatic cars and a fair amount of 'I know how to free memory (usually) therefore it must be a good thing'.

Most of the comments are more rhetorical than reasoned arguments:

"Garbage collection leads to sloppy, bloated, inefficient code", Any number of comments on laziness and incompentance: Garbage collection is not about incompetence or laziness. It is perfectly possible to be competent and still prefer a GC.
"destructor calls have to be replaced by strict nil'ing of references": You typically don't need to set objects to nil, although there are some cases where you would want to.
"There's also a problem with objects referencing each other which prevents the items releasing": Not with any recent garbage collector.
"It's inefficient, it slow, and its unacceptable.": Again, not with any recent GC. In many cases a GC app is faster than non-gced.

What is garbage collection?
Garbage collection (GC) is the automatic release of unused memory and objects.

E.g.

MyStrings:= TStringlist.Create;
// try
do stuff here
// finally
// MyStrings.Free; // no longer required
// end;

The try..finally and Free are no longer required as objects are released when (sometime after) they are no longer referenced.

See wikipedia for more info than you ever needed on the subject.

Garbage collection does not release resources (database connections, windows handles, etc). These should be released manually.

Strings, dynamic arrays and interfaces are already garbage collected in Delphi , and virtually everything is in Delphi.net.

Why use garbage collection?

Fewer memory leaks: By reducing the need for manually freeing memory, a gc significantly reduces the scope for memory leaks. It is still possible to leak memory, but much harder.
Less code: I performed a naive analysis on my most recent project by removing most .Free calls and the supporting destructors and try … finally blocks. The result was about 4% fewer lines of code.
Better code: In a non gc language, you end up with a number of idioms and practices to guard against memory leaks. Delphi has several of these.
Eg:
It is rare to return an object from a function. Typically you would create an object and then pass it to a procedure to be modified.
The use of assign rather than :=
The use of Owner, Owned and the like to solve object destruction problems)

There are some problems that it is difficult to solve without a GC. Linq is often given as an example.

What are the disadvantages of a garbage collector?

Memory use: A non gc app can free memory as soon as it is no longer required. A gc however will only free memory once it is satisfied that it is not being used, which could be some considerable time later. Typically the gc will only collect objects when it is feeling some memory pressure. Therefore a non gc app can use less memory than a gc app. However most Delphi memory managers request large chucks of memory from windows and then parcel it out to the app on request, so this disadvantage is largely theoretical.
Speed: A well written and tuned garbage collector can be faster than manual allocation. Unfortunately there is no gc tuned for Delphi. The only available gc is slightly slower than the default memory manager.
Reliance on a gc: Code that is written to take advantage of a gc cannot be readily run without a gc. If your code must work in standard Delphi, then it must be programmed accordingly and not rely on the ex.
Non deterministic finalisation: With a gc, you have little control over when the object is removed from memory. Even when it is removed, destroy is not called unless you have implemented a finalizer.

Why not use Delphi.net
I spent 3 years as a c# and c++ programmer. If I wanted to use .net, I would use c# and have access to all the latest toys such as linq.

The main drawback with .net is the need to distribute a very large runtime library with the application. This may not be a problem with web apps or internal applications, but it can be a large issue with shareware.

A second drawback specific to delphi.net is that your code may end up being used by Delphi win 32. As a result, the recommended approach is to program as if the gc did not exist. That is, to include the .Free calls, the try … finally blocks and the destructors. You end up with the all of the drawbacks of a gc, without the advantages.

A third drawback for me is that many of the libraries and controls I use are not available in Delphi.net.

What about other resources?

If your object holds other resources such as windows handles, database connections etc then you still need to release those. Depending on the object you can do this with .Free, .Close, .Disconnect or similar. .Free will always work the object will be disposed of although the memory won't be released until the gc gets around to it.

How do you use a Garbage Collector in Delphi win32?

The Delphi memory manager is designed to be easily replaceable. A garbage collected memory manger is available from http://codecentral.codegear.com/Item/21646. This is written by Barry Kelly and is a thin wrapper around the Boehm GC library. I have made some changes to make it work better. If there is any interest, I will post the code.