Why (most) Identity Management POCs Suck

For those unaware, POC is an acronym for "Proof of Concept", and consists of a client requesting a vendor to bring their software in to demonstrate ("prove") that their software can perform in their environment as promised ("concept"). The engagement usually lasts for a few days to a week, and typically the vendor foots the bill (if the potential deal is large enough). Most often, POCs are borne out of a conversation between a client and an industry analyst who completed a 2 hour identity management market briefing, concluding that it would be a good idea for the client to host a POC. A few vendors are selected to strut their stuff, and after reviewing which vendor did a better job, a winner is hailed and ultimately awarded the contract.
After being involved in one too many POCs, I've come to the following conclusion: IdM POCs suck. Usually. And here are a few reasons/scenarios that explain why:

1. A typical POC is just a glorified demo. So, wisdom states that if the vendor is integrating their software with your target apps, then it's not just a demo, but undeniable evidence that their software is good for you and your organization. That is both true and false. Just because software can be made to "seem" to work with your apps is not evidence that the integration is robust or production ready. Hacks are very common in POCs, and a lot of what is demo'd at the end of the POC is a lot of smoke and mirrors and doesn't prove that what was completed is production ready, or more importantly, can ever be production ready.

2. The success of the post-POC demo is highly dependent on 2 individuals on the vendor team: the Sales Engineer (the guy who duck-taped together the POC) and the POC-demo-guy. The vendor that has the best duo typically wins the deal, which may not reflect the best software solution for your environment. A good SE can make crappy software work (with his arsenal of scripts and tricks), and a good POC-demo-guy can bedazzle almost any audience. On the other hand, a bad SE can make good software break, and a bad POC-demo-guy can put a lively audience to sleep.

3. A POC is typical technology focused. Most IdM deployments are business process centric. When decisions are made based on the number of out-of-the-box connectors or the long list of supported standards without considering how it all applies to your specific set of business processes, the wrong vendor can be selected.

Well, I'm sure there are more reasons, but I'll leave it at that for now.
But the picture isn't that bleak. There are ways around the problems above, and exciting approaches that will not only make POCs not not suck, but actually make them effective and useful...a topic for another blog entry.