Friday, November 16, 2007
Database Mitosis
"Uh, your database is too big. We need to split it."
Your organization may have bought into a evidence and discovery management tool because it was advertised as the biggest and fastest database on Earth. Sure, the backend database to these applications (SQL Server, Oracle, etc) may have benchmark statistics proving such claims, but what are the practical limits of the software package itself? A severe limitation is the software code that's overlaid on top of the database technology. As a past programmer, I know full well that code can be written efficiently or inefficiently. You can have programming code that cuts to the chase (1+1=2), or code that takes unnecessary leaps of logic (((2 x -10) + 40) / 10 = 2). Strip away the pretty interface and expedience makes all the difference. Let's face it, there's basics that we all need in a tool. If an extra bell or an extra whistle slows things down for you and your review team, it may not be worth the purchase.
Ask the following questions to a potential vendor:
1) How much data can we host in your software package? What's the practical upper limit given x number of users?
2) Should we consider multiple databases from the outset, based on the anticipated volume of data?
3) What affect do multiple databases have on deduplication? multi-user access? consolidated reporting?
Ask these questions from the get-go, and you won't be confronted with splitting a database, after-the-fact, once the performance of your nifty software package slows to an agonizing crawl. You may experience weeks of downtime before you're up and running again with multi-database constraints that you weren't even close to understanding beforehand. ¶ 9:40 AM
Your organization may have bought into a evidence and discovery management tool because it was advertised as the biggest and fastest database on Earth. Sure, the backend database to these applications (SQL Server, Oracle, etc) may have benchmark statistics proving such claims, but what are the practical limits of the software package itself? A severe limitation is the software code that's overlaid on top of the database technology. As a past programmer, I know full well that code can be written efficiently or inefficiently. You can have programming code that cuts to the chase (1+1=2), or code that takes unnecessary leaps of logic (((2 x -10) + 40) / 10 = 2). Strip away the pretty interface and expedience makes all the difference. Let's face it, there's basics that we all need in a tool. If an extra bell or an extra whistle slows things down for you and your review team, it may not be worth the purchase.
Ask the following questions to a potential vendor:
1) How much data can we host in your software package? What's the practical upper limit given x number of users?
2) Should we consider multiple databases from the outset, based on the anticipated volume of data?
3) What affect do multiple databases have on deduplication? multi-user access? consolidated reporting?
Ask these questions from the get-go, and you won't be confronted with splitting a database, after-the-fact, once the performance of your nifty software package slows to an agonizing crawl. You may experience weeks of downtime before you're up and running again with multi-database constraints that you weren't even close to understanding beforehand. ¶ 9:40 AM
Comments:
<< Home
Love your blog Jerry. I've been managing large electronic reviews for about a year and it's great to read someone who also has that "in the trenches" perspective.
I've dealt with a mid-review database split before, and it wasn't pretty. While it can speed things up for users reviewing documents in each individual database, some of the benefits of a review platform can evaporate after a split.
For example, if you can't search across both databases simultaneously, it takes twice as much time to conduct every single search. Almost every advanced function will also be impacted, such as duplicate and near-duplicate identification and concept-based document grouping.
Since as you pointed out, the slowdown is coming from the software and not the backend database, splitting the database usually means working in parallel instances of the platform. And that means whoever is running the project might suddenly find themselves with twice as much work.
Post a Comment
I've dealt with a mid-review database split before, and it wasn't pretty. While it can speed things up for users reviewing documents in each individual database, some of the benefits of a review platform can evaporate after a split.
For example, if you can't search across both databases simultaneously, it takes twice as much time to conduct every single search. Almost every advanced function will also be impacted, such as duplicate and near-duplicate identification and concept-based document grouping.
Since as you pointed out, the slowdown is coming from the software and not the backend database, splitting the database usually means working in parallel instances of the platform. And that means whoever is running the project might suddenly find themselves with twice as much work.
<< Home
