A Better Way to Mine Data.
Posted by Ajay Kelkar on Tue, Oct 13, 2009
The good news, for marketers, is that data mining really can make a difference to most bottom lines. The bad news is that, despite what data mining can do, it is so often used so poorly that it is virtually useless. Companies are today storing huge amounts of data. Companies in the 'Petabyte Power Players' club include eBay Inc., with 5 petabytes of data, Wal-Mart Stores Inc., which has 2.5 petabytes, Bank of America Corp., which is storing 1.5 petabytes, Dell Inc., which has a 1PB data warehouse In many cases, the data is a big part of the problem. Even in the most reputable companies, data is often "dirty,"--out of date or otherwise irrelevant. Most commercially available data mining packages lack the flexibility and functionality that real world marketers need. The problem with "data quality" is ownership. No one seems to own this critical asset! Without doubt , the line functions have to own "data quality". Data quality can only be impacted substantially "at source"-either a salesperson fills up inaccurate information for a customer while he wildly chases his target or an operational group incorrectly data enters a customer record!One of the most frequent and most difficult causes of data quality is culture. If people do not think that data quality is important, it isn't.James Standen makes this interesting point;Data quality starts on the ground. The further from the ground, and the deeper into various operational systems, ETL jobs, staging tables, data warehouses or data marts we try to fix the problem, the harder it will be.http://blog.cequitysolutions.com/ The Numerical Algorithms Group (NAG), an organization that develops software to solve complex mathematical problems, has three suggestions. One, try hiring a mathematician who is a data-mining expert to guide your efforts. Two, consider developing data mining applications in-house using fully documented components (algorithms) from a reliable library. And finally, don't give up. When data mining works, it is well worth the effort.See what Rob Meyer has to say about ‘A Better Way to Mine Data’. http://www.nag.co.uk/IndustryArticles/ABetterWaytoMineData.pdf