Lack of tools preventing organisations from exploiting dark data
News / Lack of tools preventing organisations from exploiting vast amounts of dark data
3 May 2019
A large number of organisations store a lot of information about consumers and markets these days that they are simply unable to analyse or process either because they do not have the right tools or because they lack the expertise or staff necessary to extract the value from dark data.
Dark data is the information that almost every organisation processes for its daily operations but does not store such data for analysis or research. According to Gartner, dark data includes all information assets that organisations collect, process and store during regular business activities, but generally fail to use for other purposes such as data analytics, business relationships and direct monetising.
In order to gauge how much of dark data organisations store and how much of it they are able to use for data analytics and market research, Splunk carried out a survey of 1,300 IT and business leaders in the UK, the US, France, Germany, China, Japan, and Australia and found that the use of dark data for analytics and monetisation is quite insignificant because of various factors.
Too much dark data, too little resources to exploit it
The survey revealed that even though 76% of IT leaders believe that the secret of an organisation’s success is in how much data it stores and processes, a vast majority of organisations lack the right tools such as AI to exploit such data or do not have the expertise or staff necessary to extract the value from dark data.
While senior IT and business leaders agree dark data has tremendous value, many of them do not know how to use such data for analytics, a majority of them don’t have the technology, expertise or staff necessary to use AI, and 69 percent of IT leaders are loathe to learning new data skills to make themselves capable of using dark data for various purposes.
Splunk noted that it is only in China that business leaders and IT decision makers are working hard to meet the challenges of both dark data and AI, with a majority of them in other countries shirking from the challenge.
“Dark data is valuable, in part, because there’s so much of it, and it could be used to unlock business value. The more data you have, the more you can learn about every aspect of your operations. The larger the dataset, the more accurate the analysis. If you ignore the challenge of dark data you’re leaving money on the table,” says Tim Tully, chief technology officer at Splunk.
He added that organisations that possess vast amounts of dark data but are unable to process such data are incurring real monetary cost for storing such data in addition to the opportunity cost.”Companies that refuse to move into the future with new IT solutions will never be able to take advantage of dark data. They will lose the data race before they get to the starting line,” he added.
Benefits of dark data outweigh the costs of maintaining it
Stephen Mackey, an expert in Information Management, says that the company that is able to identify useful and relevant data, and subsequently adopt or make use of it to the fullest, is the organisation most likely to succeed, outperform and deliver value outwardly and inwardly. Find where your dark data is, access it, organise it, analyse it and then take action. It could prove to be extremely profitable.
“It’s important to act when the benefits of use outweigh the costs of accessing and analysing dark data. Locate, organise and understand data in order to unlock its relevance and usefulness. There may be information you can monetise, there might not, but you won’t know until you assess it.
“For the inputs and outputs of an Accounts Payable process, having all the content able to see each other is an asset that needs to be recognised and applied to the best of its ability. It will help you realise inefficiencies, even if this just means avoiding duplicate invoices or identifying previous purchasing trends,” he adds.
Storing too much dark data and not being able to maintain visibility over such data or failing to process it could also result in major data breaches as a result of hackers gaining access to unsecured cloud databases that are frequently used by businesses to store vast volumes of customer and enterprise data.
“Researchers and hackers alike are continually canvassing systems like Amazon S3, probing for wide-open buckets with misconfigured access controls. The odds are high that they will find a cache of sensitive information because organisations have a staggering amount of dark data, meaning they don’t know who has access, who should have access, and when malware or human users are behaving dangerously. The result of all of this is the recent spate of high profile data breaches that have stemmed from S3 buckets,” said Matt Lock, Director of Sale Engineers at Varonis.