What Is Data Mining?


In the 19th Century, Britain’s industrial revolution led the world into a new age. A central element in its success was a mile-deep layer of coal. The 21st Century’s revolution will be led by those who mine a new and equally immense resource - data. Almost every computer science innovation has increased those data deposits. The quantity now being collected by web trackers alone is staggering. Every company is obliged to collect and store customer and market data, so it constantly piles up, and up. If you don’t know what to do with it, it is an overhead - but if you do, it is gold dust.


Quantity changes what you can do with data. A single piece of information such as “John Smith bought a spade” is limited to what it says, but when combined with multiple other pieces of data such as the date, place, price, brand, his age, how he found you, how he navigated your website and so on - valuable correlations can be discovered. Those discoveries may enable you to radically improve your marketing pitch, product features, pricing, website design and so forth. For example, Walmart discovered that male customers who buy nappies also buy beer. So, they relocated the items next to each other and drove up sales. Looking for the things you do not know, but can exploit, is the essence of data mining.

That still doesn’t do data mining justice The applications for data mining techniques are broad and arguably inexhaustible.?

Data mining backed by AI and machine learning is already in use detecting network intrusions, malware activity and financial fraud - simply by detecting anomalies in broad patterns of behaviour. It has been applied by police departments in the US to predict where street crimes are most likely to occur.?It helps manufacturers optimise supply chains and improve quality assurance, and game developers to understand player behaviour and test their games. The sciences rely on it to identify orbital anomalies, predict earthquakes, reconstruct climate patterns and detect new particles?at the Hadron Collider, and?it has helped track and control the coronavirus epidemic.


The original technical name for data mining was KDD (knowledge discovery in databases), however, to succeed in this field commercially, it helps to have aptitudes far beyond database querying. The data science skills involved include neural networks, decision trees, descriptive modelling, predictive modelling and learning algorithms.?Presenting that data so that it can be understood is a field in itself – data visualisation.

Building a data science career in data mining

Data mining professionals are needed wherever there is data – and data is everywhere. To specialise in any given area, it helps to have some broad experience or understanding of that sector, but most data mining skills themselves are readily transferable. As such, there are many opportunities for data engineers to switch industries and build a diverse and interesting career.

There are often correlations in big pools of data that answer questions no one thought to ask. To rapidly retrieve and apply salient information from large data resources frequently requires the assistance of AI - so this is another talent looked for by recruitment agencies in this field, such as i-Gem.

i-Gem specialise in helping data professionals build their careers and organisations source the talent they need. Call i-Gem on 020 3909 3990, or email?info@i-gem.co.uk.

9 views0 comments