From the most primitive writing forms to the latest data centers, humans have loved gathering information. As technology progressed through the years, data overflow has become the new normal. This massive amount of data requires very sophisticated data storage systems.
As to why Big Data has had what we can call a comedy of sorts, we have taken a different approach and divided the history of Big Data into acts. We have done this in the form of a story, to help you connect and remember what you read for a long time.
Despite its existence for a long time, Big Data has been a topic of puzzlement for a lot of people. The biggest issue is the lack of people qualified enough to handle Big Data and utilize their practices.
Stay assured that this article will help you get into the deeper realms of Big Data and its world.
All kinds of analytics got done with the help of databases. Every imaginable type of data was present in databases. As a pattern, any of the data good enough for a database used to find itself in a flat file.
Google used to have a dislike for databases. They would have tried storing their data through databases, but it did not work. It would have been successful, but the company to make that happen definitely will have asked for too high a price. As a result, Google used files.
Google is a huge company, so they built an extensive, distributed file system. It was never surprising as it was Google, after all.
The company faced an issue, as there was a need for a query about the large files. The engineers and developers came together and devised a simple solution for a complex problem. This solution made the developers at Google geniuses in their own right. What is surprising is that they spread the news and told about it to everyone, while they made huge money with the devised solution.
The reason why they told everybody is still unclear. To increase the knowledge of people, or let others think as they did so that more people got good jobs. These are a couple of plausible reasons.
Or they knew that MapReduce was not supposed to work in the long run of time and wanted to send their competition in the wrong direction. A good enough third reason?
Whatever may have been the reason, the world got a shock, and when the world faces something entirely new, the level of interest breaks all known barriers to knowing about it a bit more.
This is when some people huddled together and implemented Google’s strategy in open source. Yahoo, the then competitor to Google, helped these people out by funding them to a considerable extent. This was the birth of Hadoop.
Now, the issue was that every company was not Google, and so, the presence of such a massive amount of data was not there. Many companies did not even have sufficient data to fill a MySQL database. Still, everybody liked MapReduce.
People thought about a lot of data and created Big Data. They thought about how a lot of data from the world can be collected, store it in an understandable format and bring about a change in the world.
This was when people also understood that scientists were handling Big Data for a long time, but referring to it as scientific computing. The world now had Big Data in literal terms.
It was apparent that Google was not very fond of databases, and some pointed out the reason to be SQL. But of course, making SQL work with big files was a little tricky. Now, this was what people love to call ‘a window of opportunity.’
Predictably, investors went into a frenzy and poured their money into Hadoop.
Soon after money got invested into Hadoop, Spark came to the fore, beating Hadoop by a large margin on performance. This was the time where Hadoop lost the race finally. The investor money was all put into Spark itself.
It is tough to understand where Hadoop ends and Spark begins. While both are MapReduce technologies, Spark came up to be the future. Because of Spark, the Machine Learning technology got a huge boost, and soon, the advent of data science takes shape.
Now Spark displayed inefficiencies while working with deep learning. Google and other companies did create new techniques, but it did not matter too much as most of the data was small.
This is the time when Big Data/AI/analytics got it all. A huge market, customers, use cases, and even investors. But there still is a difficulty. The difficulty to find out people for building such systems.
Moreover, the difficulty increases more when the architects to think it out and make systems workable are needed. Nobody is thinking about the usability yet, and problems like bias have started to creep up in operations.
Big companies with their rebranding as Big Data companies are not showing the growth expected. The chunk looks to be going in favor of the Blockchain.
There is AI, singularity, and even the robots are taking a dominant stand in our lives. Machines do threaten our presence at doing our jobs. Bots too are making their presence felt, but most of them break down pretty quickly.
However, there is still hope for Big Data. The work is being done for it already. Programmers are getting trained.
The number of people asking for analytics is increasing with each passing day. With more end to end stories and getting the usability right, Big Data can still work very successfully.
Like our content? Share the joy with your friends!
Have a project in mind? Don’t hesitate to reach us at contact@goodworklabs.com