HDFS is a core and fundamental component of Hadoop. This file system is oriented on handling huge amounts of data. From first glance you may not notice much differences from usual Linux file system as it follows lots of POSIX specifications. But behind the scene HDFS does a lot of extra work to provide stable and quick access to the data which is stored across different machines of distributed cluster. It is indeed a great data management mechanism which takes all responsibility for building most optimal data-flows according to the network topology of the cluster and which performs automatic handling of critical situations related to the breaks in hardware. In this article I’ll try to give a general overview of Hadoop file system and show some common techniques of working with the data inside it.
At this stage you probably have a general idea of what Hadoop is in technical scene. But why do we really need such a huge and complicated platform for doing such simple things like searching, counting or sorting our data. According to the research provided by Cisco last year annual global IP traffic will reach 2.3 zettabytes per year by 2020. Another research forecast performed by International Data Corporation few years ago stated that up to 2020 people will have to operate with 44 zettabytes of data. Can we really handle such capacities with our current hardware and algorithms? Hadoop is probably the best attempt to handle that problem at this time.
There is quite an interesting competition which exists in the world of Big Data called Terasort. It appeared in 2008 with the general idea to generate, sort and validate 1TB of data. At that period the result was 3 minute 48 seconds on Hadoop cluster of 910 nodes. By the time the amount of data increased to 100TB and just few month ago we got a new record of sorting 100TB of data for 98.8 seconds in the cluster of 512 nodes. The actual results are available Sort Benchmark Home page.
If you ask me what is the most complicated part of Hadoop, I will tell you that it is configuration. It’s really a nightmare to keep in sync all these parts and their dependencies. You have to know and properly configure hundreds of different properties per each Hadoop daemon. At some stage you start to update of one part of your cluster and it breaks another. You fix it and this fix breaks something else. As a result instead of working with your data and writing your code you spend days in searching correct patches and configurations for your daemons.
In my first article I want to share my experience with the steps I did to start working the world of Big Data. As a guy from .NET stack it was really quite challenging for me to understand what technologies I should start studying for getting myself into the world of Hadoop and to identify these first practical steps which need to be done to start working in this realm.