HDFS : Hadoop Distributed File System

-
The Hadoop Distributed File System (HDFS) is the primary data storage system used by Hadoop applications. It employs a NameNode and DataNode architecture to implement a distributed file system that provides high-performance access to data across highly scalable Hadoop clusters.

HDFS is a key part of the many Hadoop ecosystem technologies, as it provides a reliable means for managing pools of big data and supporting related big data analytics applications.

How HDFS works
HDFS supports the rapid transfer of data between compute nodes. At its outset, it was closely coupled with MapReduce, a programmatic framework for data processing.

When HDFS takes in data, it breaks the information down into separate blocks and distributes them to different nodes in a cluster, thus enabling highly efficient parallel processing.

Moreover, the Hadoop Distributed File System is specially designed to be highly fault-tolerant. The file system replicates, or copies, each piece of data multiple times and distributes the copies to individual nodes, placing at least one copy on a different server rack than the others. As a result, the data on nodes that crash can be found elsewhere within a cluster. This ensures that processing can continue while data is recovered.

HDFS uses master/slave architecture. In its initial incarnation, each Hadoop cluster consisted of a single NameNode that managed file system operations and supporting DataNodes that managed data storage on individual compute nodes. The HDFS elements combine to support applications with large data sets.

This master node "data chunking" architecture takes as its design guides elements from Google File System (GFS), a proprietary file system outlined in in Google technical papers, as well as IBM's General Parallel File System (GPFS), a format that boosts I/O by striping blocks of data over multiple disks, writing blocks in parallel. While HDFS is not Portable Operating System Interface model-compliant, it echoes POSIX design style in some aspects.

Continue Reading... this at techtarget

Comments

Post a Comment

Popular posts from this blog

Understanding Testing in Apex

-
Testing is the key to successful long term development and is a critical component of the development process. And testing is key to the success of application particularly if application to be deployed to customers. If we validate that application works as expected, that there are no unexpected behaviors, then application will run for longtime. There are two ways of testing an application. One is through the  salesforce  user interfaces (testing with a single record) The other way is to test for build functionality i.e testing with bulk records by using apex data loader up to 200 records can be passed through code. Before we can deploy code or package into production, the following must be true. Check out the top Salesforce Interview Questions now! → 75 % of apex code must be covered by unit tests. All of those tests must complete successfully. → When deploying to a production organization we need to have 75% of apex code cove...
Read more

5 Ways Blockchain Technology Will Change the Way We Do Business

-
Most of the media headlines on blockchain technology revolve around the outsized investment performance and extreme volatility of cryptocurrency. This trajectory closely resembled a roller coaster as Bitcoin gained 1,390 percent in 2017 (up as much as 1,935 percent when it hit a record high of $19,666 in mid-December) -- and then plummeted to a low of $5,950 in January 2018. By mid-February, it had settled around $8,400. The growing consensus among business leaders and entrepreneurs: The future of blockchain technology will be about a lot more than Bitcoin. Blockchain tech will impact every major area of business from accounting to operations, and there's evidence the revolution has begun. What is it, and why is it so crucial to the future of business? A blockchain is an electronically distributed ledger or list of entries -- much like a stock ledger -- that various participants maintain via a network of computers. Blockchains use cryptography t...
Read more

Microsoft Azure Migration Guide - Key Planning and Decision Points Before Migrating to Azure

-
Image
With such an increased interest in Cloud computing, and with companies like Microsoft expanding their datacenters to Canada, now’s a great time for organizations who are looking at a datacenter refresh to consider migrating to Azure. Migrating a datacenter to Azure is a cost-effective alternative for many organizations. By migrating to Azure , you can reduce your capital and operational expenditures and improve your organization’s elasticity, scalability, and overall security. But where do you begin on your Azure migration journey? And what do you need to do before you can migrate your datacentre to Azure? Here are some key planning and decision points. Azure Migration Guide 1 : Understand Your Business Case When determining if a move to Azure is the right one for your organization, it is important to start off by examining your business case for doing so. What are the drivers for this move? For example, are you looking to decrease operational expenditures? Reduce your c...
Read more