Hadoop
Distributed File System
A file system is an operating system organizes and
manages files on disk storage. It helps users store, maintain, and retrieve
data from the disk.
Distributed File System
DFS stands for distributed file system, it is a
concept of storing file in multiple nodes in a distributed manner. DFS actually
provides Abstraction for a single large system whose storage is equal to the
sum of storage of other nodes in a cluster.
Storing very large files (e.g., 30TB) on a single
system is impractical because:
· Disk
capacity of one machine is limited and can only grow so much.
· Processing
huge datasets on a single machine is inefficient and slow.
Distributed File Systems (DFS) overcome these issues
by storing data across multiple machines, enabling faster and scalable
processing.
Example:
A 40TB file takes 4 hours to process on a single
machine.
In a DFS, the file is divided across 4 nodes, each
storing 10TB.
All 4 nodes process data simultaneously.
Processing time reduces from 4 hours to 1 hour.
Conclusion:
DFS enables faster, more efficient big data processing through parallel
processing.
The Design of HDFS
HDFS (Hadoop Distributed File System)
is the main storage system in Hadoop. It stores large files by breaking them
into blocks (default 128 MB) and distributing them across multiple low-cost
machines.
HDFS Stores Data
HDFS divides a large file into fixed-size blocks,
commonly 128 MB by default, although this is configurable.
For example, a 300 MB file with a 128 MB block size
is divided into:
Block 1: 128 MB
Block 2: 128 MB
Block 3: 44 MB
These blocks are distributed across different Data
Nodes and replicated for fault tolerance. A common replication factor is 3.
HDFS ensures fault-tolerance by keeping copies of
data blocks on different machines. This makes it reliable, scalable and ideal
for handling big data efficiently.
Features of HDFS
· HDFS
also provides high availability and fault tolerance.
· Provides
scalability to scale up or scale down nodes as per our requirement.
· Data
is stored in distributed manner i.e. various Data nodes are responsible for
storing the data.
· HDFS
provides Replication because of which no fear of Data Loss.
· HDFS
Provides High Reliability as it can store data in a large range of Petabytes.
· HDFS
has in-built servers in Name node and Data Node that helps them to easily
retrieve the cluster information.
· Provides
high throughput.
HDFS (Hadoop Distributed File System)
HDFS is a distributed file system in Hadoop designed
to store very large files across multiple machines. It provides fault
tolerance, high-throughput data access, and scalability. HDFS has two main
components Name Node (Master) & Data Node (Slave).
1. Name Node (Master):
The Name Node acts as the master of the Hadoop
cluster. It is responsible for storing metadata data about the actual data.
This includes information like File names, File sizes, Block numbers and IDs,
Locations of blocks stored in Data Nodes, Monitors Data Nodes through
heartbeats and block reports. Does not normally store the actual file contents.
Name Node tasks
·
The Name Node manages the metadata of
the HDFS file system. It generally does not store the actual file data.
·
Metadata management: Maintains file
names, directories, permissions, and block locations.
·
Block management: Tracks which Data
Nodes contain each block.
·
File operations: Manages file creation,
deletion, renaming, and directory operations.
·
Replication management: Monitors
replication and arranges additional copies when blocks are under-replicated.
·
Heartbeat monitoring: Checks Data Node
health using heartbeats.
·
Data Node failure detection: Marks Data
Nodes as dead when heartbeats are missing beyond the configured timeout.
·
Block placement: Selects suitable Data
Nodes for storing new replicas.
·
Namespace management: Maintains the HDFS
directory and file structure.
2. Data Node (Slave):
The Data Nodes act as slaves and are responsible for
storing actual data blocks in Hadoop cluster. You can have dozens or even
hundreds of Data Nodes and more Data Nodes mean more storage capacity.
Each Data Node performs tasks like creating,
deleting or replicating data blocks all based on the instructions received from
the Name Node. To store large files efficiently, Data Nodes should have a high
storage capacity. The worker node that stores actual data blocks. Performs read
and write operations as instructed by the client. Sends regular heartbeats and
block reports to the Name Node. Replicates blocks when required.
·
Data Nodes store the actual data blocks
and perform data read/write operations.
·
Block storage: Stores actual file data
in blocks on local disks.
·
Read operations: Sends requested data
blocks to clients.
·
Write operations: Stores blocks received
from clients or other Data Nodes.
·
Replication: Creates or transfers block
replicas as instructed by the Name Node.
·
Heartbeat sending: Periodically informs
the Name Node that it is alive.
·
Block reports: Reports the blocks stored
on its disks to the Name Node.
·
Block deletion: Deletes replicas when
instructed by the Name Node.
·
Checksum verification: Helps detect data
corruption during data access and transfer.
Allows users and applications to access files.
Contacts the Name Node to obtain metadata and block
locations.
Transfers actual file data directly between the
client and Data Nodes.
4. Secondary Name Node / Checkpoint Node
Periodically merges the namespace image (FsImage)
with the edit log to create checkpoints. Helps manage metadata logs.
It is not automatically a backup or standby Name
Node. High availability uses Active and Standby Name Nodes.
HDFS Concepts
HDFS (Hadoop Distributed File System) is a
distributed file system used to store very large files across multiple
computers. It provides fault tolerance, scalability, and reliable data storage.
HDFS has two main components: Name Node (Master) and
Data Node (Worker).
1. Name Node (Master)
The Name Node manages the file system metadata. It
knows where data is stored but normally does not store the actual file
contents.
· Secondary
Name Node tasks
- The Secondary Name Node helps the Name
Node manage file system metadata by periodically creating checkpoints. It is
not a backup or standby Name Node.
- Check pointing: Creates periodic
checkpoints of HDFS metadata.
- Merges FSImage and Edit Log: Downloads
the current FSImage and Edit Log from the Name Node and merges them.
- Reduces Edit Log size: Helps prevent the
Edit Log from growing indefinitely.
- Returns checkpoint: Sends the newly
created checkpoint image back to the Name Node.
- Improves recovery: A recent checkpoint
can reduce the time needed to load metadata after a Name Node restart.
· FSImage
FSImage (File system Image) is a file
that stores a snapshot of the HDFS file system namespace and its metadata.
It contains information such as:
File and directory names.
Permissions and ownershipFile-to-block mappings and other
namespace metadata.
2. Data Node (Worker)
The Data Node stores the actual data blocks on its
local disks.
1. Replication in HDFS
Replication means storing multiple copies of a data
block on different Data Nodes to provide fault tolerance and data availability.
Example: If the replication factor is 3, HDFS stores
three copies of each block.
Ø The
Name Node manages file metadata and block locations.
Ø Data
Nodes store the actual data blocks.
Ø If
a Data Node fails, the Name Node arranges another copy when necessary.
Ø The
default replication factor is commonly 3, but it can be configured.
2. Balancing in HDFS
Balancing means distributing data blocks across Data
Nodes so that storage utilization is reasonably uniform.
3. Heartbeats in HDFS
A heartbeat is a periodic message sent by each Data
Node to the Name Node to indicate that the Data Node is alive and functioning.
HDFS stores large files as blocks across multiple Data
Nodes, while the Name Node manages metadata and block locations. Replication
provides fault tolerance, and distributed storage enables scalability.
Hi. I’m Designer of Blog Magic. I’m CEO/Founder of ThemeXpose. I’m Creative Art Director, Web Designer, UI/UX Designer, Interaction Designer, Industrial Designer, Web Developer, Business Enthusiast, StartUp Enthusiast, Speaker, Writer and Photographer. Inspired to make things looks better.
0 comments:
Post a Comment