Data Flow
HDFS (Hadoop Distributed File System) stores large
files by splitting them into blocks and distributing those blocks across
multiple machines called Data Nodes.
Name Node: The
primary purpose of Name node is to manage all the Metadata. As we know the data
is stored in the form of blocks in a Hadoop cluster. So on which Data Node or
on which location that block of the file is stored is mentioned in Metadata.
Log of the Transaction happening in a Hadoop cluster, when or who read or write
the data, all this information will be stored in Metadata.
Data Node: Data
Node is a program run on the slave system that serves the read/write request
from the client and used to store data in form of blocks.
HDFS Client: HDFS
Client is an intermediate component between HDFS and the user. It communicates
with the Data node or Name node and fetches the essential output that the user
requests.
Apache Flume is an agent-based, event-driven service optimized for harvesting and aggregating unstructured streaming data, such as server application logs, IoT streams, or social media feeds.
1. Source: Consumes
events from the data generator (e.g., tailing a log file, netcat).
2. Channel: Acts as
a buffer (Memory or File-based) that safely stores events until they are fully
consumed.
3. Sink: Delivers
the data payloads continuously to a destination like HDFS.
Apache Flume is a
data ingestion tool.
It used to collect,
aggregate, and transfer large amounts of log or streaming data from different
sources to a destination such as HDFS.
Flume mainly consists of three components:
Source, Channel, and Sink.
The Source receives the data, the Channel
temporarily stores the data, and the Sink sends the data to the final
destination, such as HDFS.
When you ingest
data into Hadoop using Apache Scoop, Scoop divides the workload among multiple Map
Reduce mappers. By default, this generates multiple small files in HDFS.
Apache Scoop is a
command-line tool designed to transfer bulk data efficiently between Apache
Hadoop and structured data stores, such as relational databases (RDBMS).
The name "Scoop"
is a combination of SQL and Hadoop.
Import: Reads rows
from databases like Oracle, MySQL, or PostgreSQL and saves them into Hadoop
file stores like Hadoop HDFS, Apache Hive, or HBase.
Export: Reads data
back out of Hadoop environments and writes it directly to external relational
database tables.
Map Reduce enables
high-speed data transfer by dividing a large database table into smaller chunks
and processing them at the same time (parallelism).
Apache Scoop is a
tool used to transfer bulk data between relational databases (such as MySQL or
Oracle) and Hadoop systems like HDFS, Hive, and HBase.
For example:
Import: MySQL →
HDFS
Export: HDFS →
MySQL
Features
of Scoop
- Supports bulk data transfer.
- Can use parallel processing to
speed up imports and exports.
- Supports incremental imports for
retrieving new or updated records under supported conditions.
- Can integrate with Hive and HBase.
- Uses database connectors and JDBC
drivers to communicate with databases.
Hadoop Archives
(HAR)
HDFS is designed to
handle large files. When there are millions of small files, the Name Node must
store metadata for every file and its blocks, consuming memory.
Hadoop Archives
(HAR) is a feature in Hadoop that combines multiple files in HDFS into a single
logical archive to reduce the pressure on the HDFS Name Node caused by too many
small files.
HAR files help
organize small files, but they do not significantly compress the data and do
not reduce the actual number of underlying HDFS blocks.
- HAR stands for Hadoop Archives.
- It combines multiple HDFS files
into a logical archive.
- It is useful for managing large
collections of small files.
- Files inside a HAR archive are
read-only through the archive interface.
- Creating or deleting individual
files inside an archive is not supported; you generally need to recreate
the archive to change its contents.
- HAR does not inherently compress
files or reduce the underlying data size.
Hadoop
Input and Output
- Input:
Data that Hadoop reads from a source such as HDFS.
- Processing:
The Mapper and Reducer process the data.
- Output:
The final result written back to HDFS.
Input Format
Input Format determines how input
files are split and which Record Reader should read them.
Common types:
·
TextInputFormat — the default; reads
text files line by line.
·
KeyValue TextInput Format — interprets each line
as a key-value pair separated by a delimiter.
·
Sequence File Input Format — reads Hadoop Sequence
Files containing serialized key-value pairs.
Input Split
An Input Split represents a logical
portion of the input data to be processed by a Mapper.
For example, a large file may be divided into several
splits, allowing multiple Mappers to process it in parallel.
Record Reader
Record
Reader converts the
data in each split into key-value pairs for the Mapper.
Compression in Hadoop is
the process of reducing the size of data so that it requires less storage space
and can often be transferred and processed more efficiently. Hadoop supports
compression for input files, intermediate Map Reduce data, and final output
files.
Gzip → good compression,
not split able
Bzip2 → split able
Snappy → very fast compression/decompression
Serialization in Hadoop is the process of converting an object into a byte stream so that it can be stored on disk or transmitted across a network. Deserialization is the reverse process: converting the byte stream back into an object.
Hadoop uses serialization extensively when transferring data between Map Reduce tasks and storing data in binary formats.
Serialization is
required to convert data into bytes so that it can be efficiently stored or transmitted
across the network.
Apache Avro in Hadoop
Apache Avro is a data serialization
framework used in Hadoop and big-data systems. It converts structured data into
a compact binary format and uses a JSON-based schema to describe the
data.
Avro is useful for storing, transferring, and sharing
data between applications written in different programming languages.
Serialization systems, Avro embeds the JSON schema
directly inside the container file (an .avro file) or transmits it at the
beginning of a network connection. This makes Avro files entirely
self-describing.
Avro is a system used to convert data into a compact
format so that it can be stored or transferred efficiently.
Features of Avro
- Schema-based:
Data is described by a schema.
- Compact binary format:
Efficient for storage and network transfer.
- Language-independent:
Supports Java, Python, and other languages.
- Schema evolution:
Supports compatible schema changes, such as adding a field with an
appropriate default value.
- Compression support:
Avro object container files can use supported compression codecs.
- Hadoop integration:
Can be used with Hadoop Map Reduce and other big-data tools.
File-based data
structures are specialized file formats designed to store large volumes of
structured, semi-structured, or unstructured data across the Hadoop Distributed
File System (HDFS).
Big Data, Hadoop and Map Reduce – Justify Questions
1) Justify why
traditional databases are insufficient for Big Data.
Answer:
Traditional databases may struggle to handle huge volumes of diverse and
rapidly generated data.
2) Justify why
Big Data is called complex data.
Answer: Big
Data is complex because it contains large amounts of data in different formats
generated at high speed.
3) Justify why
Hadoop is used for Big Data processing.
Answer: Hadoop
supports distributed storage and parallel processing of large datasets.
4) Justify the
role of the Name Node in Hadoop.
Answer: The Name
Node manages file system metadata and tracks the locations of data blocks in
HDFS.
5) Justify the
need for the Data Node in HDFS.
Answer: The Data
Node stores and retrieves actual data blocks in HDFS.
6) Justify why
Hadoop is suitable for processing large datasets.
Answer: Hadoop
distributes data and processing tasks across multiple computers.
7) Justify the
importance of the Map phase in Map Reduce.
Answer: The Map
phase processes input data and generates intermediate results.
8) Justify why
the Map phase generates key-value pairs.
Answer:
Key-value pairs help organize intermediate data for grouping and further
processing.
9) Justify why
the Map phase is performed before the Reduce phase.
Answer: The Map
phase generates intermediate data required by the Reduce phase.
10) Justify why
the Map phase divides data into smaller tasks.
Answer: Smaller
tasks enable parallel processing and improve efficiency.
11) Justify why
Big Data needs scalable storage.
Answer:
Scalable storage allows storage capacity to increase as data grows.
12) Justify why
real-time data processing is important.
Answer: It
provides timely information for quick decision-making.
13) Justify why
data quality is important in Big Data analytics.
Answer:
High-quality data helps produce accurate and reliable results.
14) Justify why
Big Data is useful for decision-making.
Answer: Big
Data identifies useful patterns and trends that support better decisions.
15) Justify why
Big Data needs scalable storage.
Answer:
Scalable storage accommodates increasing amounts of data efficiently.
16) Justify why
real-time data processing is important.
Answer: It
enables immediate analysis of incoming data for timely responses.
17) Justify why
Hadoop is an open-source framework.
Answer:
Hadoop's source code is publicly available for use and modification under its
open-source license.
18) Justify why
HDFS divides files into blocks.
Answer:
Dividing files into blocks enables distributed storage and parallel processing.
19) Justify why
Hadoop stores multiple copies of data blocks.
Answer: Data
replication improves data availability and protects against data loss due to
machine failures.
20) Justify why
the Name Node maintains metadata.
Answer:
Metadata helps the Name Node track files and locate their data blocks.
21) Justify why
YARN is required in Hadoop.
Answer: YARN
manages cluster resources and schedules applications for execution.
22) Justify why
Hadoop is suitable for distributed computing.
Answer: Hadoop
distributes storage and processing tasks across multiple computers.
23) Justify why
the Map phase processes input records.
Answer: It
transforms input records into intermediate data for further computation.
24) Justify why
the Map phase produces intermediate key-value pairs.
Answer:
Intermediate key-value pairs help group related data and pass it to the Reduce
phase.
Hi. I’m Designer of Blog Magic. I’m CEO/Founder of ThemeXpose. I’m Creative Art Director, Web Designer, UI/UX Designer, Interaction Designer, Industrial Designer, Web Developer, Business Enthusiast, StartUp Enthusiast, Speaker, Writer and Photographer. Inspired to make things looks better.
0 comments:
Post a Comment