Unit-4 Hadoop Ecosystem
Hadoop ecosystem architecture
HDFS (Hadoop Distributed File System). Serves as the primary storage foundation by splitting massive files into blocks and distributing them across multiple nodes for fault tolerance.Bottom section of above digram shows how external data
moves into and out of Hadoop.
• Sqoop. Transfers structured data (such
as tables from relational SQL databases) back and forth between external
databases and HDFS.
• Flume. Collects, aggregates, and streams
unstructured and semi-structured data (such as server logs and event feeds)
directly into HDFS.
Processing & NoSQL Database -
- MapReduce Framework. Processes large datasets in parallel by dividing tasks into mapping (filtering and sorting) and reducing (summarizing) phases.
- HBase.
Operates as a distributed, column-oriented NoSQL database on top of HDFS
to enable fast, real-time read and write access to specific records.
·
Hadoop relies on MapReduce to process
data. However, native MapReduce requires writing long, complex Java programs
even for very basic tasks.
·
Hive and Pig act as translators. You
write a few lines of simple commands, and the tool automatically converts them
into MapReduce jobs behind the scenes.
Each tool in this layer serves a specific
purpose for handling big data easily in the Analysis & ML Layer
- Apache
Hive (SQL Querying & Reporting). Used by data analysts to summarize,
query, and analyse large datasets stored in HDFS. It uses HiveQL, which
looks just like standard SQL, making it ideal for business intelligence
and generating reports.
- Apache
Pig / Pig Latin (Data Cleaning & ETL). Used by data engineers to
clean, filter, join, and transform messy or unstructured data into a clean
format before analysis. It uses a simple step-by-step scripting language
called Pig Latin.
- Apache Mahout (Machine Learning). Used by data scientists to run pre-built machine learning algorithms on massive datasets. It helps build recommendation systems (like suggesting movies or products), group similar data (clustering), and classify information.
The Hadoop Ecosystem is a collection of
open-source tools and frameworks used to store, process, manage, and analyze
very large datasets (Big Data) across distributed clusters of computers.
It enables organizations to process huge amounts of
structured, semi-structured, and unstructured data efficiently. ecosystem adds
specialized capabilities for SQL querying, real-time data streaming, NoSQL
storage, machine learning, and cluster management.
Why
Hadoop Ecosystem required?
Instead of relying on one giant, expensive
supercomputer, the Hadoop ecosystem allows organizations to connect many
standard, low-cost servers together. If one server fails, the ecosystem
automatically redirects tasks and recovers data from backups without
interrupting work.
Key Components by Layer
- Ingestion:
Sqoop imports structured SQL data, while Flume and Kafka
stream unstructured log and event data into Hadoop.
- Storage:
HDFS stores huge files safely across multiple nodes, and HBase
acts as a fast, real-time NoSQL database.
- Processing:
YARN manages cluster resources, while MapReduce and Spark
process data in parallel.
- Analysis
& ML: Hive runs SQL-like queries, Pig
cleans data via scripts, and Mahout handles machine learning.
- Management:
Oozie schedules automated workflows, ZooKeeper coordinates
cluster nodes, and Ambari monitors system health.
Apache Pig
is an open-source data flow platform in Hadoop used to clean, transform, and
process massive datasets without writing complex Java MapReduce code.
Key Points
- Language.
Uses Pig Latin, a simple, step-by-step scripting language.
- How
It Works. Automatically translates short Pig
Latin scripts into MapReduce jobs behind the scenes.
- Best
For. ETL pipelines (Extract, Transform, Load)
and processing messy, semi-structured, or unstructured data (like logs and
JSON).
Data Processing Operators
- Input/Output.
LOAD (reads data), STORE (saves to HDFS), and DUMP (displays on screen).
- Filtering/Projection.
FILTER (selects rows by condition) and FOREACH ... GENERATE (transforms
columns).
- Combining/Sorting.
GROUP (groups by key), JOIN (merges datasets), ORDER BY (sorts data), and
DISTINCT (removes duplicates).
Hive Shell is a command-line interface (CLI) used to interact with Apache Hive in the Hadoop ecosystem. It allows users to
query and manage large datasets stored in HDFS using SQL-like commands called HiveQL (HQL).
Hive Architecture
1.
Hive Client:
Connects users through JDBC, ODBC and Thrift.
2.
Hive Services:
Includes Driver, Compiler, Optimizer, Metastore and HiveServer2.
3.
Processing:
MapReduce processes data and YARN manages resources.
4. Storage: HDFS stores data.
- Apache
Hive is a data warehouse tool built on top of Hadoop.
- It
is used to analyze large datasets stored in HDFS.
- It
converts HiveQL queries into execution jobs using engines such as
MapReduce, Tez, or Spark.
Hive Metastore
is a central repository that stores metadata (data about data) of Hive tables. Hive
Services
Hive provides different services to execute queries
and manage data.
Hive Services
- Driver:
Manages query execution.
- Compiler:
Converts HiveQL into an execution plan.
- Metastore:
Stores metadata.
- Execution
Engine: Executes queries.
- HiveServer2:
Connects external applications to Hive.
HBase in Hadoop
HBase is a distributed
NoSQL database used to store and access large amounts of data in Hadoop.
Components
1.
HMaster:
Manages the cluster.
2.
RegionServer:
Handles data operations.
3.
ZooKeeper:
Coordinates services.
4.
HDFS:
Stores data.
HBase Architecture:
HBase is a distributed NoSQL database that stores large amounts of data in
Hadoop. The Client sends read and write requests, HMaster manages
the cluster and assigns regions, RegionServer handles data operations, Regions
divide tables into smaller parts, ZooKeeper coordinates cluster
services, and HDFS stores the actual data. Together, these components
provide fast, reliable and scalable data access.
Big SQL (IBM Db2 Big SQL)
is an enterprise SQL-on-Hadoop engine that lets you query massive Hadoop
datasets using standard ANSI SQL.
Big SQL is an SQL-based
tool used to access and analyze large amounts of data stored in Hadoop.
Features:
1.
Supports SQL queries.
2.
Fast data processing.
3.
Handles big data.
4.
Easy to use.
5.
Supports HDFS.
Assignment Question
1. What is Apache Hive? Explain its architecture with a neat diagram.
2. Explain the working of Hive with its components.
3. What is HBase? Explain its architecture with a neat diagram.
4. Explain the features and advantages of HBase.
5. What is Big SQL? Explain its architecture with a neat diagram.
6. What is Apache Pig? Explain its architecture with a neat diagram.
Hi. I’m Designer of Blog Magic. I’m CEO/Founder of ThemeXpose. I’m Creative Art Director, Web Designer, UI/UX Designer, Interaction Designer, Industrial Designer, Web Developer, Business Enthusiast, StartUp Enthusiast, Speaker, Writer and Photographer. Inspired to make things looks better.
0 comments:
Post a Comment