Introduction to Big Data

Big-Data Analytics

•       Big Data Analytics uses advanced analytical methods that can extract important business insights from bulk datasets.
•       Within these datasets lies both structured (organized) and unstructured (unorganized) data.
•       Big Data Analytics refers to the complex process of examining large and varied data sets—also known as big data—to uncover hidden patterns, unknown correlations, market trends, customer preferences, and other useful business insights.

Introduction to Big Data
Big Data is generated from many sources, including social media, mobile apps, online shopping, sensors, banking systems, and healthcare records. Organizations analyze this data to identify patterns, make better decisions, improve services, and predict future trends.
Big Data is a collection of extremely large and complex datasets that cannot be stored, processed, or analyzed efficiently using traditional data processing tools.
Big Data is a term used to describe very large and complex amounts of data that grow continuously. Traditional databases cannot efficiently store or process this data, so special technologies such as Hadoop and Spark are used.
Hadoop is the core framework of Big Data, but Big Data is a broader field that includes many technologies for storing, processing, and analyzing data.

Real Examples of Big Data
1.   Netflix – Recommends movies and shows.
2.   Amazon – Suggests products based on your purchases.
3.   Google Maps – Shows the best route using live traffic.
4.   Banks – Detect fraudulent transactions.
5.   Hospitals – Analyze patient data for better treatment.
6.   Facebook/Instagram – Recommend posts and ads.
7.   Uber/Ola – Match drivers with riders and find the fastest route.
8.   Weather Forecasting – Predict weather using satellite data.

The 5 Vs of Big Data:
•       Volume: Massive amounts of data from various sources (e.g., social media, sensors, logs).
•       Velocity: Speed at which data is generated, collected, and processed.
•       Variety: Different data types – structured, semi-structured, and unstructured.
•       Veracity: Reliability and accuracy of data.
•       Value: Turning data into actionable insights.
Data Sources:
•       Social media
•       IoT devices
•       Mobile apps
•       Financial transactions
•       Web server logs
•       Enterprise systems (e.g., CRM, ERP)

Benefits

Improved decision-making
Increased operational efficiency
Better customer insights
Competitive advantage

Types of Digital Data/Classification of Data

Structured Data:

Structured data is created using a fixed schema and is maintained in tabular format. The elements in structured data are addressable for effective analysis. It contains all the data which can be stored in the SQL database in a tabular format. Data that is organized into a fixed format, typically stored in relational databases or spreadsheets.

Easy to search and analyze
·      Stored in rows and columns
·      Uses predefined schemas
·      Examples
·      Relational data, Geo-location, credit card numbers, addresses, Customer records in a CRM etc.
Unstructured Data:
•       It is defined as the data in which is not follow a pre-defined standard.

•       This kind of data is also not fit for the relational database because in the relational database
•       Data that has no predefined format or structure. Store Unstructured data there are many platforms to handle it like No-SQL Database.

•       Examples -Difficult to search, process, and analyze Often requires natural language processing (NLP), image recognition, or other complex tools
•       Word, PDF, text, media logs, etc.

• Semi-Structured Data

Semi-structured data is information that does not reside in a relational database but that have some organizational properties that make it easier to analyze.

Example-XML data.

Hadoop is an open-source framework used for storing and processing large-scale data.

It was developed by the Apache Software Foundation.

Hadoop is an open-source framework, primarily written in Java.

It's designed to handle "big data" by breaking down workloads into smaller, parallel tasks that can be executed simultaneously on multiple machines, making it scalable, fault-tolerant, and cost-effective.

History of Hadoop

2003–2004: Google published papers on the Google File System (GFS) and MapReduce, which inspired open-source developers.

2005: Doug Cutting and Mike Cafarella adapted these ideas into the Nutch project.

2006: Hadoop was born when Cutting joined Yahoo!, which supported its development. Named after his son's toy elephant.

2008: Hadoop became a top-level Apache project.

2011: Hadoop 1.0 released, using MapReduce and HDFS.

2013: Hadoop 2.0 introduced YARN, allowing better resource management and support for more processing models like Apache Spark.

2015–Present: Usage declined due to cloud-native tools and simpler alternatives like Snowflake, Databricks, and AWS services.

Hadoop is an open-source framework for storing and processing large data sets across clusters of computers.
Core Components:

HDFS: Hadoop Distributed file system for storing data.
MapReduce: Processes data in parallel.

YARN: Manages resources and job scheduling.
Common: Shared tools and libraries.

Key Features:
Scalable

Fault-tolerant

Cost-effective
Handles all types of data

Used for: Big Data processing, especially in batch jobs over massive datasets.

•Hadoop was created by Doug Cutting, software developer who was working on handling large-scale data processing.

•The Name "Hadoop" Doug Cutting’s young son (around 2 years old at the time) had a toy elephant that he affectionately called “Hadoop.”
When Doug was looking for a name for his new open-source project, he wanted something.

•       Easy to remember
•       Not already taken as a domain
•       Unique
•       If you have a huge amount of data (like from social media, sensors, or e-commerce logs) that can’t fit on one computer.
•       Hadoop Store that data across many computers (nodes).
•       Process that data in parallel, much faster than a single machine could.

Core Components of Hadoop:
•       HDFS (Hadoop Distributed File System)

•       A special file system that splits large files into blocks and stores them across multiple machines.
•       It ensures fault tolerance.
MapReduce
•       A programming model for processing data in parallel.
•       It works in two steps:
•      Map: Breaks the task into smaller tasks.
•      Reduce: Combines the output from Map tasks to produce the final result.
•      YARN (yet another Resource Negotiator) Manages and schedules resources (CPU, memory) across machines.

Hadoop Common-The set of shared utilities and libraries used by other Hadoop modules.

Example Use Cases

• Processing logs for web analytics.
•  Analyzing social media trends.
•  Data warehousing and reporting.
•  Machine learning with big datasets (combined with tools like Apache Spark).

Technologies Built on Hadoop:
Apache Hive – SQL for Hadoop.

Apache Pig – A scripting language for analyzing large data sets.

Apache HBase – NoSQL database on HDFS.
Apache Spark – A faster, in-memory alternative to MapReduce.

Hadoop Work
• Data is loaded into HDFS, where it's split into blocks and distributed across DataNodes.

• MapReduce jobs are submitted to the Resource Manager.
• The job is divided into map tasks, each working on a block of data.
• Map tasks produce intermediate results, which are shuffled and sorted.

• Reduce tasks aggregate results and generate final output.
• The results are stored back in HDFS or passed to other applications.

Advantages of Hadoop

•  Scalability: Easily scale to thousands of machines.
• Cost-effective: Uses low-cost hardware to process big data.
•  Fault Tolerance: Automatic recovery from node failures.
•  High Availability: Data replication ensures no loss even if nodes fail.
•  Flexibility: Can handle structured, semi-structured and unstructured data.
•  Open-source and Community-driven: Constant updates and wide support.

Disadvantages:
•  Not ideal for real-time processing (better suited for batch processing).

•Complexity in programming with MapReduce.
•   High latency for certain types of queries.
•  Requires skilled professionals to manage and develop.
Applications
•   Hadoop is used across a variety of industries:

• Banking: Fraud detection, risk modeling.
• Government and Research

•  Retail: Customer behavior analysis, inventory management.
•  Machine Learning and AI Data Processing
•  Healthcare: Disease prediction, patient record analysis.
•  Telecom: Network performance monitoring.

• Social Media: Trend analysis, user recommendation engines.

Apache Hadoop software

• The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
• It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

• Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

What is Hadoop Streaming?

•       It is a utility or feature that comes with a Hadoop distribution that allows developers or programmers to write the Map-Reduce program using different programming languages like Ruby, Perl, Python, C++, etc.
•       We can use any language that can read from the standard input (STDIN) like keyboard input and all and write using standard output (STDOUT).

•       We all know the Hadoop Framework is completely written in java but programs for Hadoop are not necessarily need to code in Java programming language.

Analysis with the UNI
1. Easy text processing
UNIX commands like cat, grep, sort, awk, and sed are very efficient for processing text data.

2. Supports stdin and stdout

Hadoop Streaming communicates with mapper and reducer programs through standard input and output, which UNIX tools naturally support.

3. Shell scripting

You can quickly write shell scripts to perform data-processing tasks without using Java.
4. Easy integration

UNIX utilities can be combined using pipes (|) to create simple and powerful data-processing workflows.

5. Fast and lightweight

UNIX command-line tools are lightweight and well-suited for handling large text files.

Hadoop Cluster

A Hadoop Cluster is a group of interconnected computers (nodes) that work together to store and process large amounts of data using Hadoop. Hadoop Cluster is used in Hadoop Streaming because the streaming jobs run on the cluster, not on a single computer.
Use of Hadoop Cluster in Hadoop Streaming

1. Runs Mapper and Reducer programs
The cluster executes the mapper and reducer scripts (Python, Shell, Perl, etc.) on different worker nodes.
2. Parallel Processing
Large input files are divided into blocks, and multiple nodes process them simultaneously, making execution much faster.

Key Features:

Language Flexibility: Allows the use of various programming languages for MapReduce jobs.

Ease of Use: Simplifies the process of writing MapReduce jobs by allowing the use of standard input and output for communication between Hadoop and the scripts.

Versatility: Enables the integration of a wide range of scripts and executables, making it a versatile tool for data processing.

Hadoop Ecosystem was developed to store, process, and analyze massive volumes of Big Data efficiently, reliably, and at low cost.

The Hadoop Ecosystem was developed to efficiently store, process, and analyse large volumes of Big Data that traditional databases could not handle effectively.

Reasons for Developing the Hadoop Ecosystem

1. To handle Big Data

Traditional databases are not efficient for storing and processing massive datasets.

2. To process data faster
Hadoop uses parallel processing, allowing multiple computers to work on the same task simultaneously.

3. To store different types of data
It supports structured, semi-structured, and unstructured data such as text, images, videos, and logs.

4. Cost-effective storage
Hadoop runs on inexpensive commodity hardware, reducing infrastructure costs.

Component	Function
HDFS (Hadoop Distributed File System)	Stores large amounts of data across multiple machines.
MapReduce	Processes data in parallel using Map and Reduce tasks.
YARN (Yet Another Resource Negotiator)	Manages cluster resources and schedules jobs.
Hive	Provides an SQL-like language (HiveQL) for querying data.
Pig	Uses Pig Latin scripting language for data processing.
HBase	A NoSQL database for real-time data storage and retrieval.
Sqoop	Transfers data between Hadoop and relational databases (MySQL, Oracle, etc.).
Flume	Collects and transfers log and streaming data into Hadoop.
Oozie	Schedules and manages Hadoop workflows.
ZooKeeper	Coordinates and synchronizes distributed applications.
Spark	Fast data processing engine that performs in-memory computation.
Mahout	Provides machine learning algorithms for big data.
Avro	Used for efficient data serialization and data exchange.

Here are many tools in the Hadoop Ecosystem because one tool cannot perform all Big Data tasks efficiently. Each tool has a specific responsibility, and together they complete the entire data processing workflow.

·      HDFS → Stores data.
·      YARN → Manages resources and schedules
·      MapReduce/Spark → Processes data.
·      Hive → Queries data using SQL.
·      Pig → Simplifies data processing with scripts.
·      HBase → Provides real-time data storage access.
· Sqoop → Imports and exports data between databases and Hadoop.
·      Flume → Collects log and streaming data.

· Oozie → Schedules and manages workflows.

· ZooKeeper → Coordinates distributed services

IBM’s Big Data Strategy

IBM’s Big Data Strategy is a comprehensive, enterprise-grade framework designed to help organizations collect, store, manage, and extract actionable insights from massive volumes of structured and unstructured data.
IBM's approach uniquely fuses big data with Hybrid Cloud infrastructures and Artificial Intelligence (AI) to turn raw data into immediate business value.

IBM’s Key Pillars

Data Fabric: Connects data across multiple clouds and local servers without needing to move it.
AI Integration: Uses Watsonx to automate data prep and run predictive machine learning models.

Open Source: Integrates with Apache Spark and Hadoop to prevent vendor lock-in.
Automated Governance: Built-in compliance tools to automatically secure data and meet global regulations.

Info Sphere Big Insights

What it is: IBM's enterprise version of the Apache Hadoop framework.

Purpose: Stores and processes massive amounts of unstructured data (like logs and social media).
Key Feature: Adds corporate security, admin tools, and Big SQL to query Hadoop using standard SQL.

Big Sheets

What it is: A web-based data tool embedded inside Big Insights.
Purpose: Gives business analysts a familiar, Excel-like spreadsheet interface.

Key Feature: Allows users to filter, join, and visualize billions of rows of big data without writing complex code.
Storage: Move from Hadoop HDFS → Cloud Object Storage (COS).
Processing: Move from MapReduce → Apache Spark (much faster).
Analytics: Move from Big Sheets → Watson Studio / Data Refinery.

Analyzing data with UNIX tools

UNIX command-line tools remain a fundamental approach to data analysis, particularly for preprocessing, log analysis, text mining, and handling large datasets before importing them into tools such as Python, R, or databases.
Modern Big Data systems did not reinvent text processing; they took UNIX concepts and scaled them across thousands of computers.

UNIX tools
·      Fast even for large text files.
·      Simple and lightweight.
·      Easy to automate using shell scripts.
·      Can combine multiple commands into powerful pipelines.

UNIX to Big Data

· UNIX tools (grep, awk, sed, sort, cut, wc) are used to analyze small to medium-sized datasets on a single machine.
· Hadoop is used to store and process Big Data across multiple machines (cluster).

· UNIX tools are suitable for text processing and basic data analysis, while Hadoop is designed for distributed storage and parallel processing.

· Hadoop commonly runs on Linux/UNIX systems, so knowledge of UNIX commands is useful for working with Hadoop.

· UNIX tools are not part of Hadoop, but they complement Hadoop by helping users manage files, preprocess data, and administer Linux-based Hadoop environments.

UNIX tools are used for data analysis on a single system, whereas Hadoop is used for distributed storage and processing of large-scale (Big Data) datasets across multiple systems.

Hadoop is Distributed UNIX:

A Hadoop MapReduce job uses the exact same logic as a UNIX pipeline. "Map" is like grep or cut (filtering/selecting data), and "Reduce" is like sort and uniq -c (aggregating data).

· Streaming Foundations: Big Data frameworks like Apache Kafka and Apache Flink rely on "data streams." UNIX invented data streaming 50 years ago by passing text line-by-line through pipes (|) without overloading system memory.

· Process of the information, and the information is record-oriented and semi-organized.

· The data is in line-oriented ASCII format & and the line is a record.

·      it’s feasible to parallelize the processing-    datasets may grow beyond the capacity of a single machine, requiring the use of multiple machines for processing.

Analyzing Data with Hadoop
·      To discover patterns and trends in large datasets.
·      To generate business insights for better decision-making.
·      To process huge amounts of data quickly using parallel processing.
·      To summarize and filter raw data into meaningful information.
·      To support applications such as customer behaviour analysis, fraud detection, recommendation systems, and log analysis.

·      Data is analyzed in Hadoop to process large volumes of data efficiently and extract meaningful insights.
·      To take advantage of the parallel processing that Hadoop provides, as a MapReduce job.

·      Map Reduce works by breaking the processing into two phases: the map phase and the reduce phase.
·      Each phase has key-value pairs as input and output, the types of which may be chosen by the programmer.

Assignment No. 1

Define Big Data.
Explain 5 Vs in Big Data.
Define Big Data Analytics.
Explain types of Digital Data with examples.
Explain History of Hadoop.
Explain Hadoop Streaming.
What are the advantages and disadvantages of Hadoop?
Why is UNIX used in Hadoop Streaming?
Why is Hadoop called a distributed framework?
Explain the components of the Hadoop Ecosystem.
Explain the applications of Hadoop in real life.
What is Hadoop Ecosystem?
What is IBM Big Data Strategy?
What is Big Insights and Big Shifts?

Milan Tomic

Hi. I’m Designer of Blog Magic. I’m CEO/Founder of ThemeXpose. I’m Creative Art Director, Web Designer, UI/UX Designer, Interaction Designer, Industrial Designer, Web Developer, Business Enthusiast, StartUp Enthusiast, Speaker, Writer and Photographer. Inspired to make things looks better.