CODEWITHSUNDEEP
hadoophdfs
Sonia Saxena
Sonia Saxena

Reputation: 95

Hadoop pipeline write and parallel read?

We know that a client in hadoop reads data in parallel but the data is written in a pipeline anatomy where one data node writes the data into the other. I know that parallel read makes the system more fault tolerant and faster read. But what is the benefit of a pipeline write? Why doesn't a HDFS client itself write data into each node?

Upvotes: 1

Views: 4008

Answers (2)

foool
foool

Reputation: 1552

1. More efficient bandwidth consumption for the client

In the pipeline write, the client only have to transfer one replica to the first data node, and each node only get and send one replica over the network (except the last data node only receives data), thereby having a much more balanced network bandwidth consumption compared to the client writing three replicas into three different data nodes.

2. Smaller sent/ack window to maintain

The client maintain a much smaller sliding window to record which blocks in the replica are sending to the DataNodes and which blocks are waiting acks to confirm the write has been done. In a pipeline write, the client appear to write data to only one data node.

3. Speed up the write operation for faster write response time

When writing a chunks of 64MB, the client divides the chunk into pieces of size 4KB and sends data pieces in parallel, which greatly make many time-consuming operations (transfer over the network, flush the data to disk) run concurrently, just as the following figure shows.

Actions (DN=DataNode, DP=Data Piece)

Client-(send DP1)

DN1---------(get DP1/send to DN2/flush to disk)------------------(ack DP1)

DN2--------------(get DP1/send to DN2/flush to disk)------(ack DP1)

DN3-----------------------(get DP1/flush to disk)------(ack DP1)

—— —— —— —— —— —— —— —— —— —— —— —— —— —— —— —— ——> Time

This figure above only shows the parallelism of one write pipeline, actually, different pipelines for writing different data pieces can also save a lot by paralleling operations of various pipelines, such as a DataNode can flush data piece 2, get data piece 3 from the client, and ack the data piece 1 at the same time. The following figure shows how different pipelines work in parallel.

Client-(DP1)---(DP2)---(DP3)---

DN1---------(DP1)---(DP2)---(DP3)----------------------(ack DP1)---------(ack DP2)---(ack DP3)

DN2--------------(DP1)---(DP2)---(DP3)----------(ack DP1)---------(ack DP2)---(ack DP3)

DN3-----------------------(DP1)---(DP2)---(ack DP1)---(DP3)---(ack DP2)---(ack DP3)

—— —— —— —— —— —— —— —— —— —— —— —— —— —— —— —— ——> Time

Many optimizations can be done for this parallel processing, such as piggybacking ack message of a previous data piece to the return message of pipeline-writing the latter data piece for saving network transmission.

Upvotes: 2

Sandeep Singh
Sandeep Singh

Reputation: 8010

Suppose you have a file of 128MB and you want to write this file on HDFS.

The client machine first splits the file into block Say block A, Block B then client machine interact with NameNode to asks the location to place these blocks (Block A Block B).NameNode gives a list of datanodes to the clinet to write the data.

Then client choose first datanode from those list and write the first block to the datanode and datanode replicates the block to another datanode, once the second datanode receive the replicated block it gives the block received acknowledgement to primary datanode and the primary datanode update the block information to NameNode

NameNode keeps the information about files and their associated blocks.

Upvotes: 0

Related Questions