Managing a Hadoop Distributed File System cluster
NameNode startup
1. NN reads fsimage in memory.
2. NN applies edit log changes.
3. NN waits for block data from data nodes:
▪ NN does not store the physical location information of the blocks.
▪ NN exits SafeMode when 99.9% of blocks have at least one copy that
is accounted for.
1. fsimage is read.
2. Edits log is read and applied.
3. Block information sent
to NameNode.
NN loads the file system status from the edits log file and fsimage during startup.
Then, even though there are enough replicas in the cluster, it waits for DataNodes
to report their blocks before replicating them.
NN remains in SafeMode during this period. For the HDFS cluster, SafeMode for
NN is simply a read-only mode that forbids any modifications to the file system or
blocks. Normally, after DataNodes report that the majority of the file system
blocks are available, NN exits SafeMode automatically.
The following are the HDFS storage files where NN saves its metadata:
fsimage \sedits \sVERSION
Since the previous time the fsimage was updated, an edits in process file has been
accumulating edits (adds and deletes). Depending on whether of two customizable
events occurs first, this edits file is shut off and the modifications are applied to a
new version of the fsimage:
A particular size is reached in the edits file (here 1 MB, but the default is 64 MB).
NameNode startup
1. NN reads fsimage in memory.
2. NN applies edit log changes.
3. NN waits for block data from data nodes:
▪ NN does not store the physical location information of the blocks.
▪ NN exits SafeMode when 99.9% of blocks have at least one copy that
is accounted for.
1. fsimage is read.
2. Edits log is read and applied.
3. Block information sent
to NameNode.
NN loads the file system status from the edits log file and fsimage during startup.
Then, even though there are enough replicas in the cluster, it waits for DataNodes
to report their blocks before replicating them.
NN remains in SafeMode during this period. For the HDFS cluster, SafeMode for
NN is simply a read-only mode that forbids any modifications to the file system or
blocks. Normally, after DataNodes report that the majority of the file system
blocks are available, NN exits SafeMode automatically.
The following are the HDFS storage files where NN saves its metadata:
fsimage \sedits \sVERSION
Since the previous time the fsimage was updated, an edits in process file has been
accumulating edits (adds and deletes). Depending on whether of two customizable
events occurs first, this edits file is shut off and the modifications are applied to a
new version of the fsimage:
A particular size is reached in the edits file (here 1 MB, but the default is 64 MB).
