/Apache Hadoop HDFS
Apache Hadoop HDFS Monitoring
HDFS is an open-source distributed filesystem developed to manage large volumes of data distributed on low-cost, high-end hardware. In HDFS architecture, there are two types of nodes called NameNode and DataNode, the first type is responsible for managing data distribution and the other one is responsible for storing data and exchanging it with clients. Moein provides HDFS cluster monitoring by observing both types of nodes. Performance indicators of each type are as follow:
Performance indicators and parameters of NameNode monitored by Moein are:
NameNode General KPIs:
- Total Create Operations
- Total Created Files And Directories
- Total Appended Files
- Number Of GetBlockLocations Operations
- Number Of Rename Operations
- Number Of Directory Listing Operations
- Total Delete Operations
- Total Deleted Files And Directories
- Total Get FileInfo Operations
- Number Of AddBlock Operations
- Number Of GetAdditionalDatanode Operations
- Number Of CreateSymlink Operations
- Number Of GetLinkTarget Operations
- Number Of Files And Directories Listed By Directory Listing Operations
- Number Of Processing Block Reports
- Number Of Journal Transactions
- Journal Transactions Average Time
- Number Of Journal Syncs
- Journal Syncs Average Time
- Number Of Batched Transactions In Sync
- Safe Mode Time
- FSImage Load Time
- Number Of Edits Downloads From Secondary Node
- Average Edits Download Time
- Total Number Of FSImage Downloads From Secondary Node
- Average FSImage Download Time
- Total Number Of FSImage Uploads To Secondary Node
- Average FSImage Upload Time To Secondary Node
Cluster KPIs:
- Total Configured Capacity
- DFS Used Capacity
- DFS Remaining Capacity
- Current Number Of Connections
- Current Number Of Allocated Blocks In The System
- Current Number Of Files And Directories
- Number Of Blocks Pending To Be Replicated
- Number Of Blocks Under Replicated
- Number Of Blocks Scheduled For Replications
- Current FS State
- Number Of Live Datanodes
- Number Of Dead Datanodes
KPIs of DataNodes managed by NameNode:
- DFS Capacity Usage
- DFS Remaining Capacity Percentage
- Non DFS Used Capacity
- Cluster ID
- Block Pool ID
- Version
- Block Pool Used Space
- Block Pool Used Space Percentage
- Total Blocks
- Total Files
- Number Of Missing Blocks
Live Node Information:
- Datanode Name
- Number Of Blocks
- DFS Used Capacity
- Admin State
- Non DFS Used Capacity
- Last Update Time
- Total Capacity
- DFS Capacity Used Percentage
- DFS Remaining Capacity Percentage
Dead Node Information:
- Datanode Name
- Is Decommissioned?
- Dead Time
Monitoring NameNode is enough for HDFS cluster monitoring. However, if DataNode monitoring is desired separately, the following indicators are provided for each DataNode by Moein:
General Datanode Parameters:
- Number Of Active Connections
- Version
- Cluster ID
- Namenode Address
- DFS Free Space
- DFS Used Space
- DFS Used Space Percentage
DataNode KPIs:
- Number Of Bytes Written To Datanode
- Number Of Bytes Read From Datanode
- Data Written Rate To Datanode
- Data Read Rate From Datanode
- Number Of Blocks Written To Datanode
- Number Of Blocks Read From Datanode
- Number Of Replicated Blocks
- Number Of Removed Blocks
- Number Of Verified Blocks
- Number Of Verification Failures
- Local Client Read Operations
- Remote Client Read Operations
- Local Client Write Operations
- Remote Client Write Operations
- Number Of Get Local Path Info Operations
- Number Of Fsyncs
- Number Of Volume Failures
- Number Of Block Read Operations
- Block Read Average Time
- Number Of Block Write Operations
- Block Write Average Time
- Number Of Block Checksum Operations
- Block Checksum Average Time
- Number Of Block Copy Operations
- Block Copy Average Time
- Number Of Block Replace Operations
- Block Replace Average Time
- Number Of Heartbeats
- Heartbeats Average Time
- Number Of Block Reports Operations
- Block Reports Average Time
- Number Of ACK RTT
- ACK RTT Average Time
- Number Of Flushes
- Flushes Average Time
- Number Of Fsync Nanos
- Fsync Nanos Average Time
- Number Of Sending Packet Operations
- Sending Packets Average Time
JVM KPIs:
HDFS is developed in Java and requires JVM to run. Details of JVM indicators are provided in the Environments section.
Communication Protocols:
- REST
- JMX