Providing Elasticity to Data-Node

🔅Integrating LVM with Hadoop and providing Elasticity to DataNode Storage :

✅We will be using AWS for launching RHEL8 instance to setup Hadoop.

Task Requirement :

✍JDK(Java Development Kit )

✍Hadoop software

For some reason, we will be disabling SE-Linux in RHEL8:

#setenforce 0
#getenforce

Configuring hadoop cluster now :

SOURCE : shorturl.at/vFQSZ

⚡Master Node Configuration

🖋Creating directory in namenode
#mkdir /name1

🖋Configuring hdfs-site.xml

#vi hdfs-site.xml
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/name1</value>
</property>
</configuration>

🖋vi core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://<ip_of_master>:<custom_port></value>
</property>
</configuration>

Formatting namenode for first time:

#hadoop namenode -format

Starting NameNode:

#hadoop-daemon.sh start namenode

Listing connected data nodes:

#hadoop dfsadmin -report

⚡Slave Node Configuration:

Creating directory in slave node:

#mkdir /data2

Configuring hdfs-site.xml:

#vi hdfs-site.xml
<configuration>
<property>
<name>dfs.data.dir</name>
<value>/data2</value>
</property>
</configuration>

Configuring core-site.xml:

#vi core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://<ip_of_master>:<custom_port></value>
</property>
</configuration>

Starting data node:

#hadoop-daemon.sh start datanode

Listing connections with Master node:

#hadoop dfsadmin -report

🔴Integrating LVM concept in Datanode

Attach EBS volume to this EC2 instance:

checking if the disk is attached or not :

#lsblk

#fdisk -l

✍Creating Physical Volume :

#pvcreate <hdd-device>
#pvdisplay <hdd-device>

✍Creating Volume Group:

#vgcreate <vg_name> <hdd-device>
#vgdisplay <vg_name>

✍Creating Logical Volume :

#lvcreate --size +<value>G/M/K --name <lv_name> <vg_name>
#lvdisplay <vg_name>/<lv_name>

✍Formating LV :

#mkfs.ext4 /dev/<vg_name>/<lv_name>

✍Mounting folder to LV:

#mount /dev/<vg_name>/<lv_name>
#df -h

Modified hdfs-site.xml in datanode:

✍Restarting the Data node and listing the connected node:

#hadoop-daemon.sh start datanode
#hadoop dfsadmin -report

✍Extending logical volume :

#lvextend --size +<value>G/M/K /dev/<vg_name>/<lv_name>

✍Resizing the logical volume
#resize2fs /dev/<vg_name>/<lv_name>

Size of the logical volume is increased now :

Checking if size is updated as in cluster or not

Thus , we have successfully integrated LVM with hadoop implementing concept of elasticity in datanode……..

Keep sharing ….

Happy learning……😊😊

Do check : Increasing or Decreasing the Linux static partition