You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Current »

This page explains how to setup Gluster network filesystems that can be mounted by multiple virtual machines, in particular, MPI(or GNU/parallel) cluster nodes.

If you have questions and suggestions about this tutorial page and related problems, please, contact Min-Su Shin.

Case A: Filesystem with a single Gluster server

Gluster filesystem can be prepared with a single server and its disk space to serve multiple clients. In this case A, we assume that a single Gluster server is made to be used by multiple VMs (virtual machines) included in a cluster.

Step 1. Creating a single VM

Following KASI Science Cloud : VM instances#Step3.CreateaVMinstance, create a single VM with a VM flavor which fits to your requirement such as the required size of the network filesystem. The created single VM will work as a server serving the Gluster filesystem. Since the required memory for the Gluster server is not huge, the recommended flavor is large-CPU + small-memory for a given size of disk.

Step 2. Configuring the Gluster server

On the created VMs, we install the Gluster server program and configure a Gluster filesystem. For your convenience, you can find a shell script that should be executed by root account on the master node. See https://github.com/astromsshin/cloud_ex and the script https://github.com/astromsshin/cloud_ex/blob/main/tool_setup_glusterfs_single_server.sh which is presented below:

#!/bin/bash

# Directory serving the Gluster filesystem
TARGETDIR="/glusterfs/vol"
# Gluster filesystem volume available for clients
TARGETVOL="gvol"
# Name of the Gluster server
VMNAME="test-gl-vm"

# Install the server
apt install glusterfs-server -y
systemctl enable --now glusterd

# Information
# (optional)
#gluster --version
#systemctl status glusterd

# Prepare the directory
mkdir -p ${TARGETDIR}
# Produce the Gluster filesystem volume with the above directory
gluster volume create ${TARGETVOL} ${VMNAME}:${TARGETDIR} force
# Make the volume available
gluster volume start ${TARGETVOL}

# Information
# (optional)
#gluster volume info
#gluster volume status

Step 3. Mounting the created Gluster filesystemt on client VMs

The created network filesystem can be mounted on client VMs by using mount command with required options. A simple script is available as https://github.com/astromsshin/cloud_ex/blob/main/tool_setup_glusterfs_client_all_nodes-single_server.sh  in the github repository https://github.com/astromsshin/cloud_ex. The script, which needs to be executed on the client cluster master node, is given below:

#!/bin/bash

# Name of the cluster which defines the clients' names
CLUSTERNAME="ml-image"
# Last integer index of the minions in the cluster
MINIONLASTIND="8"
# IP of the Gluster server
GLUSTERSERVERIP="10.0.100.150"
# Name of the Gluster server
GLUSTERSERVERNAME="test-gl-vm"
# Directory name which is a mount point on clients
TARGETDIR="/mnt/gluster"


RUNCMD="apt -y install glusterfs-client; echo ${GLUSTERSERVERIP} ${GLUSTERSERVERNAME} >> /etc/hosts;mkdir ${TARGETDIR}; mount -t glusterfs ${GLUSTERSERVERNAME}:/gvol ${TARGETDIR}; chmod -R 777 ${TARGETDIR}"

echo "... setuping on ${CLUSTERNAME}-master"
echo $RUNCMD | bash

for ind in $(seq 0 ${MINIONLASTIND})
do
echo "... setuping on ${CLUSTERNAME}-minion-${ind}"
ssh ${CLUSTERNAME}-minion-${ind} "${RUNCMD}"
done

In the above script, we assume that the filesystem is mounted on multiple VMs created as explained in Deploy Message Passing Interface (MPI) (and/or GNU parallel) cluster.

Case B: Filesystem with multiple Gluster servers (in the form of distributed volumes)

We explain how to setup the Gluster filesystem in the form of distributed volumes by using multiple Gluster servers. As depicted in the following figure and explained in https://docs.gluster.org/en/latest/Administrator-Guide/Setting-Up-Volumes/,

there is a type of distrbiuted volume where files are distributed in multiple gluster servers.

Step 1. Creating a basic cluster of VMs

You need to deploy multiple VMs as Gluster servers. The basic procedure follows the tutorial given in Deploy Message Passing Interface (MPI) (and/or GNU parallel) cluster. However, you do not need to setup MPI and NFS share directories explained in the tutorial. Use a cluster template KASI-Cluster-Basic-Block in the KASI Cluster Templates to produce a VM cluster for the Gluster filesystem with an extra block device (i.e., /dev/vdb). The recommended configuration of the flavor is C2M8D40, and you need to estimate the required number of minions for the required Gluster volume size and the chosen extra block size. For example, if you need 30TB volume, you need 9 minions as well as 1 master (i.e., 10 virtual machines in total) with the block size 3TB (i.e., 3072GB).

Step 2. Configuring the Gluster server with multiple bricks on the cluster VMs

The configuration of the Gluster VMs can be done with the provided script https://github.com/astromsshin/cloud_ex/blob/main/tool_setup_glusterfs_multiple_servers.sh on the github repository https://github.com/astromsshin/cloud_ex. The script is given below:

#!/bin/bash

# Run this script with root permission
# on the master node of the GlusterFS cluster

# Directory serving the Gluster filesystem
TARGETDIR="/glusterfs/vol"
# Gluster filesystem volume available for clients
TARGETVOL="gvol"
# Name of the Gluster cluster
CLUSTERNAME="glustercluster"
MINIONLASTIND="0"

# Install the server
echo "... install on ${CLUSTERNAME}-master"
apt install glusterfs-server -y
systemctl enable --now glusterd
for ind in $(seq 0 ${MINIONLASTIND})
do
  echo "... install on ${CLUSTERNAME}-minion-${ind}"
  ssh ${CLUSTERNAME}-minion-${ind} "apt install glusterfs-server -y; systemctl enable --now glusterd"
done

# Probing
for ind in $(seq 0 ${MINIONLASTIND})
do
  echo "... probing ${CLUSTERNAME}-minion-${ind}"
  gluster peer probe ${CLUSTERNAME}-minion-${ind}
done

# Prepare the directory
echo "... creating directory on ${CLUSTERNAME}-master"
mkdir -p ${TARGETDIR}
for ind in $(seq 0 ${MINIONLASTIND})
do
  echo "... creating directory on ${CLUSTERNAME}-minion-${ind}"
  ssh ${CLUSTERNAME}-minion-${ind} "mkdir -p ${TARGETDIR}"
done

# Information
gluster --version
gluster peer status

# Produce the Gluster filesystem volume with the above directory
VOLSTR="${CLUSTERNAME}-master:${TARGETDIR}"
for ind in $(seq 0 ${MINIONLASTIND})
do
	VOLSTR="${VOLSTR} ${CLUSTERNAME}-minion-${ind}:${TARGETDIR}"
done
gluster volume create ${TARGETVOL} ${VOLSTR} force
# Make the volume available
gluster volume start ${TARGETVOL}

# Information
gluster volume info

Step 3. Mounting the created Gluster filesystem on client VMs

The configuration of the Gluster firesystem affects how the client VMs mount and access the Gluster filesystem. The following script, which is available on the github repository  https://github.com/astromsshin/cloud_ex as https://raw.githubusercontent.com/astromsshin/cloud_ex/main/tool_setup_glusterfs_client_all_nodes-multiple_servers.sh.

#!/bin/bash

# Name of the cluster which defines the clients' names
CLUSTERNAME="ml-image"
# Last integer index of the minions in the cluster
MINIONLASTIND="8"
# Directory name which is a mount point on clients
TARGETDIR="/mnt/gluster"

# IP of the Gluster server
GLUSTERSERVERIP="10.0.100.120 10.0.100.191"
# Name of the Gluster server
GLUSTERSERVERNAME="test-basic-master test-basic-minion-0"

# Updating /etc/hosts
IPARRAY=($GLUSTERSERVERIP)
NAMEARRAY=($GLUSTERSERVERNAME)
# ... Master
for (( i=0; i<=${#IPARRAY[@]}; i++ ))
do
	echo ${IPARRAY[$i]} ${NAMEARRAY[$i]} >> /etc/hosts
done
# ... Minion
for ind in $(seq 0 ${MINIONLASTIND})
do
	for (( i=0; i<=${#IPARRAY[@]}; i++ ))
	do
	  ssh ${CLUSTERNAME}-minion-${ind} "echo ${IPARRAY[$i]} ${NAMEARRAY[$i]} >> /etc/hosts"
	done
done

# Rest of tasks 
RUNCMD="apt -y install glusterfs-client; mkdir ${TARGETDIR}; mount -t glusterfs ${NAMEARRAY[0]}:/gvol ${TARGETDIR}; chmod -R 777 ${TARGETDIR}"
# ... Master
echo "... setuping on ${CLUSTERNAME}-master"
echo $RUNCMD | bash
# ... Minion
for ind in $(seq 0 ${MINIONLASTIND})
do
  echo "... setuping on ${CLUSTERNAME}-minion-${ind}"
  ssh ${CLUSTERNAME}-minion-${ind} "${RUNCMD}"
done

Since there are multiple Gluster servers, you need to type their ip addresses and names in the above script. You can easily find the ip addresses and names in the cloud dashboard.

Useful Tips

  • No labels