...
Becasue the NFS partition can have issues of permissions in being accessed in multiple VM cluster nodes, you need to be careful in choosing accounts for running your applications and in setting file/directory (in particular, NFS directory and files there) permissions. Probably, if you do not need SSH access from external machines and do not have concerns on security with the root account, using the root account might be the easiest way to avoid problems such as file/directory permission. If you need SSH access from external machines, you may need to change the SSH-related configurations for the root account. When you decide to use the ubuntu account with efforts of changing owner and permission of files and directories, you may need to use the script introducted in "Changing password of ubuntu account and preparing key-based ssh login environment in multiple VM nodes" of the useful tips.
...
The same steps of building custom environment is
...
repeated when a new cluster is made again. Are there better ways?
It is recommended to have some simple scripts that can make procedures of preparing custom environments easy and fast. You can have your own scripts of installing apt packages and creating conda environments following the guide given in the useful tips. If you keep the scripts in the provided external storage, which can be accessed in the KASI cloud, or others such as github repository, you can make the cluster nodes to execute your own custom script as a step to build your custom work environments. In terms of using conda environment, you can export your custom conda environment configuration and save them in the external storage (see https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#building-identical-conda-environments). When your code can be provided as pre-compiled binary in the external storage, compilation steps can be skipped for the code.
...