Setup default myDre VM

Repository for installation of frequently used software on empty virtual machines (VM) on myDRE. This might also work on SURF Research Cloud but has not been tested yet.

Developed by:

• Barbera van Schaik, b.d.vanschaik@amsterdamumc.nl
• Antoine van Kampen, a.h.vankampen@amsterdamumc.nl
• Perry Moerland, p.d.moerland@amsterdamumc.nl

Documentation

See Barbera's slide deck for examples of how to make this work on MyDre and SURF.

Requirements

The Ansible script assumes a virtual machine running Ubuntu. It might also work on Debian.

Setting up an Ubuntu VM

• Create a new Ubuntu v22 VM on myDre (warning: Ubuntu 24 does not work)

• Login to the VM using your myDRE account using RDP (X environment), or SSH (terminal).

• In the next step only do 'upgrade' if you are using Ubuntu version 22. Currently, you cannot upgrade Ubuntu 24.

sudo apt update
sudo apt upgrade 

• Stop and restart the VM, and login again.

sudo apt install software-properties-common
sudo apt install ansible git

This installed:

• ansible
• git

To configure git:

# Skip the next line on SURF Research Cloud
git config --global http.proxy http://proxy.mydre.org:3128 

#Skip the next two lines on myDre
git config --global user.email "you@example.com" # email associated with your github account
git config --global user.name "Your Name" # username associated with your github account

This will make a ~/.gitconfig file

Note:

• So far, pushing to a github directory on myDre does not work. On SURF Research Cloud it does work.

• For SURF Research Cloud: when pushing changes to a repository, you will be asked for your username and password. The password should be a 'fine-grained token' that you can set from https://github.com/settings/profile --> Developer Settings --> Personal Access Tokens. Set resource owner to EDS-Bioinformatics-Laboratory. Select all or individual repositories. Set the permission for 'Contents' to read/write.

• Clone the ansible repository

cd ~ 
git clone https://github.com/EDS-Bioinformatics-Laboratory/ansible.git
cd ansible

Alternative for SURF Research Cloud

This approach does not seem to work because ssh on myDre is disabled.

Transfer your git RSA key to the virtual machine in ~/.ssh/ Change permissions to rw for the user only. Then clone the repository via SSH (not HTTPS)

cd ~
git clone git@github.com:EDS-Bioinformatics-Laboratory/ansible.git
cd ansible

• Modify the configuration file config.yml
  • Change the username to your own myDre username.
  • Change the local_mount path
  • Change the root_path
  • Set the use_proxy setting to false when working on Surf Cloud
  • The selection tags do currently not work.

Installation and configuration of software with Ansible

• Execute playbooks

Notes:

• In case you need to debug, you can start by using -vvv instead of -v
• All steps are mandatory unless indicated.

Either run the playbooks individually, or run everything at once via the Main.yml playbook

Main playbook, to install everything

sudo ansible-playbook -i hosts -v Main.yml

Individual playbooks

sudo ansible-playbook -i hosts -v General.yml

This installed:

• rclone
• gedit

General.yml created ~/Desktop/RD to which you can mount your Research Drive account (see below). The local_mount in config.yml should point tho this directory. In addition, it cloned the ENCORE repository in ~/.

General.yml also sets the timezone:

• On Ubuntu, setting the timezone is immediate.
• timedatectl set-timezone Europe/Amsterdam (and Ansible’s timezone module) updates /etc/localtime (symlink) and /etc/timezone.
• From that moment on, all new time calculations/display use the new timezone. You don’t need to reboot.
• What does not automatically happen is “time synchronization” in the sense of adjusting the clock value via NTP. That’s separate.
• Timezone change = changes how the current time is interpreted/displayed (offset, DST rules).
• NTP sync = adjusts the system clock to be accurate.
• If your system time is already correct, you don’t need to wait for anything.
• Enabling NTP doesn’t always “snap” the clock instantly; it may take a short moment to converge depending on network and current drift, but usually it’s quick.

Note: at this point you can mount your Research Drive account following the instructions in the section further down in this document, or you can continue to first install software with the remaining playbooks. Mounting Research Drive at this point may help to easily transfer files and copy-paste (see below) in case you need to debug or change files.

sudo ansible-playbook -i hosts -v Conda.yml  
source ~/.bashrc

This installed:

• conda base environment
• numpy, pandas, notebook 7
• and activated conda

sudo ansible-playbook -i hosts -v Vscode.yml    #OPTIONAL

This installed:

• Visual Studio Code

Notes:

• VScode can run from the commandline: code or vscode (first run next playbook)

• When you start VScode for the first time, then pay attention to any message in VSCode. You may be asked to update certain parts.

• It will ask you to choose a password for new keyring: use your mydre password.

• If you want to use the same extensions as you have on your Windows/MacOS, then enable syncrhonization in VSCode. Alternatively, install the extensions manually. To enable synchronization:

  • go to Settings Sync (ctrl-shift-p)

  • Click on Backup and Sync Settings

  • Sign in to your github (or microsoft) account

  • At this moment it does not seem to sync the extensions. However, you can install extensions from the vscode menu.

  • Copilot is installed and you can login to your own copilot account.

sudo ansible-playbook -i hosts -v Bashrc.yml
source ~/.bashrc

This added the following to .bashrc

• Aliases for rdmount and rdumount (i.e., (un)mounting Research Drive; first run rclone config; see below)
• Alias h=history
• Alias lsg=ls -ag
• Alias aclone to clone the ansible git repository
• Alias eclone to clone the ENCORE git repository
• Alias vscode=code

sudo ansible-playbook -i hosts -v Compilers.yml   

This installed:

• gcc
• g++
• gfortran

sudo ansible-playbook -i hosts -v R.yml   #OPTIONAL

This installed:

• R
• RStudio
• Rtools
• Packages: renv, ggplot

Notes:

• When starting RStudio you may get a pop up that a newer version is available, since the version to be installed has now been hardcoded. You should be able to download a newer version via the browser and then install it with 'sudo apt install .deb' in the terminal.
• One can use the 'R_only.yml' and 'RStudio.yml' to seperately install these tools. This is also convenient when installation of one of them fails, for example because a newer version has been detected (e.g., if the RStudio installation has already been updated manually).

sudo ansible-playbook -i hosts -v Julia.yml  #OPTIONAL
source ~/.bashrc

This installed:

• Julia programming language version 1.12.4

• juliaup

• Change the playbook to download another/latest version, or use

  • juliaup self update followed by
  • juliaup update to update to the latest version.

• Note: if you get an error 'The read operation timed out' then run the Playbook again.

sudo ansible-playbook -i hosts -v Containers.yml   #OPTIONAL

This installed:

• Apptainer and Docker

Notes:

• Docker has been tested and seems to work.
• Apptainer has not been tested.

sudo ansible-playbook -i hosts -v Emacs.yml   #OPTIONAL
	                                          # This should install packages but I don't think it works. 
										      # However, these packages seem to already be built in into emacs

Mount (SURF) Research Drive

Make sure you have a Research Drive account.

Use Rclone to mount Research Drive. Rclone is installed by one of the ansible playbooks above. Rclone is a command-line program to manage files on cloud storage. This makes it much easier to transfer files between your local computer and myDre in comparison to the default upload/download system.

If needed, then first install Rclone manually:

sudo -v ; curl https://rclone.org/install.sh | sudo bash
or
sudo apt install rclone

Connect to your Research Drive account See: SURF Wiki for more information on how to configure Rclone and use it to mount your Research Drive.

Webdav URL (example): https://amsterdamumc.data.surf.nl/remote.php/dav/files/a.h.vankampen@amsterdamumc.nl

Configuration is stored in: .config/rclone

Example commands

rclone ls RD:
rclone copy /my/folder RD:my/destination/folder

Working with large objects When you want to upload large files to Research Drive, we recommend using a timeout of 10 minutes per gigabyte of the largest source file.

As an example, the largest file in the source directory is 5GB. Calculating the argument for --timeout gives: 10 minutes x 5GB = 50 minutes:

rclone copy --use-cookies --timeout 50m ~/my_5gb_file.bin RD:my/destination/folder

Important: Since this can cause data loss, test first with the --dry-run flag to see exactly what would be copied and deleted. Note that files in the destination won’t be deleted if there were any errors at any point. If my/destination/folder doesn’t exist, it is created and the contents of /my/folder goes there.

Use Rclone to mount file systems in user space Using Rclone to mount a file system in user space is done as follows:

rclone mount --use-cookies --timeout 15m RD: /path/to/local/mount --vfs-cache-mode writes

The flag --use-cookies is needed, to get you always on the same Research Drive back-end, to prevent file lock between back-ends. The timeout flag is useful for uploading large files, we recommend using a timeout of 10 minutes per gigabyte of the largest source file. The last part --vfs-cache-mode writes allows applications to write data to the mount.

You can unmount this file system by:

fusermount -u /path/to/local/mount

Note: the ansible playbook bashrc will add rdmount and rdumount to .bashrc

Copy-Paste workaround

It is not possible to copy-paste to and from myDre. One work around is to make use of the mounted Research Drive: just put a file copypaste.txt in the mounted directory (eg ~/Desktop/RD) and sync with your directory on Research Drive (eg. /Bioinformatics/myDre):

cd ~/Desktop/RD
rclone sync RD:/Bioinformatics/myDre .

or vice versa

rclone sync . RD:/Bioinformatics/myDre

Troubleshooting

• With Vscode.yml and Conda.yml you may sometimes get an error like: ".......Conflicting values set for option Signed-By regarding source https://packages.microsoft.com/repos/code .......". It is unclear why this happens. So far I fixed this by deleting the files in /usr/share/keyring that were installed at the moment you executed the ansible playbook. In addition, I deleted the conda.list and/or vscode.list and/or vscode.sources, and packages_microsfot_com_repost_code.list. See also: here

GitHub authorisation

Instead of using a personal access token, it should be possible to use the following:

ssh-keygen -t ed25519 -C "you@example.com"
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
cat ~/.ssh/id_ed25519.pub

and next, add the printed public key to GitHub → Settings → SSH and GPG keys. However, for unknown reason this didn't seem to work.

​

External Access list

[here]

The following websites should be added to the myDre External Access List if these are not there already. This ensures that myDre can access these sites. The External Access List is global for all Workspaces and VMs. In principle, this list will never be deleted and, therefore, only needs to be setup once. In case you need access to another website (e.g., to install an R package), then simply add it to the list.

I also provide the list here in case is magically disappears on myDre. Then just copy this list back into the External Access List..

login.live.com
azureedge.net
ocsp.msocsp.com
microsoft.com
vsassets.io
rstudio.org
r-project.org
vscode-sync.trafficmanager.net
vscode.dev
gnu.org
melpa.org
docker.com
install-github.me
office.com
microsoftonline.com
mydre.org
conda.io
anaconda.com
anaconda.org
amsterdamumc.data.surf
julialang.org
google.nl
google.com
surf.nl
rclone.org
ubuntu.com
github.com
githubassets.com
git-scm.com
githubusercontent.com
githubcopilot.com
gitlab.com
conda-forge.org
visualstudio.com
docker.io
gcr.io
posit.co
rstudio.com