Can't Find What You Need?
Have a Suggestion?
Browse by Category
To access the DTN follow the procedures below. More information about the DTN is available here: What is DTN? |
---|
Click https://servicecatalog.uncc.edu/service/research-computing/high-performance-computing-research . When your cluster account is ready, please move on to the next action item.
If you already have a cluster account, please go to the next action item.
Visit: https://servicecatalog.uncc.edu/service/research-computing/high-performance-computing-research
Please create a ticket to URC requesting access to DTN interactive node (dtn.uncc.edu)
There are 3 main ways to transfer data from the internet using the DTN: 1) within an interactive session, 2) through a batch submission, and 3) using Globus GridFTP.
You must first log in to the HPC interactive node by SSH'ing to hpc.uncc.edu . More info can be found here: How do I log in to the cluster?
Once logged into hpc.uncc.edu, you can issue the following command:
srun --partition=DTN --nodes=1 --ntasks=1 --mem=4gb --time=1:00:00 --job-name=dtn_ia --pty /bin/bash
( ...where "--time=" is set to the amount of time you think you will need on the DTN; adjust accordingly. )
Once you get an interactive session on the DTN, you can now use your favorite transfer method to download data: wget, curl, rsync, ftp, etc. More info can be found here: How do I transfer data from the internet?
If you have a data file or dataset that you think will take a very long time to download, instead of logging in interactively to the DTN to initiate the download, consider creating a batch job to execute the download, and submit it to the DTN partition. Here is an example SRA Toolkit job, which uses prefetch and fastq-dump to download an NCBI data file (you can do this on hpc.uncc.edu; no need to log in to the DTN):
#! /bin/bash
#SBATCH --job-name=fastqdump_SRR8378xx
#SBATCH --partition=DTN
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --mem=4gb
#SBATCH --output=%x_%j.out
#SBATCH --time=4:00:00
cd $SLURM_SUBMIT_DIR
module load sra-tools/2.11.0
for (( i = 19; i <= 56; i++ ))
do
prefetch SRR8378$i
done
This will download SRA files SRR837819 through SRR837856. Save the above submit script to a file; I'll name mine "submit.slurm" for my example. Once saved, submit it to the scheduler using the command: sbatch submit.slurm
Globus GridFTP
Follow the instructions here (scroll down to "Method 2") to set up your Globus account if you would like to transfer files using GridFTP.