Computing · Practical guide
Research computing with HPC & HTC
A practical route from your local terminal to a configured environment, a submitted job, and readable results at UW–Madison CHTC.
Start learning01Connect02Submit03Monitor
What you’ll learn
- Choose between tightly coupled and independent workloads.
- Connect, move files, and prepare a Python environment.
- Submit and monitor a Slurm job.
Before you begin Terminal basics
- An approved CHTC account and its assigned access point.
- A terminal on macOS, Linux, or Windows with SSH support.
- Adjust paths, resource requests, and environment names for your project.
01
Choose a computing system
Match the computing system to how your tasks communicate, rather than choosing by job size alone.
HPC · Slurm
Tightly coupled computations that coordinate work across nodes, such as MPI applications.
HTC · HTCondor
Many independent jobs, such as parameter sweeps or separate model runs.
Resource limits depend on the system and partition. Use the current CHTC system overview for limits and policies.
02
Connect and transfer files
Use the access point in your welcome email. The HPC example below uses the currently documented spark-login host.
For an editor-based workflow, see VS Code Remote SSH .
# Local terminal: replace YOUR_NETID with your account name.
ssh YOUR_NETID@spark-login.chtc.wisc.edu
# Copy a file to your home directory (run from the local terminal).
scp input.csv YOUR_NETID@spark-login.chtc.wisc.edu:/home/YOUR_NETID/
# Copy results back to the current local directory.
scp YOUR_NETID@spark-login.chtc.wisc.edu:/scratch/YOUR_NETID/results.zip .
03
Prepare a Python environment
For the HPC example, use a Conda installation available to your compute job, then create a named environment with the packages your analysis needs.
See the HPC software guide for installation policies. HTC jobs need a portable environment or container; see the HTC Conda guide .
# Run after installing Conda according to the CHTC guide.
conda env list
conda create -n research python=3.11
conda activate research
conda install numpy pandas matplotlib
conda env export > environment.yml
conda deactivate
# Run the analysis through a compute job, using the next section.
04
Submit a Slurm job
An sbatch file describes the resources and the command to run. Save this example as submit_job.sh and adapt it before submitting.
#!/bin/bash
#SBATCH --job-name=research_example
#SBATCH --partition=shared
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=01:00:00
#SBATCH --output=research_%j.out
#SBATCH --error=research_%j.err
# Replace the Conda installation, environment, and analysis paths.
source /path/to/miniconda3/etc/profile.d/conda.sh
conda activate research
cd /scratch/YOUR_NETID/project
python analysis.py
05
Monitor and inspect results
Check queue status and job accounting, then inspect the output and error logs. Use the HTC commands when your workload runs under HTCondor.
- Node
- A compute server containing processors, memory, and other hardware.
- CPU and core
- A CPU contains cores. Match Slurm CPU requests to how the application uses processes or threads.
# Slurm: submit, inspect the queue, and review accounting.
sbatch submit_job.sh
squeue -u "$USER"
sacct -j JOB_ID --format=JobID,State,Elapsed,MaxRSS
# Inspect the logs for your job.
cat research_JOB_ID.out
cat research_JOB_ID.err
# HTCondor: run on your assigned HTC access point.
condor_q
condor_status
06
Terminal command reference
Keep everyday navigation, inspection, archive, compilation, and environment commands close at hand.
# Navigate and inspect files.
pwd
cd project
ls -lh
cat notes.txt
readlink -f notes.txt
wc -l notes.txt
grep "pattern" notes.txt
echo "A B B A C C" > example.txt
python3 wc.py example.txt
# Archives: write an archive outside the directory being archived.
zip -r example.zip example/
unzip -l example.zip
unzip example.zip
tar -cvf data.tar data_folder/
tar -xvf data.tar -C destination/
tar -xzf data.tar.gz -C destination/
# Compile and run the original ecosystem-model example.
g++ xtem423e4.cpp -o temmodel
./temmodel tem4.para tem4.log
module avail
make
# Shell configuration and aliases.
nano ~/.bashrc
source ~/.bashrc
# On macOS with Zsh, use ~/.zshrc instead.
# Conda environment management.
conda env list
conda activate research
conda deactivate
# Interactive Slurm testing (adapt the resource request).
srun --partition=int --nodes=1 --ntasks=1 --cpus-per-task=4 --time=00:30:00 --pty bash
# In the vi editor: :wq saves and exits; :q! exits without saving.