From Debian Clusters

Debian Clusters for Education and Research

This site is a series of step-by-step tutorials for anyone interested in building a Debian cluster. It is geared particularly towards those interested in a Debian cluster for education or research purposes. This site is being written alongside the construction of a real Debian cluster at my institute, starting from the ground (basic networking, DHCP, DNS, authentication) up through installation a queue/scheduler and scientific software.

However, these tutorials and guides take some assumptions about users for granted, including basic Linux and Debian knowledge (see the Basic Linux Skills page for clarification and pointers to resources). This is not meant as an introduction to Debian or on how to cluster in general - thankfully, there are plenty of helpful tutorials and guides already in existence on those topics. A basic understanding of Debian and cluster technology will aid in following the tutorials.

Comments, criticisms, and suggestions are welcome and appreciated. Please send them to the project lead, Kristina (Kay) Wanous, at kwanous <at> debianclusters <dot> org.

About the Debian Clusters Project

Project Overview

• Network Topology

Setting up Debian

• Base Installation of Debian
• Compiling the Kernel "The Debian Way"
  • Setting Up the Kernel Build Environment
  • Adding Performance Counters to the Kernel
  • Adding support for PVFS2 to the Kernel

Setting up Firewall Services

• NAT with IPTables
• Fail2Ban: Preventing Brute Force SSH

Setting up Services

• Mounted File System: NFS
• File System: RAID Arrays
• Handing out IPs: DHCP
• Name Service: DNS and BIND
• User Authentication: LDAP
  • Using LDAP

Creating Head Node and Worker Node Images

• Bash Profile Modifications
• Password-less SSH for Users
• Installing Compilers, including Intel Compilers
• Udev: Renaming Network Interfaces
• Setting a Dynamic Hostname by DNS

Cloning Worker Nodes

• First, make sure the above services are set up for the head node and worker.

Cluster Time-saving Tricks

Setting up Scientific Software on Debian

• Source Installation Paradigm
• MPICH: Parallel Programming - an MPI implementation is necessary for many parallel applications
• Gromacs: Molecular Dynamics
• NAMD: Molecular Dynamics
• VMD: Visual Molecular Dynamics
• mpiBLAST: Nucleotide/Protein Searching

Using a Scheduler and Queue

• Resource Manager: Torque
• Scheduler: Maui
• Troubleshooting Torque and Maui
  • Torque and Maui Sanity Check: Submitting a Job
  • Torque and Maui: Submitting an MPI Job

Using Torque and Maui:

• Torque Qsub Scripts
• Torque Queue Configuration

Cluster Monitoring on the Web

• Setting up a Web Server
• Securing the Web Server
• Monitoring Services: Nagios
• Monitoring Load: Ganglia
• Monitoring the Network: Cacti

Notes

• Links
• Mail Server
  • Setting up Mutt
• Setting up a Web Server

Networking Tools