Personal tools
You are here: Home → Computing → Hardware → Power6 → File systems on the IBM Power6 575 system
 

File systems on the IBM Power6 575 system

Filesystem setup:

$HOME
Your home directory is in the GPFS file system /u.
AFS
AFS is only available on the Power6 login node "vip" and on "vip100" in order to access software that is distributed by AFS. If you don't get automatically an AFS token on "vip" and "vip100" during login, you can get an AFS token with the command /afs/ipp/@sys/bin/klog. There is no AFS on the Power6 compute nodes, so you have to avoid any dependencies on AFS in your job script.
If you get the message
    -------------------------------------------------------
    LoadL_starter: 2512-906 Cannot set user credentials.
    LoadL_starter: AFS token(s) were captured for the job step, however AFS was not found running on this node
    -------------------------------------------------------
in your job log files, you can ignore it.
GPFS
There are three global file systems of type GPFS, symmetrically accessible from all Power6 nodes:

/u for permanent user data. Your home directory is in /u.
(60 TB, RAID 5, no system backups)
If you need a private shell profile, you have to provide it by yourself.

/ptmp for temporary job I/O
(326 TB, RAID 5, no system backups)
Files in /ptmp that have been not accessed for more than 14 days are removed automatically. The period of 14 days may be reduced as the case arises.

As a current policy, no quotas are applied on /u and /ptmp. This gives users the freedom to manage their data according to their actual needs without administrative overhead. This liberal policy presumes a fair usage of the common file space, particularly /u. The /u file system is heavily used, so please do a regular housekeeping of your data and archive/remove files that are not used actually.

Archiving data from the GPFS file systems to tape can be done with ADSM/TSM.
Moreover, you can use the migrating file system /r (see below) to save your files that were created on the HPC machines (Power6, Blue Gene/P).

/r for migrated data (/r is a symbolic link to /ghi/r)
Each user has a subdirectory /r/<initial>/<userid> to store his/her data. Files should be packed to tar, cpio or zip files (with a size of 1 - 500 GByte) before archiving them in /r. When the file system gets full above a certain value, files will be transferred from disk to tape, beginning with the largest files which have been unused the longest time.

If you access a file which has been migrated to tape, the file will automatically be transferred back from tape to disk. This of course implies a delay. You can manually force the recall of a migrated file by using any command which opens the file. You can recall in advance all files needed by some job with a command like

file myfiles/*

You can see which files are resident on disk and which ones have been migrated to tape with the command ghi_ls (located in /usr/local/bin), optionally with the option -l. Here is a sample output:

vip001% ghi_ls -l
G -rw-r--r--   1  ifw       rzs             22 Nov 21 15:12 a1
H -rw-------   1  ifw       rzs   138958551040 Sep 18 22:22 abc.tar
H -rw-r--r--   1  ifw       rzs     1073741312 May 06 2009  core
G -rw-r--r--   1  ifw       rzs              0 Jun 20 2008  dsmerror.log
B -rw-r--r--   1  ifw       rzs     1079040000 Aug 03 2010  dummyz3
The first column states where the file resides: a 'G' means the file is resident on disk; a 'H' means the file has been transferred to the underlying HPSS archiving system, probably on tape; a 'B' means premigrated to tape (the file has already been copied to HPSS but is still present on disk and can be removed immediately if the system needs to free disk space).

 

Please note: If you want to "tar" files that are alreaddy located in /r, please carefully check the contents of the resulting TAR file if all migrated files were correctly retrieved and included into the TAR file.
Don't use "gzip" or "compress" on files that are already located in /r. It's not necessary because all files are automatically compressed by hardware as soon as they are written to tape.

For more information about /r see :
http://www.rzg.mpg.de/datastorage/tsm/adsm_PSI_qa.html

 

/tmp
Please, DON'T use the file system /tmp for your scratch data. Instead, use /ptmp which is accessible from all Power6 nodes, and set the environment variable TMPDIR to /ptmp/<userid> in your job scripts.
Document Actions