Table of Contents

1. AFS Commands
2. Crontab Tasks
3. Job Control
4. File in Linux
5. Shell Basics
6. Check Infos
7. Shell Variables
8. Archive
9. Others

organized by Xianwei Zhang, Nov 2015!

1. AFS Commands ↑top

(1) Access control list

• giver user read rights in your directory:

  $fs sa ~/private/xianeizhang l  
  $fs sa simu01 xianeizhang rlidwka  
  $fs listacl ~/private/simu01  

• remove a user

  $fs setacl -dir ~/private/simu01 -acl xianeizhang none

(2) Ticket management

• obtain and cache Kerberos ticket: $kinit
• list cached Kerberos tickets: $klist
• destroy the ticket: $kdestroy
• renew the ticket for further 25 hrs: $kinit -R
  

(3) Token management

• get tokens for authentication to AFS (after having the ticket): $aklog
• display the issuer's tokens: $tokens
• destroy the token: unlog

(4) Create keytab file for pwdless login

• steps:

  $ktutil
      #-- after -k is digit '1', NOT letter 'l'
      ktutil:  addent -password -p xianeizhang@DEPT.CS.PITT.EDU -k 1 -e aes256-cts
      Password for xianeizhang@DEPT.CS.PITT.EDU:
      #-- write out the keytab to disk
      ktutil:  wkt xianeizhang.keytab
      ktutil:  q
  #-- no output if smoothly executed
  $kinit xianeizhang@DEPT.CS.PITT.EDU -k -t xianeizhang.keytab  
  $klist -k xianeizhang.keytab  
  KVNO Principal
  ---- -------------------------------------------
     1 xianeizhang@DEPT.CS.PITT.EDU
  $ktutil:
      ktutil:  read_kt xianeizhang.keytab
      ktutil:  list
      slot KVNO Principal
      ---- ---------------------
      1   1           xianeizhang@DEPT.CS.PITT.EDU
      #-- delete the key from keytab file <delent #slot>
      ktutil:  delent 1
      ktutil:  list
      ktutil:  q

(5) Script to renew tokens

• steps

  $kinit xianeizhang
  $aklog
  $nohup $HOME/bin/krb_afs_renewal.sh &

• script
  click here to download krb_afs_renewal.sh

(6) Restore AFS User files and Project Volumes

• AFS files - less than 24 hours

When you delete or change a file by mistake, you should first checked the backup directory, which contains a complete copy of all the directories, subdirectories, and files as they appeared the last time they were backed up (backup is usually done every midnight).

To view a Project Volume backup, you must mount it. Follow these steps:

    #-- login afs server
    #-- enter project volume directory
    $cd ~/private/bench/
    #-- get the volume
    $fs lq .
    Volume Name                    Quota       Used %Used   Partition
    cs.u.x.zhang.bench         200000000  141054086   71%         55%
    #-- mount the volume's backup, <proj-volume>.backup
    #--  e.g., volume is cs.u.x.zhang.bench, and backup should be cs.u.x.zhang.bench.backup
    $cd ~/private/simu10
    $fs mkm bench.1130 cs.u.x.zhang.bench.backup
    #-- now, restore the changed file
    $cp bench.1130/<filename> ~/private/bench
    #-- you don't need to unmount the volume, as files in .backup do not count against your disk quota usage
    #-- but unmounting is possible
    $fs rmmount bench.1130

• AFS files -- after 24 hours

If you failed to discover the mistake the same day, the only way to restore files is to contact technical staff to let them load backup tapes to restore the files.

More backup restore info can be found at CMU afs and Stanford afs

2. Crontab Tasks ↑top

• Crontab can be used to schedule a job that runs on certain interval cron_cmd.

  • Crontab format (0-59, 0-23, 1-31, 1-12, 0-6):
    MIN HOUR DOM MON DOW CMD
    
  • every 5 minuts:
    */5 * * * * /home/my.sh
    
  • every 5 hours:
    0 */5 * * * /home/my.sh

• Setup a task:

  $crontab -e
  0 * * * * $HOME/bin/krb_afs_renewal.sh  
  */1 * * * * $HOME/bin/krb_afs_renewal.sh  

• Cron job cannot be used to schedule a job in seconds interval.

  while true
  do
      /home/my.sh
      sleep 5
  done
  $nohup ./every-5-seconds.sh &

3. Job Control ↑top

• list all background jobs: $jobs (kill %n)
• bring one job to the front: $fg <job id>
• suspend one running process: $^Z

• refer to a job:

   #-- %1' is 'fg %1', bringing job 1 from background to foreground
   $%n
   #-- resumes job 1 in the background, equivalent to 'bg %1'
   $%1 &

4. File in Linux ↑top

• display dir type with '/': $ls -F
• change file permission (UGO=user-group-other)
  • for all uers (a), add (+) read-write (rw) permission: chmod a+rw file.dat
  • for other users (o), cancel (-) the read-execute (rx) permission: chmod o-rx file.dat
  • for group and other users (go), cancel (-) all permissions (rwx): chmod go-rwx file.dat
• file link
  • hard link: $ln file1 file2
  • symbol link: $ln -s file1 file2

5. Shell Basics ↑top

visit blog1 and blog2 to learn more ...

(1) Shell info

• list all available shells: $cat /etc/shells
• show current shell: echo $0 or echo $SHELL ####(2) Shell autologout
• show TMOUT: $echo $autologout
• cancel autologout: $unset autologout

• shell environment variables

  $[print]env (list all)
  $setenv SVN_EDITOR vim (bash: export SVN_EDITOR=vim)
  $setenv PATH "${PATH}:/usr/local:/usr/local/bin"

• get the last n characters of a string: ${str:${#str}-$n}

  • you can simply mix bash substrings - ${string:offset[:length]}

• loop

  for bankID in {0..7}
  for (( bankID=0; bankID <= 7; bankID++ ))
  if (( $n <= 5)) i.e., if [ $n -le 5 ]

• settings

  • cfg:

    CHIP="4Gb"
    ROWS_2Gb=32768
    ROWS_4Gb=65536
    ROWS_8Gb=131072

  • .sh:

    row_name="ROWS_$CHIP"
    ROWS=${!row_name}

• value extraction:

  for chunk_dir in $(find $scheme_dir -maxdepth 1 -mindepth 1 -type d -name "1024_chunks" -o -name "16384_chunks")

• info

  NOW=$(date)
  SERVERNAME=${hostname}
  CWD=$(pwd)

• get user input via keyboard

  $read -p "Enter something:" var1 var2 var3

6. Check Infos ↑top

(1) CPU info

• show processing units: $nproc
• CPU arch info: $lscpu
• more info: $less /proc/cpuinfo
• cache info: /sys/devices/system/cpu/cpu0/cache/index[0-3]/… or getconf LEVEL1_DCACHE_LINESIZE

(2) Process info

commands:

 $top
  $shift+A to have multi-window process view
  $type k and input PID to kill a process
  $type u and input user name to get process from the user alone
 detailed info
 $ps -Af

(3) Other info

• memory: $free -m
• disk and filesystem: $df -h
• 32 or 64: $uname -a/-m or $arch

(4) GLIBC

 $ldd --version
 $strings /usr/local/gcc-4.8.2/lib64/libstdc++.so.6 | grep LIBCXX
 $gcc --version
 $ldd <executable>
 
 /pin-2.13-62732-gcc.4.4.7-linux/intel64/runtime/cpplibs/libstdc++.so.6: version  `GLIBCXX_3.4.18' not found ( required by/obj-intel64/pinatrace.so), 
 $strings libstdc++.so.6 | grep GLIBCXX
 $cp /usr/local/gcc-4.8.2/lib64/libstdc++.so.6 .

(5) Count

$getconf PAGESIZE
cout word frequency in a file
$ tr ' ' '\n' < test.trace/test_anno  | grep memR | wc -l
$grep -o -c mem test.trace/test_anno

7. Shell Variables↑top

(1) Default value

• if var is not set, set it do default: var=xxx
• if var is not set, use default: var-xxx
• if var is empty, set to default: var:=xxx

 echo "default'=': dist= ${dist=1.0}"   #-- dist=1.0
 echo "after'=': dist= $dist"     #-- dist=1.0
 echo "---------"
 echo "default '-': weather= ${weather-sunny}" #-- weather=sunny
 echo "after '-': weather= $weather"    #-- weather= ('-' just use default, never set)
 echo "---------"
 weather="" #-- weather has been set, but just an empty value
 echo "default '-': weather= ${weather-sunny}" #-- weather=
 echo "default '=': weather= ${weather=sunny}" #-- weather=
 echo "default ':=': weather= ${weather:=sunny}" #-- weather= sunny
 echo "---------"
 unset weather #--weather now is unset
 echo "default '=': weather= ${weather=sunny}" #-- weather= sunny
 echo "default ':=': weather= ${weather:=sunny}" #-- weather= sunny

(2) Perform arithmetic operations

• integer operation

  #op=+-*/% ++ -- ** and comparison
  $(( var1 op var2 ))

• float operation

  $(echo "scale=2; $total_time/$reads" | bc -l)
  $(awk -v tot=$total_time -v read=$reads 'BEGIN{printf("%.2f", tot/read)}'

(3) Comparison

• difference between [ and [[

  • [[ handles empty strings and strings with whitespaces, so no need to quote variaes

    if [ -f "$file" ] i.e., if [[ -f $file ]]
    [ ! $file ]
    [[ $sentence == $checkme ]] i.e., [ "$sentence" == "$checkme" ]

  • [ is the same as 'test' builtin, and works like the 'test' binary

    if test "$name" = "xianwei" i.e., if [ "$name" = "xianwei" ]

  • for [, operators like && and || must be used in separate brackets

    if [ $foo -ge 3 -a $foo -lt 10 ]
    if [ $first -eq 0 ] && [ $second -eq 0 ], i.e., if [[ $first -eq 0 && $second -eq 0 ]]
    if [ $first -eq $second]

  • [[ is bash specific, == and != apply bash pattern matching rules

    if [[ $suffix == *ECC* ]]

  • [[ has a =~ regex match operator

    if [ "$ANSWER" = y -o "$ANSWER" = yes ]
    or
    if [[ $ANSWER =~ ^y(es)?$ ]]
    or
    if [[ $ANSWER = y* ]] 

  • others

    empty (unset, tab/space) string: if [ -z $test_str ]
    if [[ $test_str != foo]], 
    i.e., if [ ! "test_str" == "foo" ], 
    i.e., if [ "test_str" != foo ]

8. Archive↑top

(1) Creating an archive using tar

 $tar cvf archive_name dirname/ (note: no compression here)
    c -- create a new archive
    v -- verbosely list files which are processed
    f - following is the archive file name
 $tar cvfz archieve_name.tar.gz dirname/ (note: creating a tar gizpped archive)
    z -- filter the archive through gzip
    note: .tgz is same as .tar.gz
 $tar cvfj archive_name.tar.bz2 dirname/ (note: a bzip2 tar archive)
    j -- filter the archive through bzip2
    gzip vs bzip2: bzip2 takes more time to compress and decompress than gzip, but bzip2 is smaller in size
    note: .tbz and .tb2 is same as .tar.bz2

(2) Untar an arhive using tar

 $tar xvf archive_name.tar
    x -- extract files from archive
 $tar xvf archive_name.tar -C /target/dir 
    note: extract to a specified dir
 $tar xvfz archive_name.tar.gz
    note: extract a gizpped tar archive
 $tar xvfj archive_name.tar.bz2
    note: extract a bzipped tar archive

(3) Listing an archive using tar (without extracting)

 $tar tvf archive_name.tar
 $tar tvfz archive_name.tar.gz
 $tar tvfj archive_name.tar.bz2

(4) Extract a single file (from tar, tar.gz, tar.bz2)

 $tar xvf archive_file.tar /path/to/file
 $tar xvfz archive_file.tar.gz /path/to/file
 $tar xvfj archive_file.tar.bz2 /path/to/file
    extract a/multiple dirs from a tar archive
 $tar xvf archive_file.tar /path/to/dir1/ /path/to/dir2/
    extract group of files using regex
 $tar xvf archive_file.tar --wildcards '*.cpp'

(5) Adding a file or dir to an existing archive

 $tar rvf archive_name.tar newfile
    note: cannot add file or dir to a compressed archive

(6) Estimate the tar archive size before archiving

 $tar -cf - /dir/to/archive | wc -c
 $tar -czf - /dir/to/archive | wc -c
 $tar -cjf - /dir/to/archive | wc -c

9. Others↑top

(1) wget

• no-check: wget --no-check-certificate

(2) Mac double-click run

• .command
• Terminal > Preferences > Settings > Profiles > Shell: > When the shell exits: -> Close if the shell exited cleanly

(3) Latex

underline

default \underline{} doesn’t allow line breaks; use \ul{} command of soul package instead:

• underline

  \usepackage{soul}
  \ul{xxx}

Latex citations

• author-year
  \bibliographystyle{apalike} (linebreak issue)
  solution to linebreak:

  \usepackage[square]{natbib}
  \let\citeN\citet
  \let\cite\citep
  ------------ number
  \usepackage[square, numbers]{natbib}

  Without spacing line:

  \usepackage[square, authoryear]{natbib}
  
  \let\OLDthebibliography\thebibliography
  \renewcommand\thebibliography[1]{
  \OLDthebibliography{#1}
  \setlength{\parskip}{0pt} #space bt paras
  \setlength{\itemsep}{0pt plus 0.3ex} #space bt items
  
  \begin{singlespace}
  \bibliographystyle{plainnat}
  \bibliography{refresh,twr,scaling,jedec,dram,approx_security,stacked,page_os,hpca16}
  \end{singlespace}
  OR ----> bbl
  \addtolength\itemsep{-1em}
  }
  
  \bibliographystyle{plainnat}
  

black-circuled number

\usepackage{pifont}
\ding{202}
\ding{203}

figure/table numbered in section

\usepackage{chngcntr}
\counterwithin{figure}{chapter}
\counterwithin{table}{chapter}

move up title

\usepackage{titling}
\setlength{\droptitle}{-9em}   % This is your set screw

reduce space bt author and title

\begin{document}

\title{Research Statement\vspace{-2ex}}
\author{Xianwei Zhang%\\
   % \texttt{xianeizhang@cs.pitt.edu}
   }
 \date{}
\maketitle

reduce section gap

\usepackage{titlesec}

%\titlespacing{command}{left spacing}{before spacing}{after spacing}[right]
\titlespacing\section{0pt}{12pt plus 4pt minus 2pt}{0pt plus 2pt minus 2pt}
\titlespacing\subsection{0pt}{12pt plus 4pt minus 2pt}{0pt plus 2pt minus 2pt}
\titlespacing\subsubsection{0pt}{12pt plus 4pt minus 2pt}{0pt plus 2pt minus 2pt}