Unix
This section has the stuffs which are common on all the unix systems.
ldd - prints the shared libraries required by each program or shared library specified on the command line.
To unzip to a different folder an gzip archive
# gzip -d -c /cdrom/netbacku.gz | (cd /stage; tar xf - )
Awk
AWK Programming
Variables
Operators
Formatting
Print the Next Line
Common Examples
awk is filter. awk performs the specified action on lines that match the pattern. Like 'sed', a regular expression pattern must be enclosed in a forward slashes. If no pattern is listed, the action is performed on every input line.
Relational Operators
printf fuction: printf statement can be used to specify the width and alignment of output fields. Unlike print function, it does not give any new line charater at the end of the line. The contents of the field will ne right justif ied by default. You musr specify "-" to get left justification. Thus "%-20s" out puts a string left justif ied in a fiels 20 character wide.
To find a patern "Dec" in 6th field
Scalar variable: It is either a number or string. It has magnitude but no direction. It starts with $.
Binary Assignment Operator:
Nearly all binary operators that compute a value have a corresponding binary assignment form with an appended equals sign. For example, the following two lines are equivalent:
$fred = $fred + 5; # without the binary assignment operator
$fred += 5; # with the binary assignment operator
These are also equivalent:
$barney = $barney * 3;
$barney *= 3;
To get Standard input and remember as variable, <STDIN> construct is used.
$name = <STDIN>;
Array Variable:
An array variable starts with @. An array is a variable that holds a list. Each element of the array is a separate scalar variable. A list literal consists of comma separated values enclosed in parentheses.
{ first_statement; second_statement; …… last statement; }
open(LOG, ">>logfile") || die "cannot append: $!"; $filename = <STDIN>; chomp $filename; # toss pesky newline if (-r $filename && -w $filename) { # file exists, and I can read and write it ... }
use Time::localtime; $tm = localtime; ($DAY, $MONTH, $YEAR) = ($tm->mday, $tm->mon, $tm->year);
use Time::localtime;
$tm = localtime($time);
printf("Dateline: %02d:%02d:%02d-%04d/%02d/%02d\n",
$tm->hour, $tm->min, $tm->sec, $tm->year+1900, $tm->mon+1, $tm->mday); # perl -le 'print scalar localtime 12545 * 24 * 60 * 60 ' Thu May 6 17:00:00 2004 cat time.pl #!/usr/bin/perl $a = time ; print "$a \n"; print scalar localtime (time) ; # ./time.pl 1087418422 Wed Jun 16 13:40:22 2004
$ perl -w my_program
To set the
PS1=`hostname -s`:'$PWD>' or PS1="`hostname -s` [\$PWD]# "
press ~. or break keys - ~#
Text creation and Manipulation
Changing and deleting Text
Scrolling the Screen
Searching a pattern or text
Replacing words Globally
Managing files
bc
expr
dig
ftp
finger
host
nslookup
rcp
rlogin
rsh
telnet
traceroute
scp
ssh
sftp
Variables
Operators
Formatting
Print the Next Line
Common Examples
AWK Programming
awk is filter. awk performs the specified action on lines that match the pattern. Like 'sed', a regular expression pattern must be enclosed in a forward slashes. If no pattern is listed, the action is performed on every input line.
awk 'pattern {action}' [files]
$ awk '/^S/' phone.list
$ awk '/^S/ {print $1}' phone.list
Variables
$0 represents the whole current input record $1 represents the first field of the current input record $NF is number of fields for the current input record NR is number of Lines read
Shell Variables
Shell Variables can be passed to awk using -v option awk -v var=value '{print $1, var}'
Operators
Relational Operators
< and > Less than and Greater than <= and >= Less than or equl to AND Greater than or equl to == and != Equl to AND Not qual to ~ and !~ Matches AND Doesn't match regular expressions
Boolean Operator:
|| Logical OR && Logical AND ! Logical NOT
Assignment Operators
++ Add 1 to variable. -- Subtract 1 from variable. += Assign result of addition. -= Assign result of subtraction. *= Assign result of multiplication. /= Assign result of division. %= Assign result of modulo. ^= Assign result of exponentiation. **= Assign result of exponentiation.[6]
Arithmetic Operator
+ Addition - Subtraction * Multiplication / Division % Modulo Division (Remainder) = assignment
Formatting
printf fuction: printf statement can be used to specify the width and alignment of output fields. Unlike print function, it does not give any new line charater at the end of the line. The contents of the field will ne right justif ied by default. You musr specify "-" to get left justification. Thus "%-20s" out puts a string left justif ied in a fiels 20 character wide.
Format Specifiers used in printf
c ASCII Character d Decimal Integer e Floating point format s String % Literal %
Example of formatting:
Input file is comma separated.
Input file is comma separated.
$ cat file.txt hiddenhausen,99.60,y herbstein,99.021,n bangalore,98.82,yLets try to format the above file:
$ awk '{ printf "%-15s%-8s%s\n",$1,$2,$3}' FS=\, file.txt
hiddenhausen 99.60 y
herbstein 99.021 n
bangalore 98.82 y
So its printing first field ($1) as a string of 15 characters
that are left-justified, 2nd field ($2) as 8 characters left-justified
and then third field ($3)
Another Example
# cat abc.txt
id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type:FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count:copy_count:fast_write_state
0:D53R-R1A-0915:3:prod1:online:2:D53R-R1A:256.00GB:striped:::0:HERCOPY125:60050768018F028260000000000006D9:0:1:not_empty
1:D53I-R1A-0551:2:prod0:online:1:D53I-R1A:256.00GB:striped:::1:HERCOPY126:60050768018F028260000000000006DA:0:1:not_empty
2:D83B-R1A-0717:3:prod1:online:5:D83B-R1A:256.00GB:striped:76:EDBACK077:::60050768018F028260000000000006DB:1:1:not_empty
# cat abc.txt | awk -F: '{printf "%-5s%-16s%-8s%-10s%-10s %s\n", $1,$2,$4,$7,$8,$14}'| head
id name IO_group_namemdisk_grp_namecapacity vdisk_UID
0 D53R-R1A-0915 prod1 D53R-R1A 256.00GB 60050768018F028260000000000006D9
1 D53I-R1A-0551 prod0 D53I-R1A 256.00GB 60050768018F028260000000000006DA
2 D83B-R1A-0717 prod1 D83B-R1A 256.00GB 60050768018F028260000000000006DB
3 D83B-R1A-0718 prod0 D83B-R1A 256.00GB 60050768018F028260000000000006DC
Printing the next line of a search Pattern
To search and print the next line use the following awk commands awk '/pattern/{getline;print}'
# ioscan –funC disks | awk '/ROM/{getline;print}'
disk 4 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP CD-ROM 305
Print the current line and nextline
# ioscan -funC disk | awk '$1=="disk" { getline L2 ; print $0,L2 }'
Same as above, but another way
# ioscan -funNC disk | awk '{if ($1=="disk") {L1=$0;getline; sub (" *","");print L1,$0}}'
Common Examples
To find a patern "Dec" in 6th field
ls -l | awk '$6 == "Dec"'To display the first and second fields of each line
awk '{print $1, $2}' <filename>
To display all the lines containing string1 in file called file1
awk '/string1/' file1To display first filed of all the lines containing sting "string1"
awk '/string1/ {print $1}'
To display the first and 3rd fields of /etc/passwd file
awk -F: '{print $1, $3}'
To display the first and last field of /etc/passwd file
awk -F: '{print $1, $NF}'
To display the previous field to the last filed of any file
awk '{print $(NF-1)}' file_name
To print the second field in the third line
awk 'NR==3 {print $2}'
To sum all the fields using awk command:
ls -l | awk '{sum += $5} {print sum}'
ls -l | awk '{sum += $5} END {print sum}'
To add few numbers and find out the average
echo "test 10 20 30" | awk '{total = $2 + $3 + $4} {ave = total / 3} {print ave}'
To add the value of Unix variable $var1 to the beginning of each line in file1
cat file1 | awk -v name=$var1 '{print name" " $0}'
Running Unix commands directly using awk
# ps -ef | grep syncd | grep -v grep | awk '{system("kill -9 " $2)}'
The above command find the process ID of sysncd and kills that process
To replace DS with DSsmtp.comcast.net if the line has only single word 'DS'
awk '{if ($0~/^DS$/) {print "DSsmtp.comcast.net"} \
else {print}}' /etc/mail/sendmail.cf
Another AWK Complex example
awk '{if ($0!~/^\*/ && $0~/:$/ && $1~/^default:$/) {print $0"\n\tregistry = NIS"} \
else if ($0!~/^\*/ && $0~/:$/ && $1!~/^default:$/) {print $0"\n\tregistry = files"} \
else {print}}' /etc/security/user > /etc/security/user.new
To add # mark to the beginning of each line except for the lines starting with #, ftp, shell, and telnet:
awk '{if ($1~/^#/ || $1~/^ftp/ \
|| $1~/^shell/ || $1~/^telnet/) \
{print} else {print "#"$0}}' /etc/inetd.conf
Some examples with sample output
$ awk -F":" '{ print "username: " $1 "\t\tuid:" $3" }' /etc/passwd
username: halt uid:7
username: operator uid:11
username: root uid:0
http://www.ibm.com/developerworks/linux/library/l-awk1.html
http://www.ibm.com/developerworks/linux/library/l-awk2.html
http://www.ibm.com/developerworks/linux/library/l-awk2.html
Backup and Restore
Differential Backup:
A differential backup will backup all files that have changed since the last full backup. In other words, if a full backup was done on Monday, Tuesday's differential will backup all changed files since Monday's full. Wednesday's differential will backup all changed files since Monday's full including the files that have changed on Tuesday.
A differential backup will backup all files that have changed since the last full backup. In other words, if a full backup was done on Monday, Tuesday's differential will backup all changed files since Monday's full. Wednesday's differential will backup all changed files since Monday's full including the files that have changed on Tuesday.
The big advantage to this method comes when performing
a complete restore since only the full and latest differential backups
need to be restored. The downside is that the size of the differential
backup will grow throughout the week and become progressively larger
until the next full. This can adversely affect backup windows.
Incremental Backup
An incremental backup will backup all files that have changed since the last backup, regardless whether it was a full or incremental backup. In other words, if a full backup was done on Monday, Tuesday's incremental will backup all changed files since Monday's backup. However, Wednesday's incremental will only backup files that have changed since Tuesday's incremental backup.
An incremental backup will backup all files that have changed since the last backup, regardless whether it was a full or incremental backup. In other words, if a full backup was done on Monday, Tuesday's incremental will backup all changed files since Monday's backup. However, Wednesday's incremental will only backup files that have changed since Tuesday's incremental backup.
The main advantage to this method is that a lot fewer
files are backed up daily between full backups allowing for shorter
backup windows. The disadvantage is that when performing a complete
restore, the latest full and all subsequent incremental backups must be
restored, which can take significantly longer.
The simplified technical explanation could be
summarized by stating that both full and incremental backups reset the
archive bit on a file (indicating it has been backed up) and a
differential backup does not.
Backup and restore commands
cpio
To copy the current directory and all subdirectories onto tape
# find . -print | cpio -ov >/dev/rmt0
To copy a directory and all of its subdirectories, enter:
# find . -print | cpio -pdl /home/jim/newdir
To list the files that have been saved onto a tape with the cpio command, enter:
# cpio -itv </dev/rmt0
To copy the files previously saved with the cpio command from a diskette, enter:
# cpio -idmv </dev/rfd0
Using tar
tar
tar options:
-A append tar files to an archive -c create a new archive -d find differences between archive and file system -r append files to the end of an archive -t list the contents of an archive -u only append files that are newer than copy in archive -x extract files from an archive -l stay in local file system when creating an archive -M create/list/extract multi-volume archive -v verbose mode -j filter the archive through bzip2 -z filter the archive through gzip -Z filter the archive through compress -f <file_name> Name of the archive/Device
gnu tar options
--exclude=<pattern> # to exlcude directory or files matching the pattern.
# multiple --exclude option can be defined
--exclude-from=File # to exclude the list of pattern listed int he File. Use this option to read a list of
# patterns, one per line, from FILE; `tar' will ignore files matching those patterns.
To move all files in /tmp and it's sub directories to /tmp2
# cd /tmp # tar cf - ./* |(cd /tmp2; tar xvf -)
To copy the files to a remote system using tar and ssh
tar -cf - <directory To backup> | ssh stbydb01 "cd <directory>; tar xvf -"
rsync
rsync examples:
Use below one to sync new and modified files via ssh custom port, but it will not clean the deleted files
#rsync -ave 'ssh -p 1661' -logptr <sorucedir> <remoteHost>:/<remote_dir>
Use below one to sync new and modified files,
but it will not clean the deleted files and exclude the directories in
the exclude file, It sync the data from myserver to currect directory
#rsync -ave 'ssh -p 1661' -logptr --exclude-from=/root/bin/exclude.list myserver:/mydata
Use below one to delete the files in destination also what ever deleted in source
#rsync -ave 'ssh' -logptr --delete --exclude-from=/root/bin/exclude.list myserver:/mydata/ /data/mydata
To compress the data and send it for saving bandwidth
# rsync -a -v -z -e "ssh -o Compression=no" <sorucedir> <remoteHost>:/<remote_dir>
You might want to customise how the remote shell behaves, for example this might be a better option for the backups:
# rsync -a -v -e "ssh -c arcfour -o Compression=no -x" user@3aims.com:/home/user web10
Here is what the options to -e mean:
ssh - use ssh instead of the default of rsh
-c arcfour - uses the weakest but fastest encryption that ssh supports
-o Compression=no - Turns off ssh's compression - rsync has its own if you want it which we'll discuss in a minute
-x - turns off ssh's X tunneling feature (if you actually have it on by default)
If bandwidth is a problem you might want to use
the -z option to have rsync compress data it sends across the network.
If you are using rsync compression it makes sense not to use ssh's
compression in the way demonstrated above. Here's the command using
rsync compression:
# rsync -a -v -z -e "ssh -c arcfour -o Compression=no -x" user@3aims.com:/home/user web10
If you want to test these how effective each of
these commands are you will need to delete the web10 directory rsync
creates otherwise rsync will only copy files which have changed. Whilst
that's normally what you want, it isn't too useful for tests.
Finally, since rsync is very efficient it can saturate
a network connection. If you still want to be able to use your network
connection whilst rsync is running you can use the --bwlimit option
which allows you to specify a maximum transfer rate in kilobytes per
second. Due to the nature of rsync transfers, blocks of data are sent,
then if rsync determines the transfer was too fast, it will wait before
sending the next data block. The result is an average transfer rate
equaling the specified limit. For example to limit rsync to using
100KB/sec you could do this:
# rsync -a -v -z --bwlimit=100 -e "ssh -c arcfour -o Compression=no -x" user@3aims.com:/home/user web10
You might also want to use --progress so that
rsync prints out a %completion and transfer speed while transferring
large files (but this isn't worth adding if you are running from a cron
job). If you are performing a backup which you think you might want to
restore at some point in the future you should use --numeric-ids. This
tells rsync to not attempt to translate UID <> userid or GID
<> groupid which is very important for avoiding permission
problems when restoring. You might also want the -H option which forces
rsync to maintain hardlinks on the server and the -x option which causes
rsync to only copy files from one filesystem and not any other files
which might be mounted as part of that directory structure. You can also
use the --delete option which deletes files from the backup if they
don't exist on the server. If you use --delete the files are deleted
before the copying starts.
Putting this all together the command I use to backup one of my servers looks something like this:
# rsync -aHxvz --delete --progress --numeric-ids -e "ssh -c arcfour -o Compression=no -x" root@example.com:/ pauli/
Tape Drives and mt commands
Tape Processing in the Solaris OS
This Tech Tip offers advice on working with tapes and the Solaris OS, including tips for backing up data to tape.
A single file tape consists of one or more blocks of
data followed by two EOF marks. The EOF marks are a tape drive hardware
function.
A multi-file tape consists of each file separated by
an EOF mark. The last file on a tape is marked by a sequence of two EOF
marks. The two EOF marks (one after the other) show the logical end of
tape, which is often not the physical end of tape.
Tape Terminology
The physical blocks on tape are the tape "records".
These physical records are separated from each other by the inter-record
gap, which is of fixed size, depending on the tape technology and also
the tape manufacturer. To reduce the proportion of the tape that is made
up of inter-record gaps, the block should be as large as possible,
especially on high-density drives. This is accomplished by having the
blocking factor as large as possible (this is the number of disk records
per tape block). If you have small tape blocks, most of the tape is
made up of inter-record gaps.
Often when you get a tape from a source other than the Solaris OS, the following could create problems:
- A header on the tape: For example, VERITAS NetBackup tapes have a tape header. Mainframe tapes often have a VOL1 header. The header is one tape file, so you need to skip over the first file by using the dd command.
- The character set could be different: Often tapes from the mainframe are in EBCDIC. These can be converted by the dd utility with the conv option.
- Big-endian/little-endian differences: On a 2-byte word, the left-most byte could be the high-order or low-order byte. This can be turned around using the swab option on dd.
Tape drives use the st (SCSI) tape driver. A lot of information is available from the manual pages of st.
When a new tape drive or any device is added, the
Solaris OS will "see" the new device after a reboot. You can dynamically
update the /devices tree by executing a devfsadm command.
The tape drives can be further configured by editing
the st.conf file (/kernel/drv/st.conf). Normally this would be under the
guidance of the tape drive vendor.
The most common tape device is the device with a name
like /dev/rmt/0, which is accessing tape drive 0 using the raw
interface. You can also have /dev/rmt/0c for tape compression (if the
drive supports this feature). Also possible is /dev/rmt/0n, which will
not rewind the tape after the current operation. (Tape drives will
automatically rewind tapes at the end of the current operation unless
the n option is used.) 0cn for non-rewindable compressed drive
Basic Tape Commands
To go through every tape file on the tape use:
# mt -f /dev/rmt/0n fsf 1
This moves the tape forward over one tape mark.
Don't forget the n option -- Otherwise as soon as the tape is forwarded
it will immediately rewind. This is useful if you want to skip over the
VOL1 header.
To go backwards past an EOF marker use:
# mt -f /dev/rmt/0n bsf 1
To rewind the tape use:
# mt -f /dev/rmt/0n rew
To offline a tape use:
# mt -f /dev/rmt/0n offlineThis typically opens the tape drive door and ejects the tape.
- Note that if you terminate the current tape operation it may take some time for the command prompt to come back as the tape is rewinding.
- There is no direct access to tape stacker (tape changer) functions, for example, mounting a tape on a tape drive from a tape library.
To find out the status of the tape drive use:
# mt -f /dev/rmt/0n status
You typically will get no reply while the tape drive is rewinding or the tape drive is busy.
If you need to read directly from the drive you can use the dd command:
# dd if=/dev/rmt/0 bs=<block size in bytes>
If you need to convert the character set, use this:
# dd if=/dev/rmt/0 bs=<block size in bytes> conv=ascii
This converts the data on the tape from EBCDIC to ASCII.
If you need to swap the bytes around (low endian to high endian), type:
#dd if=/dev/rmt/0 bs=<block size in bytes> conv=swab
Backing Up to Tape
Files can be backed up to tape using proprietary
solutions such as VERITAS NetBackup or IBM Tivoli software. Files can
also be backed up with the tar or cpio utility.
# cd /source # tar cvf b /dev/rmt/0cn 50 .
To make this run faster, you can get rid of the v (verbose) option.
Always make your tar backups without using absolute
file names. An absolute tar backup (such as tar cvfb /dev/rmt/0cn 50
/source) would mean that you could only restore this archive to the
/source directory.
To read back the tar file, use this:
# cd /restore # tar xfb /dev/rmt/0cn 50
To back up data with cpio, use this:
# cd /source # cpio -ocvB 50 . > /dev/rmt/0cn
Do the following to restore data using cpio:
# cd /restore # cpio -icvB 50 < /dev/rmt/0cn
General
xargs command
To run commands on standard output of some command instead of using for loop.# ls | xargs -t -n1 ls -l
Use -i or -I {} for inserting filenames in the
middle of command line. For example, To rename all the files in a
directory with an .old extension
# ls | xargs -t -i mv {} {}.old
sudo
The default configuration file for sudo is /etc/sudoers. Do not directly edit the /etc/sudoers file. Always use visudo command. If visudo command opens some other editor other than vi, we can correct it setting EDITOR variable. (eg. export EDITOR=vi)
There are four kinds of aliases in sudoers file
- User_Alias
- Runas_Alias
- Host_Alias
- Cmnd_Alias
Each alias definition is of the form
Alias_Type NAME = item1, item2, ... user or group hosts or or (Useralias) (hostalias)= (run-as alias) (command alias)
Converting time to epoch and viceversa
To get the current time in epoch seconds (On HP-UX and Linux)# date +%s 1214425649 # perl -le "print (time)" 1214426325
Convert from human readable date to epoch (On Linux)
# date +%s -d"Jan 1, 1980 00:00:01" 315561601 # perl -e perl -e "use Time::Local; print timelocal($sec, $min, $hours, $mday, $mon, $year)" # perl -e "use Time::Local; print timelocal(20, 56, 16, 25, 05, 2008)" 1214438180
To convert from epoch to Human readable format (On Linux)
# date -d @1190000000
Sun Sep 16 20:33:20 PDT 2007
$ date -r 1302972510 (In Mac OSX or Free BSD)
Sat Apr 16 10:48:30 MDT 2011
or
perl -e "print scalar(localtime(1190000000))"
Sun Sep 16 20:33:20 2007
To print the time 30 minutes ago in month, day, hour, minutes, year format
# perl -e 'use POSIX;print strftime "dM%y\n",localtime time - 1800;' 0103160313
To find out yesterday's date
$ date Sat Jul 21 01:56:30 PDT 2012 $ date --date="1 day ago" Fri Jul 20 01:56:34 PDT 2012
Using Perl
Use the following: perl -e 'use POSIX;print strftime "%Y%m%d\n",localtime time-86400;' yday=$(perl -e 'use POSIX;print strftime "%Y%m%d\n",localtime time-86400;'); echo ydat To find out the previous day of a given epoch second preday=$(perl -e 'use POSIX;print strftime "\%Y\%m%d\n",local time 1342860840-86400;'); echo preday
Links
General Unix Related Help links
To Learn Subnet
http://www.learntosubnet.com/
Linux Home Networking
http://www.linuxhomenetworking.com/
Veritas
http://www.cuddletech.com/veritas/index.shtml
Usefull shell Scripts
http://www.shelldorado.com/scripts/names.html
Perl
Integer Literals: They are straight
forward as in 12, 15, 500, -300 etc. Don’t start the number with a 0,
because Perl supports Octal numbers which starts with 0 (0377) and hex
numbers start with 0x (-0xff).
Double quated string representations:
\n newline
\r return
\t tab
\b backspace
\a bell
\e escape
\cC any control character (In this example, it is CTRL + C)
\\ backslash
\" Double quote
\l lowercase next letter
\u uppercase next letter
\U Uppercase all following letters until \E
\L Lowercase all following letters until \E
\E Terminate \U, \E
\n newline
\r return
\t tab
\b backspace
\a bell
\e escape
\cC any control character (In this example, it is CTRL + C)
\\ backslash
\" Double quote
\l lowercase next letter
\u uppercase next letter
\U Uppercase all following letters until \E
\L Lowercase all following letters until \E
\E Terminate \U, \E
Scalar Operator: An operator produces a new value (the result) from one or more other values (the operands). For example, + is an operator.
+, - , * , / (addition, substraction, multiplication and division operators)
/** is exponention operator. (2**3 is 2 to the third power, or 8)
% is modular operator ( 10 % 3 is the remainder when 10 is divided by 3 which is 1)
++ or - - (Auto incement or auto decrement)
x is string repetition operator. “fred” x 3 gives fredfredfred and 4 x 4 gives 4444 as output.
&& (logical and), || (Logical OR)
‘.’ Concatenate operator. “X” . (4 * 5) # same as “X” . 20 or X20.
= += -= *=, etc… Assignment and binary assignment ( $a=$a+5 and $a +=5 are same)
/** is exponention operator. (2**3 is 2 to the third power, or 8)
% is modular operator ( 10 % 3 is the remainder when 10 is divided by 3 which is 1)
++ or - - (Auto incement or auto decrement)
x is string repetition operator. “fred” x 3 gives fredfredfred and 4 x 4 gives 4444 as output.
&& (logical and), || (Logical OR)
‘.’ Concatenate operator. “X” . (4 * 5) # same as “X” . 20 or X20.
= += -= *=, etc… Assignment and binary assignment ( $a=$a+5 and $a +=5 are same)
Number and string comparision:
The eg (equal), ne, lt, gt, le, ge operators compares 2 strings.
($name eq "jeeva")
The eg (equal), ne, lt, gt, le, ge operators compares 2 strings.
($name eq "jeeva")
== (equal), != (not equal), <, >, <=, >= operators compares 2 numeric values.
($number == 10)
($number == 10)
Binary Assignment Operator:
Nearly all binary operators that compute a value have a corresponding binary assignment form with an appended equals sign. For example, the following two lines are equivalent:
$fred = $fred + 5; # without the binary assignment operator
$fred += 5; # with the binary assignment operator
These are also equivalent:
$barney = $barney * 3;
$barney *= 3;
$str = $str . " "; # append a space to $str
$str .= " "; # same thing with assignment operator
$str .= " "; # same thing with assignment operator
Autoincrement and Autodecrement:
$a =+ 1; # with assignment operator
++$a ; # with prefix autoincrement
$d = 17;
$e = ++$d; # $e and $d are both 18 now.
$a =+ 1; # with assignment operator
++$a ; # with prefix autoincrement
$d = 17;
$e = ++$d; # $e and $d are both 18 now.
Here, the ++ operator is being used as a prefix
operator; that is, the operator appears to the left of its operand. The
autoincrement may also be used in a suffix form (to the right of its
operand). In this case, the result of the expression is the old value of
the variable before the variable is incremented. For example:
$c = 17;
$d = $c++; # $d is 17, but $c is now 18.
$c = 17;
$d = $c++; # $d is 17, but $c is now 18.
To get Standard input and remember as variable, <STDIN> construct is used.
$name = <STDIN>;
To remove the trailing new line (\n) character, use "chomp" funtion.
chomp ($name); or chomp ($a = <STDIN>);
To remove the last character, use chop funtion.
chop ($name);
chomp ($name); or chomp ($a = <STDIN>);
To remove the last character, use chop funtion.
chop ($name);
Array Variable:
An array variable starts with @. An array is a variable that holds a list. Each element of the array is a separate scalar variable. A list literal consists of comma separated values enclosed in parentheses.
@words = ("Jeeva","Nandam","Kailasam","Nadar"); or qw (Jeeva Nandam Kailasam Nadar)
List constructor operator: Indicated
by two scalar values separated by two consecutive periodes. This
operator creates a list of valuess starting at the left scalar value up
through the right scalar value, incrementing by one esch time. For
example,
(1 .. 5) # Same as (1, 2, 3, 4, 5)
(1.2 .. 3.2) # Same as (1.2, 2.2, 3.2)
($a .. $b) # Range determined by current values of $a and $b
(1.2 .. 3.2) # Same as (1.2, 2.2, 3.2)
($a .. $b) # Range determined by current values of $a and $b
If a list literal contains only variable references
(not expressions), the list literal can also be treated as a variable.
In other words, such a list can be used on the left side of an
assignment. Each scalar variable in the list literal takes on the
corresponding value fromn the list on the right side of the assignment.
For example:
($a,$b,$c) = (1,2,3); # give 1 to $a, 2 to $b, 3 to $c
($a,$b) = ($b,$a); # swap $a and $b
($d,@fred) = ($a,$b,$c); #give $a to $d and ($b,$c) to @fred
($a,$b) = ($b,$a); # swap $a and $b
($d,@fred) = ($a,$b,$c); #give $a to $d and ($b,$c) to @fred
Array element access: Array element are numbered using
sequential integers, beginning at zero and incresing by one for each
element.
@fred = (7,8,9);
$b = $fred[0] ; # give 7 to $b
$fred[2]++; # incremet the third element of @fred
@fred = (7,8,9);
$b = $fred[0] ; # give 7 to $b
$fred[2]++; # incremet the third element of @fred
Accessing a list of elements from the same array is called a slice. This slice uses an @ prefix rather than $.
@fred[0,1]; # same as ($fred[0], $fred[1])
@fred[1,2] = (9,10); # change the last 2 values to 9 and 10.
@fred[0,1]; # same as ($fred[0], $fred[1])
@fred[1,2] = (9,10); # change the last 2 values to 9 and 10.
Slices also work on literal lists, or any funtion that returns a list value.
@who =(qw(fred baney betty wilma)) [2,3]; # same as @who = wq(betty wilma);
@who =(qw(fred baney betty wilma)) [2,3]; # same as @who = wq(betty wilma);
Assigning a value to beyond the end of the current
array automatically extends the array (giving value of undef to all
intermediate values). For example:
@fred = (1,2,3);
$fred[3] = “hi”; # @fred is now (1,2,3,”hi”)
$fred[6] = “ho”; # @fred is now (1,2,3,”hi”,undef,undef,”ho”)
print $#fred; # prints the index value of the last element. Here it is 6.
@fred = (1,2,3);
$fred[3] = “hi”; # @fred is now (1,2,3,”hi”)
$fred[6] = “ho”; # @fred is now (1,2,3,”hi”,undef,undef,”ho”)
print $#fred; # prints the index value of the last element. Here it is 6.
The push and pop funtions: Push and Pop funtions do things to the right side or a list.
push (@mylist,$newvalue); # Like @mylist = (@mylist, $newvalue);
$oldvalue = pop (@mylist); # removes the last element of @mylist.
$oldvalue = pop (@mylist); # removes the last element of @mylist.
The shift and unshift function: Unlike push and pop, shift and unshift functions performs actions on the left side of the list.
@fred = (5,6,7);
unshift (@fred,2,3,4); #fred is now (2,3,4,5,6,7)
$x = shift (@fred); # $x is now 2 and @fred is now (3,4,5,6,7)
unshift (@fred,2,3,4); #fred is now (2,3,4,5,6,7)
$x = shift (@fred); # $x is now 2 and @fred is now (3,4,5,6,7)
The reverse function: The reverse operator takes a
list of values (which may come from an array) and returns the list in
the opposite order.
@fred = 6..10;
@barney = reverse(@fred); # gets 10, 9, 8, 7, 6
@wilma = reverse 6..10; # gets the same thing, without the other array
@fred = 6..10;
@barney = reverse(@fred); # gets 10, 9, 8, 7, 6
@wilma = reverse 6..10; # gets the same thing, without the other array
The sort Function: The sort operator takes a list of
values (which may come from an array) and sorts them in the internal
character ordering
@rocks = qw/ bedrock slate rubble granite /;
@sorted = sort(@rocks); # gets bedrock, granite, rubble, slate
@rocks = qw/ bedrock slate rubble granite /;
@sorted = sort(@rocks); # gets bedrock, granite, rubble, slate
H-ash Variable:
A hash variable starts with %. Each element of hash is a separate scalar
vaiable. But it is referenced by a key value.
A hash variable starts with %. Each element of hash is a separate scalar
vaiable. But it is referenced by a key value.
%words = qw ( fred camel barney funny betty aplaca );
In the above example, camel is referenced by a key fred and funny by the
key barney. $words{fred} gives camel and $words{barney} gives funny.
key barney. $words{fred} gives camel and $words{barney} gives funny.
Regular Expressions:
Regular expressions are deliminated by slashes.
To look for any string that begin with 'randal'
if ($name =~ /^randal/)
if ($name =~ /^randal/)
To ignore the case, append 'i' after the closing slash
if ($name =~ /^randal/i)
if ($name =~ /^randal/i)
Word boundary special marker \b is used to mention there is no letter, digits or any special charabtes after this marker.
if ($name =~ /^Randal\b/i/)
In the above example, the condition is true if the
$name starts with Randal and there is no letters or digits after
the letter 'l' and also it is not case sensitive.
\W look for non-word character (some thing besides letters, digit, or underscore)
To find out the first non word character and to zap everything from there to the end of the string from $name:
$name =~ s/\W.*//;
$name =~ s/\W.*//;
\W ---> looks for non word character
.* ---> any character from there to the end of the line
s ---> substitute operator. It is followd by slash deliminated regular expression and string (in the above example, it is empty string which replaces non word charaters with nothing)
.* ---> any character from there to the end of the line
s ---> substitute operator. It is followd by slash deliminated regular expression and string (in the above example, it is empty string which replaces non word charaters with nothing)
To put the contents of $name in to lowercase letters ‘tr’ operator is used:
$name =~ tr/A-Z/a-z/;
$name =~ tr/A-Z/a-z/;
Here are some special RE characters and their meaning
. # Any single character except a newline
^ # The beginning of the line or string
$ # The end of the line or string
* # Zero or more of the last character
+ # One or more of the last character
? # Zero or one of the last character
\w # Any alphanumeric (word) character. Same as [a-zA-Z0-9_]
\W # Any non-word character. Same as [^a-zA-Z0-9_]
\d # Any digit. The same as [0-9]
\D # Any non-digit. The same as [^0-9]
\s # Any whitespace character: Space, tab, newline, etc
\S # Any non-whitespace character
\b # A word boundary, outside [] only
\B # No word boundary
CONTROL Structures
Statement Blocks: It is a sequence of statements, enclosed in catching curly braces. It looks like{ first_statement; second_statement; …… last statement; }
if /unless Statement
if (some expression) {
true_satemets;
} else {
false_statemenrs;
}
We can replace the if conditon with ‘unless’
condition. ‘unless’ execute the statements, if the conditions is false.
‘unless’ can also have an else, just like an if.
If you have more than two choices, add an elsif branch to the if statement.
while / until Loop
while ( some_expression) {
statement_1;
statement_2;
}
The do {} while/until Statement:
The while/until statement tests its condition at the top of every loop,
before the loop is entered. If the condition was already false to begin
with, the loop won't be executed at all.
But sometimes you don't want to test the condition at
the top of the loop. Instead, you want to test it at the bottom. To fill
this need, Perl provides the do {} while statement, which is just like
the regular while statement except that it doesn't test the expression
until after executing the loop once.
do {
statement_1;
statement_2;
} while some_expression;
The for Statement
for ( initial_exp; test_exp; re-init_exp ) {
statemenr_1;
statement_2;
}
The foreach Statement: This
statement takes a list of values and assigns them one at a time to a
scalar variable., executing block of code with each successive
assignment. It looks like this:
foreach $i (@some_list) {
Statement_1;
Statement_2;
}
Example of foreach:
@a=(1,2,3,4,5);
foreach $b (reverse @a) {
print $b;
}
The last command is like the break statement in C (as used in loops); it immediately exits the loop in question
Perl's Favorite Default Variable: $_
If you omit the control variable from the beginning of the foreach loop, Perl uses its favorite default variable, $_.
foreach (1..10) { # Uses $_ by default
print "I can count to $_!\n";
}
Although this isn't Perl's only default by a
long shot, it's Perl's most common default. Perl will automatically use
$_ when you don't tell it to use some other variable or value, thereby
saving the programmer from the heavy labor of having to think up and
type a new variable name. One of those cases is print, which will print
$_ if given no other argument:
$_ = "Yabba dabba doo\n";
print; # prints $_ by default
File Handles and File tests
$! variable, which contains the error string describing the most recent operating system error. It's used like this:open(LOG, ">>logfile") || die "cannot append: $!"; $filename = <STDIN>; chomp $filename; # toss pesky newline if (-r $filename && -w $filename) { # file exists, and I can read and write it ... }
File Tests and Their Meanings
Test Meaning
-r File or directory is readable
-w File or directory is writable
-x File or directory is executable
-o File or directory is owned by user
-R File or directory is readable by real user, not effective user
(differs from -r for setuid programs)
-W File or directory is writable by real user, not effective user
(differs from -w for setuid programs)
-X File or directory is executable by real user, not effective user
(differs from -x for setuid programs)
-O File or directory is owned by real user, not effective user
(differs from -o for setuid programs)
-e File or directory exists
-z File exists and has zero size (directories are never empty)
-s File or directory exists and has nonzero size
(the value is the size in bytes)
-f Entry is a plain file
-d Entry is a directory
-l Entry is a symlink
-S Entry is a socket
-p Entry is a named pipe (a "fifo")
-b Entry is a block-special file (like a mountable disk)
-c Entry is a character-special file (like an I/O device)
-u File or directory is setuid
-g File or directory is setgid
-k File or directory has the sticky bit set
-t isatty() on the filehandle is true
-T File is "text"
-B File is "binary"
-M Modification age in days
-A Access age in days
-C Inode-modification age in days
Functions
timelocal and localtime are two different functions.
To find out today’s date:
($DAY, $MONTH, $YEAR) = (localtime)[3,4,5];
( the out put format : month is 0 for Jan and 11 for Dec and year is actual year – 1900)
Or, use Time::localtime, which overrides localtime to return a Time::tm object:
use Time::localtime; $tm = localtime; ($DAY, $MONTH, $YEAR) = ($tm->mday, $tm->mon, $tm->year);
Converting DMYHMS (Date, Month, Year, Hour, Min, Sec) to Epoch Seconds
Use the timelocal or timegm functions in the standard
Time::Local module. Replace 'timelocal' with 'timegm' if your input date
is GMT/UTC
use Time::Local;
$TIME = timelocal($sec, $min, $hours, $mday, $mon, $year);
Here's an example that shows how to find Epoch
seconds for a time in the current day. It gets the day, month, and year
values from localtime:
use Time::Local;
$time = timelocal($seconds, $minutes, $hours, (localtime)[3,4,5]);
time function gives today’s date in epoch seconds
localtime(epoch seconds) converts epoch seconds to readable format.
localtime(epoch seconds) converts epoch seconds to readable format.
Converting epoch seconds to DMYHMS:
($seconds, $minutes, $hours, $day_of_month, $month, $year,
$wday, $yday, $isdst) = localtime($time);
or
($seconds, $minutes, $hours, $day_of_month, $month, $year,
use Time::localtime;
$tm = localtime($time);
printf("Dateline: %02d:%02d:%02d-%04d/%02d/%02d\n",
$tm->hour, $tm->min, $tm->sec, $tm->year+1900, $tm->mon+1, $tm->mday); # perl -le 'print scalar localtime 12545 * 24 * 60 * 60 ' Thu May 6 17:00:00 2004 cat time.pl #!/usr/bin/perl $a = time ; print "$a \n"; print scalar localtime (time) ; # ./time.pl 1087418422 Wed Jun 16 13:40:22 2004
PERL ADMIN STUFFS:
To run your program with warnings turned on, use the -w option on the command line:$ perl -w my_program
To find out the Installed perl modules, use the following script.
- !/usr/bin/perl
use ExtUtils::Installed;
my $instmod = ExtUtils::Installed->new();
foreach my $module ($instmod->modules()) {
my $version = $instmod->version($module) || "???";
print "$module -- $version\n";
}
my $instmod = ExtUtils::Installed->new();
foreach my $module ($instmod->modules()) {
my $version = $instmod->version($module) || "???";
Each time a module is installed on your system, it
appends information to a file called perllocal.pod which can be found in
/usr/local/lib/perl5/version number/architecture/ or something akin to
that. We can find out the installed modules by running the following
command.
# cat perllocal.pod | grep Module
To install new Perl modules:
The CPAN module is designed to automate the make and
install of perl modules and extensions. It includes some searching
capabilities and knows how to use Net::FTP or LWP (or lynx or an
external ftp client) to fetch the raw data from the net. Modules are
fetched from one or more of the mirrored CPAN (Comprehensive Perl
Archive Network) sites and unpacked in a dedicated directory.
Installing a new module can be as simple as typing perl -MCPAN -e 'install Chocolate::Belgian'.
To go to interactive mode,
perl -MCPAN -e shell
Once you are on the command line, type 'h' and the rest should be self-explanatory.
To automatically install the perl modules
# perl -MCPAN -e 'install DBD::mysql' Python
a = 10
b = 10
print(a)
c = a + b
print(c)
We use colons to separate each key from its value, and each key and value is surrounded by single quotes. Notice, too, that the items in a map are enclosed in braces ({}), not parentheses or square brackets.
b = 10
print(a)
c = a + b
print(c)
To use more than one line of text in your string
(called a multiline string), use three single quotes ('''), and then hit
enter between lines
>>> fred = '''How do dinosaurs pay their bills? With tyrannosaurus checks!'''
If we use three single quotes, we can put any
combination of single and double quotes inside the string (as long as we
don’t try to put three single quotes there).
>>>silly_string = ' ' ' He said, "Aren't can't shouldn't wouldn't." ' ' '
Embedding values
>>> myscore = 1000 >>> message = 'I scored %s points' >>> print(message % myscore) I scored 1000 pointsHere, we create the variable myscore with the value 1000 and the variable message with a string that contains the words “I scored %s points,” where %s is a placeholder for the number of points. On the next line, we call print(message) with the \% symbol to tell Python to replace %s with the value stored in the variable myscore.
Another example:
>>> joke_text = '%s: a device for finding furniture in the dark' >>> bodypart1 = 'Knee' >>> bodypart2 = 'Shin' >>> print(joke_text % bodypart1) Knee: a device for finding furniture in the dark >>> print(joke_text % bodypart2) Shin: a device for finding furniture in the dark
>>> var1 = "Jeeva"
>>> print('My name is %s' % var1)
My name is Jeeva
>>> var = 'a' >>> print(var * 10) aaaaaaaaaa
LISTS:
>>> wizard_list = ['spider legs', 'toe of frog', 'eye of newt', 'bat wing', 'slug butter', 'snake dandruff'] >>> print(wizard_list) ['spider legs', 'toe of frog', 'eye of newt', 'bat wing', 'slugbutter', 'snake dandruff'] >>> print(wizard_list[2]) eye of newt
To change the item in index position 2, previously eye of newt, to snail tongue.
>>> wizard_list[2] = 'snail tongue' >>> print(wizard_list) ['spider legs', 'toe of frog', 'snail tongue', 'bat wing', 'slugbutter', 'snake dandruff']
Writing [2:5] is the same as saying, “show the items
from index position 2 up to (but not including) index position 5”—or in
other words, items 2, 3, and 4.
>>> print(wizard_list[2:5]) ['snail tongue', 'bat wing', 'slug butter']
And lists might even store other lists:
>>> numbers = [1, 2, 3, 4] >>> strings = ['I', 'kicked', 'my', 'toe', 'and', 'it', 'is', 'sore'] >>> mylist = [numbers, strings] >>> print(mylist) [[1, 2, 3, 4], ['I', 'kicked', 'my', 'toe', 'and', 'it', 'is', 'sore']]
To add a item to the list
>>> wizard_list.append('bear burp')
>>> print(wizard_list)
['spider legs', 'toe of frog', 'snail tongue', 'bat wing', 'slugbutter', 'snake dandruff', 'bear burp']
To remove items from a list, use the del command
>>> del wizard_list[5] >>> print(wizard_list) ['spider legs', 'toe of frog', 'snail tongue', 'bat wing', 'slugbutter', 'bear burp', 'mandrake', 'hemlock', 'swamp gas']
Tuples
A tuple is like a list that uses parentheses, as in this example:>>> fibs = (0, 1, 1, 2, 3) >>> print(fibs[3]) 2The main difference between a tuple and a list is that a tuple cannot change once you’ve created it.
Maps or Dictionaries
In Python, a map (also referred to as a dict, short for dictionary) is a collection of things, like lists and tuples. The difference between maps and lists or tuples is that each item in a map has a key and a corresponding value.We use colons to separate each key from its value, and each key and value is surrounded by single quotes. Notice, too, that the items in a map are enclosed in braces ({}), not parentheses or square brackets.
>>> favorite_sports = {'Ralph Williams' : 'Football',
'Michael Tippett' : 'Basketball',
'Edward Elgar' : 'Baseball',
'Rebecca Clarke' : 'Netball',
'Ethel Smyth' : 'Badminton',
'Frank Bridge' : 'Rugby'}
To get Rebecca Clarke’s favorite sport
>>> print(favorite_sports['Rebecca Clarke']) Netball
To replace a value in a map, we also use its key:
>>> favorite_sports['Ralph Williams'] = 'Ice Hockey'
>>> print(favorite_sports)
{'Rebecca Clarke': 'Netball', 'Michael Tippett': 'Basketball', 'Ralph Williams': 'Ice Hockey', 'Edward Elgar': 'Baseball', 'Frank Bridge': 'Rugby'}
To remove Ethel Smyth
>>> del favorite_sports['Ethel Smyth']
>>> print(favorite_sports)
{'Rebecca Clarke': 'Netball', 'Michael Tippett': 'Basketball', 'Ralph Williams': 'Football', 'Edward Elgar': 'Baseball', 'Frank Bridge': 'Rugby'}
To add Tom to the list
>>>favorite_sprots.update({'Tom' : 'Hoecky'})
Inbuilt Functions
To convert string to number>>> age = '10' >>> converted_age = int(age)
To convert number to strings
>>> age = 10 >>> converted_age = str(age)
IF statements:
Simple if condition>>> age=20
>>> if age > 20:
print('You are too old!')
If else condition
>>> age=20
>>> if age > 20:
print('Your are too old!')
else:
print('It is Secret.')
if, elif condition
>>> age = 20
>>> if age == 10:
print('It is 10')
elif age == 15:
print('It is 15')
elif age == 20:
print('It is 20')
Combining Conditions:
>>> if age >= 10 and age <= 13 or age >= 50 and age <= 50:
print('What is 13 + 49 + 84 + 155 + 97? A headache!')
else:
print('Huh?')
Loops
To print Hello five times; >>> for x in range(0, 5):
print ('Hello')
>>> coins=10
>>> bills=10
>>> stolen=5
>>> for week in range(1,53):
coins =coins + bills - stolen
print('Week %s = %s' % (week, coins))
Week 1 = 15
Week 2 = 20
Week 3 = 25
Week 4 = 30
Week 5 = 35
........
........
>>> x=10 >>> y=20 >>> while x < 25 and y < 35: x = x + 1 y = y + 1 print(x,y) 11 21 12 22 13 23 ...... ......
while True: lots of code here if some_value == True: break
Using Functions:
Sed
Sed
- Sed is a Non-interactive stream-oriented editor.
- It allows you to edit files and save the output. This prevents you from permanently modifying an original file.
- You can keep sed commands in a file called a sed script if you need to perform mlutiple edits or if you do not want to worry about quoting sed commands at the shell command line.
- sed processed a file or input one line at a time and sends its output to the screen. It stores the line it is currently processing in a temporary buffer called pattern space. after sed has finished the line in the pattern space, the line is sent to the screen.
- sed uses addressing to specify which lines are to edited. The addressing can be in the form of numbers or regular expressions, or both. if a line is not specified, sed processes all the lines in the input file.
- When the address is a number, the number represents a line number. A dollar sign can be used to represent the last line of the input file. If the comma separates two line numbers, the addresses that are processed are within that range.
-n Suppresses all information normally written to standard output -f <Script file> Uses the ScriptFile variable as the source of the edit script -i or --in-place sed creates no output. But changes the source file itself.
s for substitution and /g for Global replacement
To replace "MA" with "Massachusetts in first occurance$ sed 's/MA/Massachusetts/' list
To replace "MA" with "Massachusetts in all occurances
$ sed 's/MA/Massachusetts/g' list
/ as Delimeter
The character after the s is the delimiter. It is conventionally a slash, because this is what ed, more, and vi use. It can be anything you want, however. If you want to change a pathname that contains a slash, you could use the backslash to quote the slash:sed 's/\/usr\/local\/bin/\/common\/bin/' <old >new
Gulp. It is easier to read if you use an underline instead of a slash as a delimiter:
sed 's_/usr/local/bin_/common/bin_' <old >new
Some people use commas, others use the "|" character. Pick one you like
Using & as the matched string
Sometimes you want to search for a pattern and add some characters, like parenthesis, around or near the pattern you found. It is easy to do this if you are looking for a particular string:$ sed 's/abc/(abc)/' <old >new
This won't work if you don't know exactly what
you will find. How can you put the string you found in the replacement
string if you don't know what it is?
The solution requires the special character "&." It corresponds to the pattern found.
$ sed 's/[a-z]*/(&)/' <old >new
You can have any number of "&" in the replacement string. You could also double a pattern, e.g. the first number of a line:
$ echo "123 abc" | sed 's/[0-9]*/& & &/' 123 123 123 abc
sed -n noprinting and /p print
By default, sed prints every line. If it makes a substitution, the new text is printed instead of the old one. The "-n" option will not print anything unless a explicit request to print is found. When the "-n" option is used, the "p" flag will cause the modified line to be printed. Here is one way to duplicate the function of grep with sed:$ sed -n 's/pattern/&/p' <file
Multiple commands with -e command
One method of combining multiple commands is to use a -e before each command:$ sed -e 's/a/A/' -e 's/b/B/' <old >new
sed -f scriptname
If you have a large number of sed commands, you can put then into a file and use$ sed -f sedscript <old >new
sed in shell script
If you have many commands and they won't fit neatly on one line, you can break up the line using a backslash:sed -e 's/a/A/g' -e 's/e/E/g' -e 's/u/U/g' <old >new
A quote can cover several lines too:
#!/bin/sh sed ' s/a/A/g s/o/O/g s/u/U/g' <old >new
Exmaples:
Append a string "hello" to starting of each line
sed 's/^/hello /g' filesname > filename.out
Printing with p
If sed wasn't started with an "-n" option, the "p" command will duplicate the input
To print the first 10 lines of a file:
# sed -n '1,10 p' <file
To print the 10th line alone
# sed -n 10p <file
To print the lins between 10th and and the last line
# sed -n -e 's,$p' file_name
To print the next line after a matched pattern:
# sed -n '/blah/{n;p;}' file_name
Deleting lines
$ sed '/The Netherlands/d` $ sed '/The Netherlands/!d` # deletes lines that don't contain The Netherlands
sed options:
c\ Replaces text in the current line with new text
a\ Appends on or more lines of text to the current line
d Deletes line
i\ Inserts text above the current line
s Subtitues on string for another
! Applies the command to all lines except the selected ones
h (H) Copies or appends the contests of the pattern space to the holding buffer
g (G) Retrives what is the holding buffer and copies it into a pattern buffer,
appending or overwriting what was there
Shell Scripts
Variables
To separare variable from the rest of the text, use {}
$ echo ${name}7
To remove a variable $name
$ unset $name
To assign a result of a command to a variable
$NAME=`whoami`
or
$NAME=$(whoami)
To set the value of $* using content of a file
set $(cat file1)
shift is a shell builtin that
operates on the positional parameters. Each time you invoke shift, it
"shifts" all the positional parameters down by one. $2 becomes $1, $3
becomes $2, $4 becomes $3, and so on.
Standard I/O Redirection
To send the stdout and stderr to two separate files
$ prog >file1 2>file2
To send both stdout and stderr to the same file
$ prog >file 2>&1
To pipe stdout and stterr of prog1 to stdin of of prog2
$ prog1 |& prog2
Arithmatic in shell scripts
To add $a and $b
$ let c=$a+$b; echo $c
or
$ c=$(( a+b )) ; echo $c
or
$ echo $a+$b | bc
or
$ expr $a + $b
$ expr 2 \* 2
Assign the variable value to another variable
eval reads its arguments as input to the shell and executes the resulting command(s). This is usually used to execute commands generated as the result of command or variable substitution.
eval reads its arguments as input to the shell and executes the resulting command(s). This is usually used to execute commands generated as the result of command or variable substitution.
i.e unmount.ksh #!/bin/ksh #usage $0 [ MYHOME | MYDATA | MYLOG ] UMOUNT=$1 MYHOME="/u01/myhome" MYDATA="/u01/mydata" MYLOG="/u01/mylog" eval MNTDIR=\$(UMOUNT} /usr/sbin/umount $MNTDIR
Array Variables
In ksh, the syntax isset -A name value1 value2 ... valueN
In Bash
name=(value1 ... valueN)To assign the output of command to array variable
dirs=($(ls))
echo ${dirs[2]}
Array variables can be accessed as follows
${name[index]}
example:
$ set -A address 404 'Broadway st' dublin ca
$ echo ${address[0]}
404
$ echo ${address[1]}
Broadway st
$ echo ${address[2]}
dublin
Setting array variable using std input
while read lines; do set -A var $lines; echo ${var[1]}; echo ${var[2]}; done # ksh
while read lines; do var ($lines); echo ${var[1]}; echo ${var[2]}; done # bash
Using read to get user input for a variable value, also to prompt the user for input
read user_name?" Please enter the user ID (Login Name) to add: " Please enter the user ID (Login Name) to add:
Testing
Test command can perform tests on numeric and string data as well as on files. The test command can be used in explicit or implicit mode.
Test in explicit mode
$ test "ABC" = "abc"
Test in implicit mode
$ [ "ABC" = "abc" ]
Test result of command execution using "if" directly instead of using $?
if ! diff $TMPFILE_old $TMPFILE_new > /dev/null ; then ........ ........ fi
Testing Numeric Values
-e Equality check -ne Not equal -lt Less than -gt Greater than -le Less than or eqaul to -ge greater than or equal to
Testing String Values
-z string True if string length is zero -n string True if string lenth is non zero string1 = string2 True if string1 and string2 are equal string1 != string2 True if string1 is not equal to string2
Testing Files
-d file True if file is a directory -f file True if file exists and not directory -s file True if file is more than 0 bytes in length -r file True if file is readable -w file True if file is writable -x file True if file is executable -L file True if file is symbolic link file1 -nt file2 True if file is newer than file2
Loop
While loop
cat "filename" | while read data
do
.......
.......
done
do
.......
.......
done
case function
case $variable-name in
pattern1) command
...
command;;
*) command
...
command;;
esac
Breaking the loop
The break command discontinues the execution of loop immediately and transferes control to the command following the done
keyword. You can pass a number n as an aargument to the break command.
In that case, the break command jumps to the command after the nth
occurrence of the done keyword.
Continue command
The continue skips the remaining part of the loop and transfer the control to the start of the loop for the next iteration.
Sample script to parallel copy files using break
threats=2
ls| while read file; do
echo $file
nohup sleep 30 &
while true; do
process=$(ps -ef | grep sleep | grep -v grep| wc -l)
test $process -lt $threats && break
done
done
Functions
To define function function_name ()
{
......
......
}
Getopts (To process multiple options)
Use getopts internal command to process mutiple options in a shell scripts while getopts ":ab:c" opt; do
case $opt in
a ) process option -a ;;
b ) process option -b
$OPTARG is the option's argument ;;
c ) process option -c ;;
\? ) print 'usage: bob [-a] [-b barg] [-c] args...'
return 1
esac
done
shift $(($OPTIND - 1))
normal processing of arguments...
- Its first argument is a string containing all valid option letters. If an option requires an argument, a colon follows its letter in the string. An initial colon causes getopts not to print an error message when the user gives an invalid option.
- Its second argument is the name of a variable that will hold each option letter (without any leading dash) as it is processed.
- If an option takes an argument, the argument is stored in the variable OPTARG.
- The variable OPTIND contains a number equal to the next command-line argument to be processed. After getopts is done, it equals the number of the first "real" argument.
example:
while getopts ":i:d:s:" opt; do
case "$opt" in
i) initial=$OPTARG ;;
d) dir=$OPTARG ;;
s) sub=$OPTARG ;;
*) printf "unknown flag supplied "${OPTARG}\nUsageMessageGoesHere\n"
exit 1;;
esac
done
shift $(( OPTIND - 1 ))
path="/$initial/$dir/$sub"
mkdir -p "$path"
for file in "$@"; do
touch "$path/$file"
done
or
for subdir in $sub;do
for file in $files;do
echo $subdir/$file
done
done
Tips
In the bash shell, to enable interpretation of backslash escapes, use the -e option$ echo -e "this is a test line \n"
basename is a standard UNIX
computer program, when basename is given a pathname, it will delete any
prefix up to the last slash ('/') character and return the result.
basename is described in the Single UNIX Specification and is a
primarily used in shell scripts.
# basename /home/user1/test.sh test.sh
dirname delivers all but the last level of the path name in string
# dirname /usr/local/bin /usr/local
trap
trap "" 1 2 3 trap 'clear' 2 # clear screen when break signal is sent trap - 2 ## To undo the trap
TMOUT variable
It is used to set auto time out variable the ksh or posix compliant shell. The TMOUT value is defined in seconds. We can define the following lines /etc/profile to make users not to change these values
It is used to set auto time out variable the ksh or posix compliant shell. The TMOUT value is defined in seconds. We can define the following lines /etc/profile to make users not to change these values
TMOUT=600 export TMOUT readonly TMOUT
In C-Shell autologout variable is used instead of TOMOUT. Define "set autologout=1200" in /etc/csh.login file
'''set options"
set -n # Reads commands but does not execute them set -f # disable wildcard filename expansion (globbing) set -e # Exits the program if a command returns a nonzero value set -x # echo line to stderr before executing it
Tips
PS1 variable:
PS1=`hostname -s`:'$PWD>' or PS1="`hostname -s` [\$PWD]# "
Do not user hostname -s in solaris, use uname -n
PS1="`uname -n`@\$PWD>"
or
PS1="`uname -n` [\$PWD] # "
To send the standard error and output to error.txt file:
$ ls -l > error.txt 2>&1
To make the History file separate for each user when they su to root :
Add the following lines to the .profile file of root:
ME=`who am i | awk '{print $1}'`
HISTSIZE=2000
HISTFILE=$HOME/sh_history/.sh_history.$ME
export HISTSIZE HISTFILE
To delete the files which are modified more than 30days
$ find /u/arnold -ctime +30 -exec \rm {} \;
To find out the files older than
touch -t YYYYMMDDhhmmss /tmp/touchfile
find ./ -older /tmp/touchfile -exec rm -f {}\;
touch -t [4hours ago] touchfile
find /path/to/files ! -newer touchfile -exec rm -f {} \;
To find out files newer that 2 hours and take action on them:
#!/usr/bin/ksh
#----subtract 30 minutes (1800 seconds) from time
STAMP=$(perl -e '($ss, $mm, $hh, $DD, $MM, $YY) = localtime(time() - 1800);
printf "%04d%02d%02d%02d%02d", $YY + 1900, $MM + 1, $DD, $hh, $mm')
#----create an empty file with a timestamp of 30 minutes ago
touch -t $STAMP /tmp/flagfile
#----find older files
find /start/dir -type f ! -newer /tmp/flagfile -print | while read FILE
do
: something with $FILE
done
To print sequence of numbers from 0 to 255 with increment of 1
# seq 0 1 255
Another way to do the same using for
for i in {1..255}; do echo $i; done
To print sequence of numbers from 0 to 255 with increment of 5
# seq 0 5 255
0
5
10
......
255
To traslate all lower case letters to UPPER case
$ tr 'a-z' 'A-Z' < textfile
xargs
# ls | xargs -t ls -l
To kill all the defunct process
# ps -ef | grep defunct | awk '{print $2}' | xargs -i kill -9 {}
To rename all the files in a directory with an .old extension
# ls | xargs -t -i mv {} {}.old
Connecting to console via console server device
$ telnet consoleserver 1001
To quit from the console
press ~. or break keys - ~#
Generating SSH Key
# 1024 bits with a key type of dsa
ssh-keygen -b 1024 -t dsa
Man
Run "catman" to build the "/usr/share/lib/whatis" file.
Run "catman" to build the "/usr/share/lib/whatis" file.
Then you can run the "man -k [some command] to see
what manual sections the command is in. For example, if you run "man -k
mkdir", it will show that "mkdir" is in sections 1 and 2.
To see section 2 you would run "man 2 mkdir".
sort:
-t to define delimeter
-r to sort in revers order
-n to say sort that the sorting field is numbers
To start a password file by user ids
# sort -t : -n +2
grep, egrep and fgrep
- egrep command is an extended grep command that supports more regular expression characters
- fgrep command treats all characters as literals, meaning that regular expression meta-characters are not special
-n # prints the line number the the actual line
-v # display lines only that do not match
-e # one or more patterns to be used during search
-i # ignore case
-h # does not display file names
Backing up MySQL
To backup any single database, you just have to type:
code:
mysqldump --add-drop-table -u dbusername -p dbname > dbname.bak.dump
// or better yet...
mysqldump --opt -u dbusername -p dbname > dbname.bak.dump
To backup any single database, you just have to type:
e.g. If I had to backup a certain database with the following details:
MySQL username is jds_user1 and MySQL database name is jds_db1
code:
mysqldump --add-drop-table -u jds_user1 -p jds_db1 > jds_db1.bak.dump
// or better yet...
mysqldump --opt -u jds_user1 -p jds_db1 > jds_db1.bak.dump
MySQL username is jds_user1 and MySQL database name is jds_db1
vi Editor
Cursor Movement
Text creation and Manipulation
Changing and deleting Text
Scrolling screen
Replacing words globally
Managing files
Other vi tips
Text creation and Manipulation
Changing and deleting Text
Scrolling screen
Replacing words globally
Managing files
Other vi tips
$HOME/.exrc - vi control file
j,k move cursor down or up h,l move cursor left or right + first character of next line - first character of previous line e end of word w Forward by word b Backward by word $ end of line 0 beginning of line H top line on screen L last line on screen M middle line on screen nG Go to given line n G Go to end of file
Text creation and Manipulation
i - insert text at current position a - append text at current position r - replace a singe character with new character o - Open blank line below cursor for text s - Delete character at cursor and substitute text x - Delete current cursor position A - Append text to end of current line I - Insert text at beginning of line O - Open blank line above cursor for text S - Delete line and substitute text R - Replace existing characters with new characters X - Delete back one character . - repeat last action u - undo last action U - Restore line to original state J - Join current and next line ~ - Toggle case
Changing and deleting Text
cw - change word cc - change line C - change text from current position to end of line dd - delete line ndd - delete n lines D - delete remainder of line dw - delete word yy - copy current line p - Insert last deleted or yanked text after cursor P - Insert last deleted or yanked text before cursor
Scrolling the Screen
^F Scroll forward one screen ^B Scroll backward one screen ^D Scroll forward half screen (down) ^U Scroll backward half screen (up)
Searching a pattern or text
/patern Search forward for pattern ?pattern Search backward for pattern n Repeat last search N Repeat last search in opposite direction fx Move to next occurrence of x in current line Fx Move to previous occurrence of x in current line tx Move to just before next occurrence of x in current line Tx Move to just after previous occurrence of x in current line
Replacing words Globally
To change xxxx to yyyy globally in a file
:%s/xxxx/yyyy/g
or
:g/xxx/s//yyy/
Managing files
:sh Temporarily goes to command prompt :w write file :q! quit file without saving :wq! Write and quite file :x save and quit :e! return to version of current file at time of last write :n Edit next file :n! forces next file :w file1 Save copy to file1 :e file2 Edit file2 without leaving vi :r file1 Read in contents of file1 after cursor :r !command Read in output from command after current line
- press ^G to find out the current line number anytime
- :set number to set line number in front of each line in vi
- :set nonum to not to display numbers
- :set all to list all available options
Commands Reference
ls -a # list all files including hidden files -l # long listing -t # sort by time modified -R # List all sub-directories -r # list in reverse order -d # to list the directory itself instead of files inside it
echo $PATH : Displays the path
/sbin/ifconfig -a : finds the IP Address
cat : look or combine
cp -r(copy all files,directories & sub-directories) -p (retains permission and owners)
cut -f <field> -d <deliminator)
date : Displays date and time
df -k, -v, -h, -m, -g : disk space
diff : Displays two files and prints the lines that are different.
du -s(usage including subdirectories) -k -h : file usage
echo : Display line of text
env : Displays unix environment variables
file : Determine file type
find -name -size -ctime -mtime -exec
grep -i -v -w .
gunzip : uncompress
gzip : compress
head : diplays ten lines of a file
hostname : systems host name
id : print user identity
kill -9 -1
last last login user -b bad login
ln -s
ls -l(long listing) -a(hidden files)
mkdir -p : make directory)
more : displays single page)
mv : move
nohup
passwd : set password
ps -e -f (list of current process)
reboot
rm : remove
rmdir : remove directory
sleep
sort -u
stty
su
tail
tee
test
tar txcrvf
touch <testfile> # creates an empty file called "testfile"
tr -s : replace repeated character with single
tty
uptime(how long its on)
uname -a(all) (print system information like kernel name,node name,os type,machine type......)
w
wall
wc -l -c -w
who Displays who is on the system.
who am i
whoami
xargs
/sbin/ifconfig -a : finds the IP Address
cat : look or combine
cp -r(copy all files,directories & sub-directories) -p (retains permission and owners)
cut -f <field> -d <deliminator)
date : Displays date and time
df -k, -v, -h, -m, -g : disk space
diff : Displays two files and prints the lines that are different.
du -s(usage including subdirectories) -k -h : file usage
echo : Display line of text
env : Displays unix environment variables
file : Determine file type
find -name -size -ctime -mtime -exec
grep -i -v -w .
gunzip : uncompress
gzip : compress
head : diplays ten lines of a file
hostname : systems host name
id : print user identity
kill -9 -1
last last login user -b bad login
ln -s
ls -l(long listing) -a(hidden files)
mkdir -p : make directory)
more : displays single page)
mv : move
nohup
passwd : set password
ps -e -f (list of current process)
reboot
rm : remove
rmdir : remove directory
sleep
sort -u
stty
su
tail
tee
test
tar txcrvf
touch <testfile> # creates an empty file called "testfile"
tr -s : replace repeated character with single
tty
uptime(how long its on)
uname -a(all) (print system information like kernel name,node name,os type,machine type......)
w
wall
wc -l -c -w
who Displays who is on the system.
who am i
whoami
xargs
bc
expr
at
chgrp
chmod
chown
cpio
dd
dmesg
fuser
iostat
mount
nice
netstat
renice
sar
umount
vmstat
xhost
chgrp
chmod
chown
cpio
dd
dmesg
fuser
iostat
mount
nice
netstat
renice
sar
umount
vmstat
xhost
dig
ftp
finger
host
nslookup
rcp
rlogin
rsh
telnet
traceroute
scp
ssh
sftp
Keine Kommentare:
Kommentar veröffentlichen