Sponsored Content
Top Forums Shell Programming and Scripting Pipes with the < and >> meanings. Post 302517622 by DGPickett on Wednesday 27th of April 2011 09:53:48 AM
Old 04-27-2011
You are very kind! It's dangerous to encourage me! Smilie



Knowing "how simple it really is" is simpler than learning tons by rote. Using truss/tusc/strace can be very educational, showing you all the calls under the covers. To round it out (never ends), this needs:
  1. <<WORD\n . . . \nWORD\n converts script lines to stdin (shell writes it to a temp file). It is nice to manage lists on separate lines and through pipes, not on the command line explicity and flowing horizontally unaligned all over the page. The <<-WORD variant allows you to add prefix tabs for indentation that are removed. Some command line expansions are done here, so for complex data, I prefer the more predictable "echo 'whatever . . .' | . . . ." (use single quotes first, always, as they are less meta-processed (more literal) than double quotes; slip out of single to double where you need it and then go back to single, e.g.: echo 'I got home at '"$time"', well before curfew.' Think of echo as a conversion tool to take command line strings to stdout, the opposite of the next item.
  2. $( command . . . ) When not in sh, this is nicer than `command . . .` 2 ways: nestable and has vi % support to match parens. Converts stdout to a string on the command line; may contain multiple commands whose stdout is inherited, concatenated,
  3. $(<file) Turns the content of a file into a command line string,
  4. ' . . . | command . . . $(cat)' Allows all of the output of a pipeline of commands to end up on the command line without the mess of nesting it all in `` or $(),
  5. ' . . . | xargs -n999 command . . .' Similar, but runs as many command lines of limited length as necessary (scales well), and starting sooner so you get lower latency,
  6. $*, $#, shift, $@, $0-9 Managing the command line args,
  7. . . . | read variable_name Getting the one value or first line only from a command or pipeline of commands,
  8. | while read variable_name do . . . done Processing each line comng out of the pipeline separately, with low latency, robust for all data set sizes but scaling not so well (shell processing speed possibly with one or more fork/exec per line processed).
  9. Named pipes from "mknode pipe_path p" or "mkfifo pipe_path", especially useful if you do not have the /dev/fd/# ksh <() >() (bash uses named pipes under the covers for >() <(), ksh does not).
  10. ${var_name#pattern} with #, ##, %, %%: Chews off nose # minimally or ## aggressively or tail %, %%. Mnemonic: Pound on the nose, get your percent in the end.
  11. trapping signals and kill,
  12. advanced data and storage: typedef, arrays, relational maps (value addressed arrays).
  13. export for child processes, subshells, and how to stay in the first shell so you can modify variables in subroutines and
  14. sourced files (". filename" includes another file in the stdin flow of this shell).
Mostly a basic ksh list, ksh93 and bash goes on much farther. Begs the need for the arithmetic bits, date/time bits, test, with regex matching, case. I love 'case', because it structures testing so all cases are addressed, and facilitates nice commenting! Just use balanced () in case patterns, so the vi % feature is not incapacitated.

BTW, a nice use of 'line' and subshell is to avoid using a command that must pass all the data, just to remove the header line(s) before processing:
Code:
( . . . | ( line >&2 ; sort ) >output_file ) 2>>log_file

Subshell is also great for collecting logging or multipel commands' output without saying >>file_name over and over, like the stderr to log_file above. Trailer lines are a different problem, maybe time to learn some sed:
Code:
 . . . | sed '
  ${
    w trailer_file
    d
   }
 ' | . . . .

The best training is for use, not to pass a certification by knowing every dumb feature; to know which pragmas are useful, clear, fast and robust. It does not hurt to teach formatting in a normal way, so it reads well for review and maintenance. In all code, a common sin is not formatting lists of predicates or actions on separate lines, not throwing in blank lines and indentation so multi-line formatted commands and container expressions like if stand out from surrounding single-line commands. Both the quality of your work and the size and speed of project you can execute are enhanced by good formatting, good logging and good error handling. Don't learn to be a weak, hope and prayer, cryptic script writer; it is not worth the pain or the limitation. You, too, deserve well behaved, robust, easily read, tracable code, even in shells!

PPS: if you only use cd inside subshells, you never leave ~ $HOME, and all your commands can be recalled and rerun using $EDITOR=vi and ksh set -o vi and the like. Increase you $HISTSIZE to 32767, put your $HISTFILE on permanent, and if possible net mounted, disk (not /tmp), periodically save your $HISTFILE, and recall how you did it the last time using escape /pattern. This is the nicest Character User Interface (CUI) this side of becoming an emacs addict. It saves you more typing than cd of your login shell ever did, and reduces mistakes - running a command in the wrong directory. The path you used to type in to cd can be recalled and reused with the whole command 'line' (can be many line single shell command). Recalled commands are also good starters for scripts. Really long command lines need to be passed through vi (escape /pattern v) to avoid being truncated, but still work over and over!

Last edited by DGPickett; 04-27-2011 at 11:56 AM..
These 6 Users Gave Thanks to DGPickett For This Post:
 

10 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

PIPEs and Named PIPEs (FIFO) Buffer size

Hello! How I can increase or decrease predefined pipe buffer size? System FreeBSD 4.9 and RedHat Linux 9.0 Thanks! (1 Reply)
Discussion started by: Jus
1 Replies

2. Shell Programming and Scripting

cd using pipes

Hi, Can the cd command be invoked using pipes??? My actual question is slightly different. I am trying to run an executable from different folders and the path of these folders are obtained dynamically from the front end. Is there a way in which i can actually run the executable... (2 Replies)
Discussion started by: Sinbad
2 Replies

3. Shell Programming and Scripting

where can I get exit code meanings?

I'm investigating strange behaviour on two boxes (Sun OS 5.10 and AIX 5.1) in ksh have used $? to get exit codes returned:- 137 and 34 where can I find what these mean? thank you (1 Reply)
Discussion started by: speedieB
1 Replies

4. Shell Programming and Scripting

named pipes

How to have a conversation between 2 processes using named pipes? (5 Replies)
Discussion started by: kanchan_agr
5 Replies

5. Cybersecurity

Syslog events meanings

Hi everybody, I'm writing to know what the following event stands for. I know that the following event is about a "su to root" action but I don't have any Idea about what action could rise this message. For example If an acction performed by the root crontab, a sudo command or something like that.... (1 Reply)
Discussion started by: PVelazco
1 Replies

6. Shell Programming and Scripting

Pipes not working

Hi, thanks for b4. can anyone tell me why following not working: noUsers=$(who | cut -d" " -f1 | wc -l) What i'm trying to do is get a list of logged on users and pass it to 'wc -l' and store the output to a variable. Any ideas? (1 Reply)
Discussion started by: Furqan_79
1 Replies

7. UNIX for Dummies Questions & Answers

learning about pipes!

im trying to figure out how to do the following: using pipes to combine grep and find commands to print all lines in files that start with the letter f in the current directory that contain the word "test" for example? again using pipes to combine grep and find command, how can I print all... (1 Reply)
Discussion started by: ez45
1 Replies

8. Shell Programming and Scripting

need meanings for FTP codes

Hi Friends, Could i get the meaning for the following FTP codes? 421 425 426 530 450 550 451 551 452 552 553 Thanks, Raja. (1 Reply)
Discussion started by: smr_rashmy
1 Replies

9. Programming

Pipes in C

Hello all, I am trying to learn more about programming Unix pipes in C. I have created a pipe that does od -bc < myfile | head Now, I am trying to create od -bc < myfile | head | wc Here is my code, and I know I might be off, thats why I am here so I can get some clarification. #include... (1 Reply)
Discussion started by: petrca
1 Replies

10. UNIX for Dummies Questions & Answers

top command: abbrevations and meanings - Please !

Hi all, I was trying see some CPU utilization of a Red hat Linux machine using 'top' command. Any way I got high level idea from the out puts, but when I observed the following line: Cpu(s): 7.4%us, 0.5%sy, 0.0%ni, 91.6%id, 0.4%wa, 0.0%hi, 0.1%si, 0.0%st I couldn't make out what... (2 Replies)
Discussion started by: a99u
2 Replies
All times are GMT -4. The time now is 06:37 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy