Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Using xargs for multiple functions
Top Forums Shell Programming and Scripting Using xargs for multiple functions Post 302436181 by methyl on Friday 9th of July 2010 12:35:10 PM
Old 07-09-2010
Quote:
for i in `nawk '/From/ {print $3}' syslogs | uniq`; do grep $i syslogs >> $i
More points:
1) This is not a good command structure for an open ended list because it leaves the shell with a potentially massive "for" command to process.
We can turn the "for" into a "while" and remove that issue.
2) We need to "sort" the node selection or "uniq" will not work properly and we will be making selections multiple times.
3) We can replace ">>" with ">" because each file is now only written once.
4) The "cat" to a pipeline was harmless and can be quicker at getting records into "awk" than getting "awk" to read the file direct. I've left it out because it is the convention of this board.

Overall this should be quicker and more importantly should not give multiple repeat blocks of data because of the missing "sort".

Code:
nawk '/From/ {print $3}' syslogs | sort | uniq | while read node
do
      grep "${node}" syslogs > "${node}"
done

Last edited by methyl; 07-09-2010 at 01:36 PM.. Reason: typos
This User Gave Thanks to methyl For This Post:
krypton
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Can Xargs execute multiple commands of evry input file

Hello , I am trying to print the footer of evry file in the given directory with xargs command like follows ls -1 | xargs -I {} gzcat {} | tail -1 now problem with this is only last file foooter is getting printed as " | tail -1 " is getting executed for the last file. I know this can... (4 Replies)
Discussion started by: nilesrex
4 Replies

2. Shell Programming and Scripting

Multiple functions

Hi, which of the shell support multiple functions in a shell script? Thanks Amit (1 Reply)
Discussion started by: amitrajvarma
1 Replies

3. Shell Programming and Scripting

Run Multiple Functions over SSH (BASH)

I am trying to write a script that will ssh into a remote machine and recurse through a specified directory, find mp3 files which may be two or three directories deep (think iTunes: music/artist/album/song.mp3), and scp them back to the machine running the script. The script should also maintain... (3 Replies)
Discussion started by: johnnybg00de
3 Replies

4. Shell Programming and Scripting

Calling multiple functions in parallel

Hello, I have multiple functions within a shell script. eg. function_database_backup, unix_tar_creation, etc. I would like to run these functions in parallel, as each is independent of the other. If these were simple commands, I could have probably run each of the commands in background. ... (1 Reply)
Discussion started by: neil.k
1 Replies

5. Shell Programming and Scripting

How to run multiple functions in Background in UNIX Shell Scripting?

Hi, I am using ksh , i have requirement to run 4 functions in background , 4 functions call are available in a case that case is also in function, i need to execute 1st function it should run in background and return to case and next i will call 2nd function it should run in background and... (8 Replies)
Discussion started by: karthikram
8 Replies

6. Shell Programming and Scripting

Xargs: multiple commands to each argument

Hello. There is my one-liner to get subjects of potential spam mails sudo exiqgrep -bf "spamer@example.com" |cut -d' ' -f1 |xargs -I ~ sudo /usr/sbin/exim -Mvh ~ |grep 'Subject: ' I want to insert blank line after each iteration to make output more readable. I tried sudo exiqgrep -bf... (1 Reply)
Discussion started by: urello
1 Replies

7. Shell Programming and Scripting

Parallel processing for functions in xargs

I have a script (ksh) which tries to run a function in parallel for performance gains. I am also trying to limit the number of parallel child processes to avoid overloading the system by using a variable to count triggered processes and waiting for completion e.g. do_something () { ... } ... (9 Replies)
Discussion started by: jawsnnn
9 Replies

8. Shell Programming and Scripting

Python passing multiple parameters to functions

Hi, I am a beginner in python programming. In my python script have a main function which calls several other functions. The main function gets its input by reading lines from a input text file. I call the main function for every line in input text file through a loop. def main(line): var1... (6 Replies)
Discussion started by: ctrld
6 Replies

9. Shell Programming and Scripting

Shell script to apply functions to multiple columns dynamically

Hello, I have a requirement to apply hashing algorithm on flat file on one or more columns dynamically based on header sample input file ID|NAME|AGE|GENDER 10|ABC|30|M 20|DEF|20|F say if i want multiple columns based on the header example id,name or id,age or name,gender and hash and... (13 Replies)
Discussion started by: mkathi
13 Replies

10. Shell Programming and Scripting

Xargs to call python executable to process multiple bam files

I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when | xargs sh is added I get the error. I tried adding | xargs... (4 Replies)
Discussion started by: cmccabe
4 Replies

LEARN ABOUT NETBSD

uniq

UNIQ(1) 						    BSD General Commands Manual 						   UNIQ(1)

NAME

     uniq -- report or filter out repeated lines in a file

SYNOPSIS

     uniq [-cdu] [-f fields] [-s chars] [input_file [output_file]]

DESCRIPTION

     The uniq utility reads the standard input comparing adjacent lines, and writes a copy of each unique input line to the standard output.  The
     second and succeeding copies of identical adjacent input lines are not written.  Repeated lines in the input will not be detected if they are
     not adjacent, so it may be necessary to sort the files first.

     The following options are available:

     -c      Precede each output line with the count of the number of times the line occurred in the input, followed by a single space.

     -d      Don't output lines that are not repeated in the input.

     -f fields
	     Ignore the first fields in each input line when doing comparisons.  A field is a string of non-blank characters separated from adja-
	     cent fields by blanks.  Field numbers are one based, i.e. the first field is field one.

     -s chars
	     Ignore the first chars characters in each input line when doing comparisons.  If specified in conjunction with the -f option, the
	     first chars characters after the first fields fields will be ignored.  Character numbers are one based, i.e. the first character is
	     character one.

     -u      Don't output lines that are repeated in the input.

     If additional arguments are specified on the command line, the first such argument is used as the name of an input file, the second is used
     as the name of an output file.

     The uniq utility exits 0 on success, and >0 if an error occurs.

COMPATIBILITY

     The historic +number and -number options have been deprecated but are still supported in this implementation.

SEE ALSO

     sort(1)

STANDARDS

     The uniq utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible.

BSD
								  January 6, 2007							       BSD
All times are GMT -4. The time now is 03:25 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy