for i in `nawk '/From/ {print $3}' syslogs | uniq`; do grep $i syslogs >> $i
More points:
1) This is not a good command structure for an open ended list because it leaves the shell with a potentially massive "for" command to process.
We can turn the "for" into a "while" and remove that issue.
2) We need to "sort" the node selection or "uniq" will not work properly and we will be making selections multiple times.
3) We can replace ">>" with ">" because each file is now only written once.
4) The "cat" to a pipeline was harmless and can be quicker at getting records into "awk" than getting "awk" to read the file direct. I've left it out because it is the convention of this board.
Overall this should be quicker and more importantly should not give multiple repeat blocks of data because of the missing "sort".
Code:
nawk '/From/ {print $3}' syslogs | sort | uniq | while read node
do
grep "${node}" syslogs > "${node}"
done
Last edited by methyl; 07-09-2010 at 01:36 PM..
Reason: typos
Hello ,
I am trying to print the footer of evry file in the given directory with xargs command like follows
ls -1 | xargs -I {} gzcat {} | tail -1
now problem with this is only last file foooter is getting printed as " | tail -1 " is getting executed for the last file.
I know this can... (4 Replies)
I am trying to write a script that will ssh into a remote machine and recurse through a specified directory, find mp3 files which may be two or three directories deep (think iTunes: music/artist/album/song.mp3), and scp them back to the machine running the script. The script should also maintain... (3 Replies)
Hello,
I have multiple functions within a shell script. eg. function_database_backup, unix_tar_creation, etc.
I would like to run these functions in parallel, as each is independent of the other.
If these were simple commands, I could have probably run each of the commands in background.
... (1 Reply)
Hi,
I am using ksh , i have requirement to run 4 functions in background , 4 functions call are available in a case that case is also in function, i need to execute 1st function it should run in background and return to case and next i will call 2nd function it should run in background and... (8 Replies)
Hello. There is my one-liner to get subjects of potential spam mails
sudo exiqgrep -bf "spamer@example.com" |cut -d' ' -f1 |xargs -I ~ sudo /usr/sbin/exim -Mvh ~ |grep 'Subject: '
I want to insert blank line after each iteration to make output more readable. I tried
sudo exiqgrep -bf... (1 Reply)
I have a script (ksh) which tries to run a function in parallel for performance gains. I am also trying to limit the number of parallel child processes to avoid overloading the system by using a variable to count triggered processes and waiting for completion e.g.
do_something ()
{
...
}
... (9 Replies)
Hi,
I am a beginner in python programming. In my python script have a main function which calls several other functions. The main function gets its input by reading lines from a input text file. I call the main function for every line in input text file through a loop.
def main(line):
var1... (6 Replies)
Hello,
I have a requirement to apply hashing algorithm on flat file on one or more columns dynamically based on header
sample input file
ID|NAME|AGE|GENDER
10|ABC|30|M
20|DEF|20|F
say if i want multiple columns based on the header example id,name or id,age or name,gender and hash and... (13 Replies)
I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when
| xargs sh is added I get the error. I tried adding | xargs... (4 Replies)
Discussion started by: cmccabe
4 Replies
LEARN ABOUT LINUX
uniq
UNIQ(1) User Commands UNIQ(1)NAME
uniq - report or omit repeated lines
SYNOPSIS
uniq [OPTION]... [INPUT [OUTPUT]]
DESCRIPTION
Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).
With no options, matching lines are merged to the first occurrence.
Mandatory arguments to long options are mandatory for short options too.
-c, --count
prefix lines by the number of occurrences
-d, --repeated
only print duplicate lines
-D, --all-repeated[=delimit-method]
print all duplicate lines delimit-method={none(default),prepend,separate} Delimiting is done with blank lines
-f, --skip-fields=N
avoid comparing the first N fields
-i, --ignore-case
ignore differences in case when comparing
-s, --skip-chars=N
avoid comparing the first N characters
-u, --unique
only print unique lines
-z, --zero-terminated
end lines with 0 byte, not newline
-w, --check-chars=N
compare no more than N characters in lines
--help display this help and exit
--version
output version information and exit
A field is a run of blanks (usually spaces and/or TABs), then non-blank characters. Fields are skipped before chars.
Note: 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use `sort -u' without
`uniq'. Also, comparisons honor the rules specified by `LC_COLLATE'.
AUTHOR
Written by Richard M. Stallman and David MacKenzie.
REPORTING BUGS
Report uniq bugs to bug-coreutils@gnu.org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
Report uniq translation bugs to <http://translationproject.org/team/>
COPYRIGHT
Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO comm(1), join(1)
The full documentation for uniq is maintained as a Texinfo manual. If the info and uniq programs are properly installed at your site, the
command
info coreutils 'uniq invocation'
should give you access to the complete manual.
GNU coreutils 8.5 February 2011 UNIQ(1)