piping problem with xargs


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting piping problem with xargs
# 1  
Old 08-05-2011
piping problem with xargs

I'm trying to pipe the output from a command into another using xargs but is not getting what I want. Running this commands:

find . -name '33_cr*.rod' | xargs -n1 -t -i cut -f5 {} | sort -k1.3n | uniq | wc -l
give the following output:
cut -f5 ./33_cr22.rod
cut -f5 ./33_cr22.rod
...
9224236

What I want to see is the unique count number for each file instead of the the total count.

Any help is appreciated. Thank you.
# 2  
Old 08-05-2011
xargs format is wrong .


find . -name '33_cr*.rod' | xargs -n1 -t -i cut -f5
there should be no "{}" it is used only for exec with find command.



Refer to the address:Linux,C,C++,Shell,perl: xargs vs exec in find command

it may give you a better idea.

Smilie
# 3  
Old 08-05-2011
Try the following
Code:
$ find . -name '33_cr*.rod' | xargs -l  cut -f5  | sort -k1.3n | uniq | wc -l

# 4  
Old 08-05-2011
Quote:
Originally Posted by h@foorsa.biz
Try the following
Code:
$ find . -name '33_cr*.rod' | xargs -l  cut -f5  | sort -k1.3n | uniq | wc -l

The OP seems to want per-file counts. Beginning with the xargs stdout, that pipeline obliterates file boundaries and generates an aggregate result.

Regards,
Alister
# 5  
Old 08-06-2011
Consider using GNU Parallel instead:
Code:
 find . -name '33_cr*.rod' | parallel "echo -n {}': '; cut -f5 {} | sort -k1.3n | uniq | wc -l"

Watch the intro video to learn more: youtube. com/watch?v=OpaiGYxkSuQ
# 6  
Old 08-06-2011
I solved the problem with this:

for f in *.rod; do echo $f; cut -f5 $f | sort -k1.3n | uniq | wc -l; done

---------- Post updated at 02:36 PM ---------- Previous update was at 02:17 PM ----------

Quote:
Originally Posted by tange
Consider using GNU Parallel instead:
Code:
 find . -name '33_cr*.rod' | parallel "echo -n {}': '; cut -f5 {} | sort -k1.3n | uniq | wc -l"

Watch the intro video to learn more: youtube. com/watch?v=OpaiGYxkSuQ
Hi Tange,

I find the GNU Parallel interesting. I'm not exactly a computing person, can you clarify the following for me:

1. what is the difference between GNU parallel and multi-threaded process? For example, if my program allows multi-threading and I run 4 threads in my quart-core computer, am I right to assume that the GNU parallel is not going to make any difference?
2. If I already running some other process and still specify -j+0, what is going to happen?

Thank you.
# 7  
Old 08-09-2011
GNU Parallel makes it often easy to generate command lines:
Code:
parallel "echo {}; cut -f5 {} | sort -k1.3n | uniq | wc -l" ::: *.rod

It will also run the jobs in parallel. If you jobs are multithreaded and you see 100% utilization of your cores all the time then you will not see a speed up using GNU Parallel.

If, however, your jobs have I/O and therefore sometimes wait for data, then it might be faster to have more processes running, that can utilize the CPU when other processes are waiting.

GNU Parallel can be made to look at the load average and not start more jobs if the load average is above a certain limit. Default (-j+0) is to start 1 job per CPU core regardless of load average.

If you want multiple instances of GNU Parallel to communicate, and not have more than N jobs in total - no matter how many times you start parallel, you should look at 'sem': http://www.gnu.org/software/parallel/sem.html
This User Gave Thanks to tange For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Piping commands using xargs

Need help in piping commands using xargs I have several .tar.gz files that I need to list the folder content in a subdirectory. For example, a.tar.gz b.tar.gz c.tar.gz The following command works great for each .tar.gz file but it's a pain to run the tar command for each file. tar -tf... (11 Replies)
Discussion started by: april
11 Replies

2. Shell Programming and Scripting

Problem with xargs

I entered the following <zzz.list xargs showtell |more which does a echo "<<<<<<<<<<<<<<<<<<<<<<<<< $1 >>>>>>>>>>>>>>>>>>" head -20 $1 The file zzz.list contains 525 lines representing user scripts (1 per line), but only the first, 181st, and 399th lines were processed. What am I missing? TIA (2 Replies)
Discussion started by: wbport
2 Replies

3. Shell Programming and Scripting

Problem using xargs with rm

I am trying to call xargs, however rm is complaining find . -maxdepth 1 -name "*tests*" -print0 | xargs -0 rm rm: missing operand Try `rm --help' for more information. (3 Replies)
Discussion started by: kristinu
3 Replies

4. Homework & Coursework Questions

Unix Piping Problem

Hey guys. I'm very new to Unix. I'm pretty fluent in Java and C, but I have never actually used Unix for anything. I am in an Operating Systems course now and I have an assignment to write a piece of code that involves forks and piping. I'm stuck. 1. The problem statement, all variables and... (6 Replies)
Discussion started by: itsjimmy91
6 Replies

5. UNIX for Dummies Questions & Answers

[SOLVED] Piping Problem

Hey, I want to create a new file (devices) with the 39th and the 40th character of the line wich is in the array line and in the file drivers. But unfortunately my try doesn't work: sed -n '$linep' drivers | cut -c 39-40 | echo >>devices Perhaps one of you can help me. Thank you! emoly ... (0 Replies)
Discussion started by: emoly
0 Replies

6. Shell Programming and Scripting

xargs command problem

Hi I am trying to use "xargs" command to loop through each file, modify it and overwrite the old file with the modification but with the same file name. I thought it is easy but I just can't get it to work I tried the following I thought {} would give me the current file name, but it... (1 Reply)
Discussion started by: tiger66
1 Replies

7. Solaris

Piping results of 'ls' to 'find' using 'xargs'

I'm trying to get a count of all the files in a series of directories on a per directory basis. Directory structure is like (but with many more files): /dir1/subdir1/file1.txt /dir1/subdir1/file2.txt /dir1/subdir2/file1.txt /dir1/subdir2/file2.txt /dir2/subdir1/file1.txt... (4 Replies)
Discussion started by: MartynAbbott
4 Replies

8. Shell Programming and Scripting

xargs problem

Hi From the xargs man page (Solaris): ls $1 | xargs -I {} -t mv $1/{} $2/{} This would move all the files in directory $1 to directory $2 Problem 1: In a shell script if I want to move files in d1 to d2 dir1=~/d1 dir2=~/d2 ls $dir1 | xargs -I {} -t mv $dir1/{} $dir2/{} does not... (3 Replies)
Discussion started by: encrypted
3 Replies

9. UNIX for Dummies Questions & Answers

use of xargs and prune piping with find command.

Can anyone interpret and tell me the way the below command works? find * -name "*${msgType}" -mtime +${archiveDays} -prune -type f -print 2>/dev/null | xargs rm -f 2> /dev/null Please tell me the usage of prune and xargs in the above command? Looking forward your reply. Thanks in... (1 Reply)
Discussion started by: venkatesht
1 Replies

10. Shell Programming and Scripting

problem with exit code when piping

i am writing a script to perform some mysqldumps and gzip them. The problem I am running into is that if the user specifies a database that doesn't exist, the error the mysql engine produces is still piped into gzip, and the exit code returned is 0. If I don't pipe into gzip, an exit code... (4 Replies)
Discussion started by: bitoffish
4 Replies
Login or Register to Ask a Question