Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Xargs error: insufficient space for argument

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Xargs error: insufficient space for argument
# 1  
Old 04-26-2014
Xargs error: insufficient space for argument
I'm trying to crudely hack my way through some data processing.

I have file.txt with around 17,000 lines like this:
Code:
ACYPI002690-PA.aa.afa.afa.trim_phyml_tree_fullnames_fullhomolog.txt 3 72 71
ACYPI002690-PA.aa.afa.afa.trim_phyml_tree_fullnames_fullhomolog.txt 97 111 71
ACYPI003779-PA.aa.afa.afa.trim_phyml_tree_fullnames_fullhomolog.txt 7 66 66

The first column corresponds to file names, the second to line numbers. I am running this code to generate the text of a command that I need to execute for each of these lines (I don't think the code matters for this question but just in case...).
Code:
awk '{print $1,$2}' file.txt | sed 's/\(A.*txt\) \([0-9][0-9]*\)/cat \1 | awk "NR==\2" | grep -Eo "comp[0-9]{1,6}_c[0-9]{1,2}" | sort | uniq | wc -l/

When I try to run these commands using xargs
Code:
awk '{ if ($4 >= 10) print $1,$2}' file.txt | head -5 | sed 's/\(A.*txt\) \([0-9][0-9]*\)/cat \1 | awk "NR==\2" | grep -Eo "comp[0-9]{1,6}_c[0-9]{1,2}" | sort | uniq | wc -l/' | xargs -0 -s 5000 bash -c

it works for a small number of lines (i.e. the head -5 term). But when I try to run it on the entire file I get the error:
Code:
awk '{print $1,$2}' file.txt | sed 's/\(A.*txt\) \([0-9][0-9]*\)/cat \1 | awk "NR==\2" | grep -Eo "comp[0-9]{1,6}_c[0-9]{1,2}" | sort | uniq | wc -l/' | xargs -0 -s 5000 bash -c
xargs: insufficient space for argument

I also tried saving the commands as a file (commands.txt) using the above script, then executing using a while loop
Code:
while read -r line; do command " $line"; done <commands.txt

But I get the error like this for each command:
Code:
-bash:  cat ACYPI002749-PA.aa.afa.afa.trim_phyml_tree_fullnames_fullhomolog.txt | awk "NR==3" | grep -Eo "comp[0-9]{1,6}_c[0-9]{1,2}" | sort | uniq | wc -l: command not found

Any ideas how I can get this done?

EDIT: I realized I can get this done using source but I'm still interested to know what's wrong with the above approaches.
Code:
source commands.txt

Last edited by pathunkathunk; 04-26-2014 at 01:41 AM.. Reason: solution found
# 2  
Old 04-26-2014
So you want this line:

Code:
ACYPI002690-PA.aa.afa.afa.trim_phyml_tree_fullnames_fullhomolog.txt 3 72 71

to run this command:

Code:
cat ACYPI002690-PA.aa.afa.afa.trim_phyml_tree_fullnames_fullhomolog.txt | awk "NR==3" | grep -Eo "comp[0-9]{1,6}_c[0-9]{1,2}" | sort | uniq | wc -l

If you have GNU Parallel you should be able to do:

Code:
cat file.txt | parallel --colsep '\s+' 'cat {1} | awk "NR=={2}" | grep -Eo "comp[0-9]{1,6}_c[0-9]{1,2}" | sort | uniq | wc -l'

Last edited by Scrutinizer; 04-26-2014 at 07:08 PM.. Reason: Added code tags, removed site reference.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Xargs: multiple commands to each argument

Hello. There is my one-liner to get subjects of potential spam mails sudo exiqgrep -bf "spamer@example.com" |cut -d' ' -f1 |xargs -I ~ sudo /usr/sbin/exim -Mvh ~ |grep 'Subject: ' I want to insert blank line after each iteration to make output more readable. I tried sudo exiqgrep -bf... (1 Reply)
Discussion started by: urello
1 Replies

2. Shell Programming and Scripting

Getopts - space in argument (OPTARG)

Hi, I want to capture space as well from the argument eg: script.ksh -m "Message from xyz" -e "email@xyz.com" script.ksh -m 'Message from xyz' -e 'email@xyz.com' I am parsing using getopts, but for option "m" OPTARG is returning only "Message". Please use code tags next time for... (9 Replies)
Discussion started by: tostay2003
9 Replies

3. UNIX for Advanced & Expert Users

sendmail error - 452 4.3.1 Insufficient system resources

Hi All, I am using svnnotify (which is available on net) script to send svn commit notification. Everything was working fine up to yesterday but from yesterday i started seeing error. 29CD01FCE44 7228 Thu Apr 5 11:11:51 apache@vm0001.host.com (host mail.mailserver.com said: 452... (6 Replies)
Discussion started by: vishal_vsh1
6 Replies

4. UNIX for Advanced & Expert Users

Error:--test: argument expected--Even though i give an argument.

Hi All, I am running the script VBoxManage list vms |sed 's/"//g' | cut -d " " -f1 > har1out.mytxt result=`cat har1out.mytxt | grep $1' echo $result echo $1 { if then echo pass else echo fail fi (2 Replies)
Discussion started by: harsha85
2 Replies

5. Programming

Fortran runtime error: Insufficient data descriptors in format after reversion

Hello, I have some Fortran code that is providing the following error: At line 1165 of lapc_shells_2.f Fortran runtime error: Insufficient data descriptors in format after reversion Here are the lines just above and including 1165: WRITE (*,"('ATTEMPTING TO READ PLATE... (1 Reply)
Discussion started by: jm4smtddd
1 Replies

6. UNIX for Advanced & Expert Users

Issue with insufficient swap or memory space

Hi, When I execute one of my shellscript I am getting the below mentioned error message .This application takes 2input files which have the records counts 26463 and 1178046 exec(2): insufficient swap or memory available. exec(2): insufficient swap or memory available. exec(2): insufficient swap... (3 Replies)
Discussion started by: kavithakuttyk
3 Replies

7. Shell Programming and Scripting

Issue with insufficient swap or memory space

Hi, When I execute one of my shellscript I am getting the below mentioned error message .This application takes 2input files which have the records counts 26463 and 1178046 exec(2): insufficient swap or memory available. exec(2): insufficient swap or memory available. exec(2):... (2 Replies)
Discussion started by: kavithakuttyk
2 Replies

8. Solaris

sar : insufficient address space to load xxxx device records

Hello, i am using Solaris 10, The sar running in my system might be corrupted, but not sure why as there has been no updates to it ( to the best of my knowledge) and it was working fine until few days back. If i try to get sar reports using sar -o <filename> 60 180 there is no error but the... (2 Replies)
Discussion started by: nimi20
2 Replies

9. Shell Programming and Scripting

replacing single space in argument

I want to write a script which will check the arguments and if there is a single space(if 2 more more space in a row , then do not touch), replace it with _ and then gather the argument so, program will be ran ./programname hi hello hi usa now hello hello so, inside of program,... (7 Replies)
Discussion started by: convenientstore
7 Replies

10. Solaris

insufficient metadevice database replicas ERROR

First I would like to thank this forum for assisting me in setting up my 1st sunbox. Could not have done it if it had not been for you guys and google :D I have mirrored my box and have SUCCESSFULLY tested booting from the rootdisk and rootmirror successfully. I am now looking at configuring... (2 Replies)
Discussion started by: mr_manny
2 Replies
Login or Register to Ask a Question