Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

sed cannot execute [Argument list too long]

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sed cannot execute [Argument list too long]
# 1  
Old 04-15-2014
sed cannot execute [Argument list too long]
Hi All,

This question has been asked many times, but my problem is slightly different.

In my shell script i am connecting to oracle database and loading the results to .dat file. This .dat file is later used to create to .xls file[to be sent to the users] Some times the size of .dat file becomes more than 120000 bytes. In these cases, the sed and other commands used in the script fails with cannot execute [Argument list too long] error.

I tried splitting the .dat file to smaller files, but i am not even able to split the file and it again throws me the same [Argument list too long] error.

The problem is not just with sed, but all commands used in the shell script.

My piece of code :
Code:
cat tgt_type_${tmp_file_suffix}i.dat | while read line
do
 
export FETCH=`sqlplus -silent ${USERNAME}/${PASS}@${DATABASE} <<EOF
whenever sqlerror exit 1
whenever oserror exit 1
set pagesize 2000
set linesize 2000
set underline off
set feedback off
set heading off
set trim on
spool missing_$line_${tmp_file_suffix}i.dat 
/*Oracle sql query comes here */;
spool off
EOF`
fi
 
sed  /^$/d  *.dat
cat missing_$line_${tmp_file_suffix}i.dat | while read line
do
#
#coding
#coding
#
done

done

# 2  
Old 04-15-2014
What does ls -l *.dat return? The same error? If so, that's the number of files, not their size - look at using find and possibly xargs. E.g. something like:
Code:
find . -type f -name "*.dat" | xargs sed /^$/d

Incindetally, sed /^$/d *.dat won't do anything to the actual files (unless you've aliased sed). Is that your intention?
# 3  
Old 04-15-2014
Because of the kernel ARG_MAX value is limited and asterisk character expands the all matching files,directory,links or any other forms.
Therefore your bash(shell) command argument buffer's ( commandline length ) is fill and and then overflow.

You can use
* for loop
* while loop
* or any scriptable methods

* split the arguments to meaning values
Code:
ls -1 tgt_type_${tmp_file_*}.dat

* xargs (to split to the arguments with find is one good method )

* ls (use the folder hierarchy with asteriks )
Code:
ls -1 yourfolder/

* find (use the folder hierarchy without asteriks )
Code:
find /yourfolder -name '*'


For example :
Code:
while read myfile ; do sed '/^$/d' $myfile ; done | find /yourdolfer -name '*.dat'|xargs -n1

# (use the while loop and read the arguments one by one (-n1) )
Code:
find /yourdolfer -name '*.dat' -print0 | xargs -0 -n5 sed '/^$/d'

# (use the spaces instead of newlines with xargs and use getdents to get the file entries with find)


and consider the @CarloM 's notice..
your sed command's default output is your stdout (usually your screen tty,pts,console,...) ,
therefore you must redirect to another file for save the changes or use the '-i' parameter , if you have gnu sed for save to your file same time.


regards
ygemici
# 4  
Old 04-16-2014
Hi ygemici, I am trying to implement your suggestions and i have made couple of changes to my script to use while loop and read from file.
But i am stuck in the below section:
Code:
cat missing_$line_${tmp_file_suffix}i.dat | while read line
do

echo "<Row>" >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$1F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$2F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$3F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$4F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$5F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$6F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$7F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$8F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$9F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$10F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$11F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$12F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$13F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo "</Row>" >> excel_CT_${tmp_file_suffix}i.dat
done
echo "</Table>" >> excel_${tmp_file_suffix}i.dat
echo "</Worksheet>" >> excel_${tmp_file_suffix}i.dat
done
echo "</Workbook>" >> excel_${tmp_file_suffix}i.dat

the above section basically echoes the contents of file and sets up the excel file to be created. Any suggestions on this?

Regards,
# 5  
Old 04-16-2014
Quote:
Originally Posted by galaxy_rocky
Hi ygemici, I am trying to implement your suggestions and i have made couple of changes to my script to use while loop and read from file.
But i am stuck in the below section:
Code:
cat missing_$line_${tmp_file_suffix}i.dat | while read line
do

echo "<Row>" >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$1F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$2F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$3F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$4F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$5F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$6F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$7F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$8F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$9F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$10F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$11F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$12F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo $line | awk -F';' '{print("<Cell><Data ss:Type=\"String\">"$13F"</Data></Cell>")}' >> excel_${tmp_file_suffix}i.dat
echo "</Row>" >> excel_CT_${tmp_file_suffix}i.dat
done
echo "</Table>" >> excel_${tmp_file_suffix}i.dat
echo "</Worksheet>" >> excel_${tmp_file_suffix}i.dat
done
echo "</Workbook>" >> excel_${tmp_file_suffix}i.dat

the above section basically echoes the contents of file and sets up the excel file to be created. Any suggestions on this?

Regards,
Hi galaxy_rocky,
if you can write the input file (or sample) and desired output file maybe i can help more Smilie

firstly i dont understand exactly what is the desired output ? (what is your $1F , is it fields that in te your input lines plus "F" )

i try some like below.
Code:
# cat missing.dat
test1
test2;test3
test4;test5;test6

Code:
# awk -F';' -vi=1 'BEGIN{print "<Row>"}{print "<Cell><Data ss:Type=\"String\">"$(i++)"F</Data></Cell>"}
END{print "</Row>\n</Table>\n</Workbook>"}' missing.dat
<Row>
<Cell><Data ss:Type="String">test1F</Data></Cell>
<Cell><Data ss:Type="String">test3F</Data></Cell>
<Cell><Data ss:Type="String">test6F</Data></Cell>
</Row>
</Table>
</Workbook>

# 6  
Old 04-16-2014
Hi ygemici,

$1F is the first field from the first line of the input file. Similarly $2F is the the second field from the first line of input file. This should loop for all the lines in the input file. Totally there are 13 fields in the input file. That is the reason i am using 13 awk commands!

My sample input file looks like below:

Code:
SG0010001;TRANSACTION;24:.ENTRY;31-MAR-14;SG0010001;123456789.020001;TRANSACTION;123;Metal XYZ Xy;N/A;N/A;N/A;CD_VALUE_TST;
SG0010001;TRANSACTION;24:.ENTRY;31-MAR-14;SG0010001;789456123.020001;TRANSACTION;123;Metal XYZ Xy;N/A;N/A;N/A;CD_VALUE_TST;
SG0010001;TRANSACTION;24:.ENTRY;31-MAR-14;SG0010001;147852369.040001;TRANSACTION;123;Metal XYZ Xy;N/A;N/A;N/A;CD_VALUE_TST;
SG0010001;TRANSACTION;24:.ENTRY;31-MAR-14;SG0010001;159753456.020001;TRANSACTION;123;Metal XYZ Xy;N/A;N/A;N/A;CD_VALUE_TST;

And my sample expected output is :

Code:
<Worksheet ss:Name="CD_VALUE_TST">
<Table>
<Row>
<Cell><Data ss:Type="String">EC</Data></Cell>
<Cell><Data ss:Type="String">TARGET_TRANSLATION</Data></Cell>
<Cell><Data ss:Type="String">SOURCE_FILE_NAME</Data></Cell>
<Cell><Data ss:Type="String">SOURCE_EFF_FROM</Data></Cell>
<Cell><Data ss:Type="String">CO_CODE</Data></Cell>
<Cell><Data ss:Type="String">SOURCE_ID</Data></Cell>
<Cell><Data ss:Type="String">SOURCE_CODE_SET_1</Data></Cell>
<Cell><Data ss:Type="String">SOURCE_FIELD_1</Data></Cell>
<Cell><Data ss:Type="String">SOURCE_FIELD_1_DESC</Data></Cell>
<Cell><Data ss:Type="String">SOURCE_CODE_SET_2</Data></Cell>
<Cell><Data ss:Type="String">SOURCE_FIELD_2</Data></Cell>
<Cell><Data ss:Type="String">SOURCE_FIELD_2_DESC</Data></Cell>
<Cell><Data ss:Type="String">TARGET_CODE_SET</Data></Cell>
</Row>
<Row>
<Cell><Data ss:Type="String">SG0010001</Data></Cell>
<Cell><Data ss:Type="String">TRANSACTION</Data></Cell>
<Cell><Data ss:Type="String">24:.ENTRY</Data></Cell>
<Cell><Data ss:Type="String">31-MAR-14</Data></Cell>
<Cell><Data ss:Type="String">SG0010001</Data></Cell>
<Cell><Data ss:Type="String">123456789.020001</Data></Cell>
<Cell><Data ss:Type="String">TRANSACTION</Data></Cell>
<Cell><Data ss:Type="String">123</Data></Cell>
<Cell><Data ss:Type="String">Metal XYZ Xy</Data></Cell>
<Cell><Data ss:Type="String">N/A</Data></Cell>
<Cell><Data ss:Type="String">N/A</Data></Cell>
<Cell><Data ss:Type="String">N/A</Data></Cell>
<Cell><Data ss:Type="String">CD_VALUE_TST</Data></Cell>
</Row>
<Row>
<Cell><Data ss:Type="String">SG0010001</Data></Cell>
<Cell><Data ss:Type="String">TRANSACTION</Data></Cell>
<Cell><Data ss:Type="String">24:.ENTRY</Data></Cell>
<Cell><Data ss:Type="String">31-MAR-14</Data></Cell>
<Cell><Data ss:Type="String">SG0010001</Data></Cell>
<Cell><Data ss:Type="String">789456123.020001</Data></Cell>
<Cell><Data ss:Type="String">TRANSACTION</Data></Cell>
<Cell><Data ss:Type="String">123</Data></Cell>
<Cell><Data ss:Type="String">Metal XYZ Xy</Data></Cell>
<Cell><Data ss:Type="String">N/A</Data></Cell>
<Cell><Data ss:Type="String">N/A</Data></Cell>
<Cell><Data ss:Type="String">N/A</Data></Cell>
<Cell><Data ss:Type="String">CD_VALUE_TST</Data></Cell>
</Row>
</Table>
</Worksheet>

Basically the fields in the input value are seperated by ";". Based on the number of records in the input file, there should be corresponing entries in the output file.
Please help on this issue.

Regards,
Last edited by galaxy_rocky; 04-16-2014 at 07:56 AM.. Reason: code tag
# 7  
Old 04-16-2014
If every field is treated the same then you can do the Cell tags fairly simply in one awk command. Something like:
Code:
echo "$line" | awk -F';' '{ for (i=1;i<NF;i++) { print "<Cell><Data ss:Type=\"String\">" $i "</Data></Cell>"}}' >> excel_${tmp_file_suffix}i.dat

EDIT: Actually, you can do away with the while as well:
Code:
awk -F';' '{ printf "<Row>\n"; for (i=1;i<NF;i++) { print " <Cell><Data ss:Type=\"String\">" $i "</Data></Cell>"} printf "</Row>\n"}' missing_$line_${tmp_file_suffix}i.dat >> excel_${tmp_file_suffix}i.dat

This User Gave Thanks to CarloM For This Post:
galaxy_rocky
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Argument list too long

Hi Team, Here's the situation. I have approximately 300000 to 500000 jpg files in /appl/abcd/work_dir mv /appl/abcd/work_dir /appl/abcd/process_dir The above move command will work if the jpg files count is close to 50000 (not sure). If the count is less this mv command holds good. But if... (14 Replies)
Discussion started by: kmanivan82
14 Replies

2. UNIX for Advanced & Expert Users

Argument list too long w/ sed

Hi all, I am using GNU sed (named gsed under macports) in OSX. I have a directory with a series of files named pool_01.jpg through pool_78802.jpg. I am trying to use this command to rename the files to their checksum + extension. md5sum * | gsed -e 's/\(*\) \(.*\(\..*\)\)$/mv -v \2 \1\3/e' ... (3 Replies)
Discussion started by: openthomas
3 Replies

3. Shell Programming and Scripting

mv : Argument list too long

Hi I am using find command -- find "directory1" -type f | xargs -i mv {} "directory2" to avoid above argument list too long problem. But, issue i am facing is directory1 is having subdirectories due to this i am facing directory traversal problem as i dont want to traverse subdirectories... (9 Replies)
Discussion started by: VSom007
9 Replies

4. Shell Programming and Scripting

grep : Argument list too long

Hi, i am having some trouble with the below command, can some one suggest me the better way to do it. grep -l 'ReturnCode=1' `find $Log -newer /tmp/Failed.tmp -print | xargs ls -ld | egrep SUB | egrep -ve 'MTP' -ve 'ABC' -ve 'DEF' -ve 'JKL' -ve 'XYZ' | awk '{print $9}'` > $Home1 Its... (2 Replies)
Discussion started by: Prateek007
2 Replies

5. Shell Programming and Scripting

Argument list too long!!

Dear Experts, I have a list of 10K files in a directory. I am not able to execute any commands lile ls -lrt, awk, sed, mv, etc........ I wanna execute below command and get the output. How can I achieve it?? Pls help. root# awk -F'|' '$1 == 1' file_20120710* | wc -l /bin/awk: Argument list... (2 Replies)
Discussion started by: Naga06
2 Replies

6. Shell Programming and Scripting

Argument list too long for date command

Dear Friends, The following script processes a 14508 lines log file. #!/bin/sh while read line do d=`sed 's/* - * \*\/*\/* *\)\] .*/\1/' | tr '/' ' ' | sed 's/\(*\):\(*\)/\1 \2/'` y=`date -d "${d}" "+%Y%m%d%H%M%S"` echo "${y}" done While running the above script, I am... (4 Replies)
Discussion started by: tamil.pamaran
4 Replies

7. Shell Programming and Scripting

Argument list too long problem

I have a huge set of files (with extension .common) in my directory around 2 million. When I run this script on my Linux with BASH, I get /bin/awk: Argument list too long awk -F'\t' ' NR == FNR { a=NR } NR != FNR { sub(".common", "", FILENAME) print a, FILENAME, $1 } '... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

8. UNIX for Dummies Questions & Answers

Argument list too long for Sed command

Hi guys Following command results in sed -i 's/#/\\#/g' /home/test/sqlstents* -bash: /bin/sed: Argument list too long Please help me solve it.. is there any other way i can do this?.. thanks (4 Replies)
Discussion started by: depakjan
4 Replies

9. Shell Programming and Scripting

Argument too long list error

I have a wrote a script which consits of the below line.. Below of this script I'm getting this error "ksh: /usr/bin/ls: arg list too long" The line is log_file_time=`ssh -i $HOME/.ssh/id_rsa -q $i ls -lrt /bp/karthik/test/data/log/$abc*|tail -1|awk '{print $8}'` And $abc alias is as "p |... (1 Reply)
Discussion started by: 22karthikreddy
1 Replies

10. UNIX for Dummies Questions & Answers

Argument list too long - SSH

Hi I executed the code for file in `ls pdb*.ent` do new_name=`echo $file | sed 's/^pdb//;s/.ent/.txt/'` mv $file $new_name done Its giving error at ' ls pdb*.ent' argument list too long i have around 150000 entries please help Thank you (6 Replies)
Discussion started by: empyrean
6 Replies
Login or Register to Ask a Question