Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Argument list too long problem
Top Forums Shell Programming and Scripting Argument list too long problem Post 302553124 by shoaibjameel123 on Tuesday 6th of September 2011 10:27:06 PM
Old 09-06-2011
Argument list too long problem
I have a huge set of files (with extension .common) in my directory around 2 million. When I run this script on my Linux with BASH, I get /bin/awk: Argument list too long


Code:
awk -F'\t' '
  NR == FNR { a[$1]=NR }
  NR != FNR {
    sub(".common", "", FILENAME)
    print a[$2], FILENAME,  $1
  }
' dictionary.dit *.common

Any idea how can I fix this? One way is to put the non-overlapping files in two or three directories and then run the above script but I am looking for a better solution.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Argument list too long - Shell error

Trying to tar specific files from a directory causes problems when the number of files is too large. ls ~/logs | wc -l 5928 In the logs directory - I have 5928 files If I want to include all files with today's date - I run the following command tar cf ~/archive/LoadLogs_20060302.tar... (8 Replies)
Discussion started by: dad5119
8 Replies

2. UNIX for Dummies Questions & Answers

Argument list too long - SSH

Hi I executed the code for file in `ls pdb*.ent` do new_name=`echo $file | sed 's/^pdb//;s/.ent/.txt/'` mv $file $new_name done Its giving error at ' ls pdb*.ent' argument list too long i have around 150000 entries please help Thank you (6 Replies)
Discussion started by: empyrean
6 Replies

3. Shell Programming and Scripting

Argument too long list error

I have a wrote a script which consits of the below line.. Below of this script I'm getting this error "ksh: /usr/bin/ls: arg list too long" The line is log_file_time=`ssh -i $HOME/.ssh/id_rsa -q $i ls -lrt /bp/karthik/test/data/log/$abc*|tail -1|awk '{print $8}'` And $abc alias is as "p |... (1 Reply)
Discussion started by: 22karthikreddy
1 Replies

4. UNIX for Dummies Questions & Answers

Argument list too long for Sed command

Hi guys Following command results in sed -i 's/#/\\#/g' /home/test/sqlstents* -bash: /bin/sed: Argument list too long Please help me solve it.. is there any other way i can do this?.. thanks (4 Replies)
Discussion started by: depakjan
4 Replies

5. Shell Programming and Scripting

Argument list too long for date command

Dear Friends, The following script processes a 14508 lines log file. #!/bin/sh while read line do d=`sed 's/* - * \*\/*\/* *\)\] .*/\1/' | tr '/' ' ' | sed 's/\(*\):\(*\)/\1 \2/'` y=`date -d "${d}" "+%Y%m%d%H%M%S"` echo "${y}" done While running the above script, I am... (4 Replies)
Discussion started by: tamil.pamaran
4 Replies

6. Shell Programming and Scripting

Argument list too long!!

Dear Experts, I have a list of 10K files in a directory. I am not able to execute any commands lile ls -lrt, awk, sed, mv, etc........ I wanna execute below command and get the output. How can I achieve it?? Pls help. root# awk -F'|' '$1 == 1' file_20120710* | wc -l /bin/awk: Argument list... (2 Replies)
Discussion started by: Naga06
2 Replies

7. Shell Programming and Scripting

grep : Argument list too long

Hi, i am having some trouble with the below command, can some one suggest me the better way to do it. grep -l 'ReturnCode=1' `find $Log -newer /tmp/Failed.tmp -print | xargs ls -ld | egrep SUB | egrep -ve 'MTP' -ve 'ABC' -ve 'DEF' -ve 'JKL' -ve 'XYZ' | awk '{print $9}'` > $Home1 Its... (2 Replies)
Discussion started by: Prateek007
2 Replies

8. Shell Programming and Scripting

mv : Argument list too long

Hi I am using find command -- find "directory1" -type f | xargs -i mv {} "directory2" to avoid above argument list too long problem. But, issue i am facing is directory1 is having subdirectories due to this i am facing directory traversal problem as i dont want to traverse subdirectories... (9 Replies)
Discussion started by: VSom007
9 Replies

9. UNIX for Advanced & Expert Users

Argument list too long w/ sed

Hi all, I am using GNU sed (named gsed under macports) in OSX. I have a directory with a series of files named pool_01.jpg through pool_78802.jpg. I am trying to use this command to rename the files to their checksum + extension. md5sum * | gsed -e 's/\(*\) \(.*\(\..*\)\)$/mv -v \2 \1\3/e' ... (3 Replies)
Discussion started by: openthomas
3 Replies

10. Shell Programming and Scripting

Argument list too long

Hi Team, Here's the situation. I have approximately 300000 to 500000 jpg files in /appl/abcd/work_dir mv /appl/abcd/work_dir /appl/abcd/process_dir The above move command will work if the jpg files count is close to 50000 (not sure). If the count is less this mv command holds good. But if... (14 Replies)
Discussion started by: kmanivan82
14 Replies

LEARN ABOUT OSF1

itruns

itruns(8)						      System Manager's Manual							 itruns(8)

NAME

       itruns - set up files for use by the it command

SYNOPSIS

       /sbin/itruns file [file...] in run-state [, file [file...] in run-state] [,...]

DESCRIPTION

       The  itruns  program  sets  up scripts to be run by it(8).  (These programs are typically used within system configuration and installation
       scripts.)  The itruns command searches for /sbin/it.d/bin/file. If itruns finds the file, it  creates  one  or  more  symbolic  links  from
       /sbin/it.d/run-state.d/file to /sbin/it.d/bin/file.

       The  run-state  argument can be a single run state, or it can be two run states separated by either the word or or the word and. If two run
       states are specified with the or keyword, itruns checks for (and creates, if necessary) a directory whose name combines both run states and
       places  a link in that directory so that the file will be executed when the it program detects either state.  You can collapse two or-sepa-
       rated states into a single expression; for example, 2 or 3 can be expressed as 23. If the and keyword is used, itruns checks for (and  cre-
       ates, if necessary) two separate directories and places a link in each of them so that either state triggers execution of the file indepen-
       dently of the other state.

       If two or more files are specified for a single run-state argument, links are created for all of the files as indicated	by  the  run-state
       argument.

       If  two or more combinations of files and run states are specified as comma-separated expressions, itruns treats each combination individu-
       ally.

RESTRICTIONS

       The itruns command must be executed from the root directory (/). The command finds only files that are located in the /sbin/it.d/bin direc-
       tory.

EXAMPLES

       This  example  sets  up	the doconfig script to be run when the system is booted to run state 2: itruns doconfig in 2 Either of the command
       lines in this example sets up the doconfig and getdate scripts to be run when the system is booted to run state 2 or to run state 3: itruns
       doconfig  getdate  in  2 or 3 itruns doconfig getdate in 23 This example sets up the doconfig script to be run when the system is booted to
       run state 2 and again when the system comes to run state 3, the getdate script to be run only in run state 2, and the getsysname script	to
       be run only in run state 3: itruns doconfig in 2 and 3, getdate in 2, getsysname in 3

SEE ALSO

       Commands: it(8)

																	 itruns(8)
All times are GMT -4. The time now is 05:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy