Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Argument list too long for date command

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Argument list too long for date command
# 1  
Old 09-16-2011
Argument list too long for date command
Dear Friends,

The following script processes a 14508 lines log file.

Code:
#!/bin/sh
while read line
do
     d=`sed 's/[0-9|.]* - [-|a-zA-Z]* \[\([0-9]*\/[a-zA-Z]*\/[0-9:]* [0-9+]*\)\] .*/\1/' | tr '/' ' ' | sed 's/\([.]*\):\([.]*\)/\1 \2/'`
     y=`date -d "${d}" "+%Y%m%d%H%M%S"`
     echo "${y}"
done

While running the above script, I am getting argument list long error. Can anyone help me to come out this error. I don't this issue if I split two processes(sed and date) in to different files.

Thanks,
Tamil Pamaran
Last edited by pludi; 09-17-2011 at 09:01 PM..
# 2  
Old 09-16-2011
Please provide a small sample of your log file with one or more date strings in it.
# 3  
Old 09-16-2011
fpmurphy,

Thanks for you response, part of the log file is given below,

Code:
213.64.56.208 - - [01/Jan/2003:08:18:42 +0100] "GET /scripts/..%%35c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:08:18:44 +0100] "GET /scripts/..%25%35%63../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:08:18:56 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:09:46:06 +0100] "GET /scripts/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:34 +0100] "GET /scripts/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:36 +0100] "GET /MSADC/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:39 +0100] "GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:42 +0100] "GET /d/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:47 +0100] "GET /scripts/..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:49 +0100] "GET /_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:52 +0100] "GET /_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.64.56.208 - - [01/Jan/2003:10:14:57 +0100] "GET /msadc/..%255c../..%255c../..%255c/..%c1%1c../..%c1%1c../..%c1%1c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.67.145.223 - - [01/Jan/2003:11:00:06 +0100] "HEAD / HTTP/1.0" 401 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET /scripts/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:15 +0100] "GET /MSADC/root.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET /d/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:16 +0100] "GET /scripts/..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET /_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:17 +0100] "GET /_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET /msadc/..%255c../..%255c../..%255c/..%c1%1c../..%c1%1c../..%c1%1c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET /scripts/..%c1%1c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:18 +0100] "GET /scripts/..%c0%2f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""


I have modified the code in the following manner and its works fine now,

Code:
while read data
do
     datestring=`echo "${data}" | sed 's/[0-9|.]* - [-|a-zA-Z]* \[\([0-9]*\/[a-zA-Z]*\/[0-9:]* [0-9+]*\)\] .*/\1/' | tr '/' ' ' | sed 's/\([.]*\):\([.]*\)/\1 \2/'`
     date -d "${datestring}" "+%Y%m%d%H%M%S" > /dev/null
     done

Appreciate if you can explain what was the issue.Smilie

-Tamil Pamaran
Last edited by pludi; 09-17-2011 at 09:01 PM..
# 4  
Old 09-17-2011
Your second version processes each line of input one by one, but your first does not !

Because of this line ..

Code:
d=`sed 's/[0-9|.]* - [-|a-zA-Z]* \[\([0-9]*\/[a-zA-Z]*\/[0-9:]* [0-9+]*\)\] .*/\1/' | tr '/' ' ' | sed 's/\([.]*\):\([.]*\)/\1 \2/'`

Where is the input for the 1st sed command executed in the above line ??


There is no so it tries reading from STDIN.. Guess you have redirected the input so the remaining the sed reads... when all input are read and send to date ... blah blah blah !
This User Gave Thanks to thegeek For This Post:
tamil.pamaran
# 5  
Old 09-18-2011
MySQL
thegeek,

Your guess is right. Thanks for your explaination.

Cheers,
Tamil Pamaran
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Argument list too long

Hi Team, Here's the situation. I have approximately 300000 to 500000 jpg files in /appl/abcd/work_dir mv /appl/abcd/work_dir /appl/abcd/process_dir The above move command will work if the jpg files count is close to 50000 (not sure). If the count is less this mv command holds good. But if... (14 Replies)
Discussion started by: kmanivan82
14 Replies

2. UNIX for Advanced & Expert Users

Argument list too long w/ sed

Hi all, I am using GNU sed (named gsed under macports) in OSX. I have a directory with a series of files named pool_01.jpg through pool_78802.jpg. I am trying to use this command to rename the files to their checksum + extension. md5sum * | gsed -e 's/\(*\) \(.*\(\..*\)\)$/mv -v \2 \1\3/e' ... (3 Replies)
Discussion started by: openthomas
3 Replies

3. Shell Programming and Scripting

mv : Argument list too long

Hi I am using find command -- find "directory1" -type f | xargs -i mv {} "directory2" to avoid above argument list too long problem. But, issue i am facing is directory1 is having subdirectories due to this i am facing directory traversal problem as i dont want to traverse subdirectories... (9 Replies)
Discussion started by: VSom007
9 Replies

4. Shell Programming and Scripting

grep : Argument list too long

Hi, i am having some trouble with the below command, can some one suggest me the better way to do it. grep -l 'ReturnCode=1' `find $Log -newer /tmp/Failed.tmp -print | xargs ls -ld | egrep SUB | egrep -ve 'MTP' -ve 'ABC' -ve 'DEF' -ve 'JKL' -ve 'XYZ' | awk '{print $9}'` > $Home1 Its... (2 Replies)
Discussion started by: Prateek007
2 Replies

5. Shell Programming and Scripting

Argument list too long!!

Dear Experts, I have a list of 10K files in a directory. I am not able to execute any commands lile ls -lrt, awk, sed, mv, etc........ I wanna execute below command and get the output. How can I achieve it?? Pls help. root# awk -F'|' '$1 == 1' file_20120710* | wc -l /bin/awk: Argument list... (2 Replies)
Discussion started by: Naga06
2 Replies

6. Shell Programming and Scripting

Argument list too long problem

I have a huge set of files (with extension .common) in my directory around 2 million. When I run this script on my Linux with BASH, I get /bin/awk: Argument list too long awk -F'\t' ' NR == FNR { a=NR } NR != FNR { sub(".common", "", FILENAME) print a, FILENAME, $1 } '... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

7. UNIX for Dummies Questions & Answers

Argument list too long for Sed command

Hi guys Following command results in sed -i 's/#/\\#/g' /home/test/sqlstents* -bash: /bin/sed: Argument list too long Please help me solve it.. is there any other way i can do this?.. thanks (4 Replies)
Discussion started by: depakjan
4 Replies

8. Shell Programming and Scripting

Argument too long list error

I have a wrote a script which consits of the below line.. Below of this script I'm getting this error "ksh: /usr/bin/ls: arg list too long" The line is log_file_time=`ssh -i $HOME/.ssh/id_rsa -q $i ls -lrt /bp/karthik/test/data/log/$abc*|tail -1|awk '{print $8}'` And $abc alias is as "p |... (1 Reply)
Discussion started by: 22karthikreddy
1 Replies

9. UNIX for Dummies Questions & Answers

Argument list too long - SSH

Hi I executed the code for file in `ls pdb*.ent` do new_name=`echo $file | sed 's/^pdb//;s/.ent/.txt/'` mv $file $new_name done Its giving error at ' ls pdb*.ent' argument list too long i have around 150000 entries please help Thank you (6 Replies)
Discussion started by: empyrean
6 Replies

10. Shell Programming and Scripting

command find returned bash: /usr/bin/find: Argument list too long

Hello, I create a file touch 1201093003 fichcomp and inside a repertory (which hava a lot of files) I want to list all files created before this file : find *.* \! -maxdepth 1 - newer fichcomp but this command returned bash: /usr/bin/find: Argument list too long but i make a filter all... (1 Reply)
Discussion started by: yacsil
1 Replies
Login or Register to Ask a Question