testfile has various repetitions of "AAA:BBB\n".
Like other posters said, if you can use cut for you problem you should choose it instead of awk, but there are situations where cut just isn't enough.
Code:
# du -h file1
207M file1
# time cut -d":" -f1 file1 > /dev/null
real 0m46.075s
user 0m43.075s
sys 0m0.396s
# time awk -F":" '{print $1}' file1 > /dev/null
real 0m41.344s
user 0m38.422s
sys 0m0.324s
# time cut -d":" -f1 file1 > /dev/null
real 0m45.266s
user 0m43.055s
sys 0m0.328s
# time awk -F":" '{print $1}' file1 > /dev/null
real 0m41.220s
user 0m38.358s
sys 0m0.452s
But why?? The results were similar regardless whether the first or third field was printed, whether another delimiter was chosen, although awk did slow down with larger fields. (50% longer when '-' was used as the delimiter -- meaning the fields were longer).
It could be that GNU coreutils' cut is not very optimized. (GNU awk was used here.)
So when is cut shorter? Perhaps it's the parsing routines that make awk slower sometimes. To test this, I took 10 lines of my HTTP access file and timed two runs each of processing this same file 8000 times inside a bash-while loop. One run used field 1, the second run used field 3.
cat to /dev/null
cut to /dev/null
awk to /dev/null
For cut and awk, the cat was part of the pipeline. Thus we should be able to subtract the first time from the other two. Here's what I got:
cat: 16.1 (real) 1.9s (user)
cut: 29.3s (real) 6.5s (user)
awk: 28.9s (real) 8.0s (user)
The idea was to see if cut was better on smaller files. It is relatively better, but even for short files, GNU awk takes less processing time than GNU cut! However, cut would appear to take fewer user-clockticks, if that's any concern to anyone for accounting reasons.
To sum, cut isn't as sharp as it's awkward cousin.
I have nginx web server logs with all requests that were made and I'm filtering them by date and time.
Each line has the following structure:
127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br)
These text files are... (21 Replies)
I have the below command which is referring a large file and it is taking 3 hours to run. Can something be done to make this command faster.
awk -F ',' '{OFS=","}{ if ($13 == "9999") print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12 }' ${NLAP_TEMP}/hist1.out|sort -T ${NLAP_TEMP} |uniq>... (13 Replies)
I have script like below, who is picking number from one file and and searching in another file, and printing output.
Bu is is very slow to be run on huge file.can we modify it with awk
#! /bin/ksh
while read line1
do
echo "$line1"
a=`echo $line1`
if
then
echo "$num"
cat file1|nawk... (6 Replies)
Hi,
I have a large number of input files with two columns of numbers.
For example:
83 1453
99 3255
99 8482
99 7372
83 175
I only wish to retain lines where the numbers fullfil two requirements. E.g:
=83
1000<=<=2000
To do this I use the following... (10 Replies)
awk "/May 23, 2012 /,0" /var/tmp/datafile
the above command pulls out information in the datafile. the information it pulls is from the date specified to the end of the file.
now, how can i make this faster if the datafile is huge? even if it wasn't huge, i feel there's a better/faster way to... (8 Replies)
Hi,
I have a script below for extracting xml from a file.
for i in *.txt
do
echo $i
awk '/<.*/ , /.*<\/.*>/' "$i" | tr -d '\n'
echo -ne '\n'
done
.
I read about using multi threading to speed up the script.
I do not know much about it but read it on this forum.
Is it a... (21 Replies)
Hi,
A datafile containing lines such as below needs to be split:
500000000000932491683600000000000000000000000000016800000GS0000000000932491683600*HOME
I need to get the 2-5, 11-20, and 35-40 characters and I can do it via cut command.
cut -c 2-5 file > temp1.txt
cut -c 11-20 file >... (9 Replies)
hii everyone ,
i have a file in which i have line numbers.. file name is file1.txt
aa bb cc "12" qw
xx yy zz "23" we
bb qw we "123249" jh
here 12,23,123249. is the line number
now according to this line numbers we have to print lines from other file named... (11 Replies)
Hi I'm new to unix programming so struggling with something thats probably simple to many of you
I have data files of the format :
ID, date, value1, value2, blank
on each line either value1 or value2 will be zero. I need my output file to contain
ID, date, non-zero value
The input... (3 Replies)
Hi, I'm very new to scripting.
grep $s $filename | awk '{print $2}' | cut -c 1-8
How can I optimize this using a single awk?
I tried:
awk '/$s/ {print $2}' $filename | cut -c 1-8
However didn't work, I think the awk is not recognizing $s and the verbal is something else. (6 Replies)
08-25-2008
