count lines of file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting count lines of file
# 8  
Old 06-06-2007
Hi.
Quote:
Originally Posted by panknil
thanks for the reply but i have already tried with this command `wc -l`
its taking lot of time for 2GB size of file
Are you concerned about wall-clock time or CPU time? ... cheers, drl
# 9  
Old 06-06-2007
( removed - duplicate post ... cheers, drl )
# 10  
Old 06-06-2007
Some tests :
Code:
$ time sed -n '$=' big_file
2502607

réel    0m2,31s
util    0m1,79s
sys     0m0,50s
$ time wc -l <big_file
 2502607

réel    0m2,51s
util    0m2,33s
sys     0m0,18s
$ time awk '{x++}END{ print x}' big_file
2503305

réel    0m4,73s
util    0m4,52s
sys     0m0,20s
$ time awk 'END{print NR}' big_file
2503305

réel    0m3,21s
util    0m3,02s
sys     0m0,19s
$ time cat -n big_file | tail -1
2502607 

réel    0m7,86s
util    0m7,46s
sys     0m0,40s
$

Size of big_file : 72Mb

I don't understand the result given by awk !

Jean-Pierre.
# 11  
Old 06-06-2007
i have an actual ~ 2GB file created. Here's my results
Code:
# time awk 'END{print NR}' twogigfile.txt
39810582

real    1m8.784s
user    0m10.297s
sys     0m3.056s
# time awk '{x++}ED{ print x}' twogigfile.txt
39810582

real    1m6.738s
user    0m15.365s
sys     0m3.044s

# time wc -l twogigfile.txt
39810582 twogigfile.txt

real    1m5.920s
user    0m2.716s
sys     0m2.952s

# time sed -n '$=' twogigfile.txt
39810582

real    5m33.276s
user    5m12.508s
sys     0m3.368s

wc is comparable to awk methods. sed and cat methods pales.
# 12  
Old 06-06-2007
Quote:
Originally Posted by aigles

I don't understand the result given by awk !

Jean-Pierre.
This is surprising me ! Smilie Smilie Smilie

I tried with 600 MB file
though the time taken to compute number of lines were different

result were unique for each of the commands.
# 13  
Old 06-06-2007
number of lines : 587810152 ( 587 million )
file size : 2.737 GB
Code:
time wc -l d1
587810152
wc -l d1  10.62s user 2.65s system 21% cpu 1:02.15 total


time sed -n '$=' d1
587810152
sed -n '$=' d1  134.83s user 2.74s system 81% cpu 2:47.91 total

time awk '{x++}END{ print x}' d1
587810152
awk '{x++}END{ print x}' d1  487.43s user 3.02s system 95% cpu 8:31.66 total

time perl -e ' while (<>) { } print "$.\n" '  d1
587810152
perl -e ' while (<>) { } print "$.\n" ' d1  212.17s user 2.87s system 93% cpu 3:50.42 total

# 14  
Old 06-06-2007
Based on c.u.s. post.

Code:
$ uname -a
SunOS xxxxx 5.9 Generic_118558-36 sun4u sparc SUNW,Sun-Fire-V490
$ echo $BASH_VERSION
2.05.0(1)-release
$ du -k FILE1
124652  FILE1
$ type wc
wc is hashed (/usr/bin/wc)
$ time wc -l FILE1
 1475071 FILE1

real    0m5.519s
user    0m1.300s
sys     0m0.490s
$ # grep -c ^ will give you wrong numbers (one to high) 
$ # depending on whether the last line is incomplete or not.
$ time grep -c ^ FILE1 
1475071

real    0m1.411s
user    0m1.050s
sys     0m0.360s
$ /usr/dt/bin/dtksh
$ print ${.sh.version}
Version M-12/28/93d
$ type wc
wc is a shell builtin version of /bin/wc
$ time wc -l FILE1
 1475071 FILE1

real    0m1.36s
user    0m0.95s
sys     0m0.40s
$ # but ...
$ du -k FILE2
3314730 FILE2
$ wc -l FILE2
/usr/dt/bin/dtksh: wc: FILE2: cannot open [Value too large for defined data type]


Last edited by radoulov; 06-06-2007 at 04:22 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If file exists count lines

Hello, Currently I have: FILE=/home/file.txt if ; then echo "File $FILE exists" else echo "File $FILE does not exist" fi exit I would like to make it such that if the file *does* exist, it performs a wc -l count of the file and then if the count is greater than 3 performs... (3 Replies)
Discussion started by: holyearth
3 Replies

2. Shell Programming and Scripting

count lines in file to variable

I have a text file in which you need to identify the number of lines that looks like this: awk '{x + +} END {print x}' filename The problem is that I do not know how this data to any variable in which then need to continue to work in a cycle for .. do not know someone help? Sorry for my... (4 Replies)
Discussion started by: gizmo16
4 Replies

3. Shell Programming and Scripting

Help with script to read lines from file and count values

Hi, I need some help with a script I'm trying to write. I have a log file containing references to a number of different webservices. I wish to write a script that will list the webservices with a count as to how many times they appear in the log. An example of the log file content: ... (2 Replies)
Discussion started by: gman2010
2 Replies

4. Solaris

WC -l does not count all the lines in a file? HELP

I have a file that I need to merge with another like file. Normally I remove the trailer reocrd and merge the file and update the trailer record of the second file. I did a WC -l on the first file before I removed the trailer record, and again afterwards. The count came back the same. I opened the... (6 Replies)
Discussion started by: Harleyrci
6 Replies

5. Shell Programming and Scripting

Trying to do a count on multiple lines in a file

Guys I am having a problem with being able to do a count of entries in a file. What I am trying to get a count of the total number of members that are listed in the files. So I need to pull the number of the lines after members. I tried using sed but it only seems to count the first... (7 Replies)
Discussion started by: scottzx7rr
7 Replies

6. Shell Programming and Scripting

Count the number of lines in a file with one condition

Hi Everyone, 1.txt Mon 08 Feb 2010 12:30:44 AM MYT;1265560244;e164:0000116047275464;T;Central;0; Mon 08 Feb 2010 12:30:46 AM MYT;1265560246;e164:0000116047275464;T;Central;0; Mon 08 Feb 2010 12:30:48 AM MYT;1265560248;e164:0000116047275464;T;Central;0; Mon 08 Feb 2010 12:30:50 AM... (1 Reply)
Discussion started by: jimmy_y
1 Replies

7. UNIX for Dummies Questions & Answers

Count of Number of Lines in a File

Dear Members, I want to count the number of lines in a file; for that i am using the following command : FILE_LINE_COUNT=`wc -l $INT_IN/$RAW_FILE_NAME` if i do an echo on FILE_LINE_COUNT then i get 241 /home/data/testfile.txt I don't want the directory path to be displayed. Variable... (1 Reply)
Discussion started by: sandeep_1105
1 Replies

8. Shell Programming and Scripting

Count lines between two patterns inside a file

Hi, Im doing a script to find the number of lines included inside a file newly. These lines are in between #ifdef FLAG1 and #else or #endif or #else and #endif. I tried like this, awk '/#ifdef Flag1/,/#e/{print}' aa.c | wc -l awk '/#ifndef Flag1/,/#endif/{print}' aa.c | awk... (6 Replies)
Discussion started by: priyadarshini
6 Replies

9. Programming

to count the number of commented lines in a file

can someone guide me how to have a C pgm to count the number of commented lines? (3 Replies)
Discussion started by: naan
3 Replies

10. Shell Programming and Scripting

Count number of lines in Binary file

Hi Friends Please help me out to count number of lines in binary file. It gives some wrong(less) using wc -l. Is there any other way to count lines of binary file. Thanks. (3 Replies)
Discussion started by: vanand420
3 Replies
Login or Register to Ask a Question