Sponsored Content
Top Forums Shell Programming and Scripting quickest way to get the total number of lines in a file Post 302708251 by SkySmart on Monday 1st of October 2012 09:50:54 AM
Old 10-01-2012
Quote:
Originally Posted by jim mcnamara
Sky Smart - can you cite a verified reference as to why reading a file from record 1 to EOF is NOT the most efficient for carriage control files. rdrtx1's sample code does that and so does wc -l

I'll answer:
No such valid reference exists. You have to count the number of \n characters to get a line count. The only other possibility is for a fixed length record file. In that case you call stat, ls, or some code you have to get the number of bytes (reading file metadata: struct stat st_size) and then do integer division: bytes/recsz.

One other ' reliable' way is to call x=ftell() on the end of a file when you know the file has finished being written and divide x/recsz - again for fixed record length files. This is an even less efficient way to do stat.


NOTHING else exists. In other words: how can you know how many \n characters exist in the file?
when you run a "wc -l" on a file that big, it takes a while to get the total line count. i understand that the entire file must be read in order to get the total lines.

i'm also aware that in UNIX, there are more than one ways to get something done. on some linux systems a "grep -P" will get you want you want a lot faster than any other utility can. on others, the -P option is not available.

overall, i'm more concerned about speed and what the quickest way is to get total line count on a file that big.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Total of lines w/out header and footer incude for a file

I am trying to get a total number of tapes w/out headers or footers in a ERV file and append it to the file. For some reason I cannot get it to work. Any ideas? #!/bin/sh dat=`date +"%b%d_%Y"` + date +%b%d_%Y dat=Nov16_2006 tapemgr="/export/home/legato/tapemgr/rpts"... (1 Reply)
Discussion started by: gzs553
1 Replies

2. Shell Programming and Scripting

total number of lines

Hi have following file |abcd 2|abcd |sdfh |sdfj I want to find total number of files haivng nothing in feild 1 using awk will command awk -F "|" '( $1=="") {print NR}' test_awk will work??? (4 Replies)
Discussion started by: mahabunta
4 Replies

3. Shell Programming and Scripting

total number of lines in a file

Hi , How about find the total number of lines in a file ? How can i do that with the "grep" command ? (9 Replies)
Discussion started by: Raynon
9 Replies

4. Shell Programming and Scripting

Appending line number to each line and getting total number of lines

Hello, I need help in appending the line number of each line to the file and also to get the total number of lines. Can somebody please help me. I have a file say: abc def ccc ddd ffff The output should be: Instance1=abc Instance2=def Instance3=ccc Instance4=ddd Instance5=ffff ... (2 Replies)
Discussion started by: chiru_h
2 Replies

5. Shell Programming and Scripting

Removing lines from large files.. quickest method?

Hi I have some files that contain be anything up to 100k lines - eg. file100k I have another file called file5k and I need to produce filec which will contain everything in file100k minus what matches in file 5k.. ie. File100k contains 1FP 2FP 3FP File5k contains 2FP I would... (2 Replies)
Discussion started by: frustrated1
2 Replies

6. Shell Programming and Scripting

perl script on how to count the total number of lines of all the files under a directory

how to count the total number of lines of all the files under a directory using perl script.. I mean if I have 10 files under a directory then I want to count the total number of lines of all the 10 files contain. Please help me in writing a perl script on this. (5 Replies)
Discussion started by: adityam
5 Replies

7. Shell Programming and Scripting

Select lines in which column have value greater than some percent of total file lines

i have a file in following format 1 32 3 4 6 4 4 45 1 45 4 61 54 66 4 5 65 51 56 65 1 12 32 85 now here the total number of lines are 8(they vary each time) Now i want to select only those lines in which the values... (6 Replies)
Discussion started by: vaibhavkorde
6 Replies

8. Shell Programming and Scripting

Help with sum total number of record and total number of record problem asking

Input file SFSQW 5192.56 HNRNPK 611.486 QEQW 1202.15 ASDR 568.627 QWET 6382.11 SFSQW 4386.3 HNRNPK 100 SFSQW 500 Desired output file SFSQW 10078.86 3 QWET 6382.11 1 QEQW 1202.15 1 HNRNPK 711.49 2 ASDR 568.63 1 The way I tried: (2 Replies)
Discussion started by: patrick87
2 Replies

9. UNIX for Dummies Questions & Answers

Write the total number of rows in multiple files into another file

Hello Friends, I know you all are busy and inteligent too... I am stuck with one small issue if you can help me then it will be really great. My problem is I am having some files i.e. Input.txt1 Input.txt2 Input.txt3 Now my task is I need to check the total number of rows in... (4 Replies)
Discussion started by: malaya kumar
4 Replies

10. UNIX for Dummies Questions & Answers

How to find count total number of pattern in a file …?

How to find count total number of pattern in a file … File contains : a.txt ------------- aaa bbb nnn ccc aaa bbb aaa ddd aaa aaa aaa aaa grep -c aaa a.txt Op: 4 ( But my requirement is should count the total no of patterns as 7 ) (4 Replies)
Discussion started by: Jitten
4 Replies
wc(1)							      General Commands Manual							     wc(1)

NAME
wc - Counts the lines, words, characters, and bytes in a file SYNOPSIS
wc [-c | -m] [-lw] [file...] The wc command counts the lines, words, and characters or bytes in a file, or in the standard input if you do not specify any files, and writes the results to standard output. It also keeps a total count for all named files. STANDARDS
Interfaces documented on this reference page conform to industry standards as follows: wc: XCU5.0 Refer to the standards(5) reference page for more information about industry standards and associated tags. OPTIONS
Counts bytes in the input. Counts lines in the input. Counts characters in the input. Counts words in the input. OPERANDS
Specifies the pathname of the input file. If this operand is omitted, standard input is used. DESCRIPTION
A word is defined as a string of characters delimited by white space as defined in the X/Open Base Definitions for XCU4. The wc command counts lines, words, and bytes by default. Use the appropriate options to limit wc output. Specifying wc without options is the equivalent of specifying wc -lwc. If any options are specified, only the requested information is output. The order in which counts appear in the output line is lines, words, bytes. If an option is omitted, then the corresponding field in the output is omitted. If the -m option is used, then character counts replace byte counts. When you specify one or more files, wc displays the names of the files along with the counts. If standard input is used, then no file name is displayed. EXIT STATUS
The following exit values are returned: Successful completion. An error occurred. EXAMPLES
To display the number of lines, words, and bytes in the file text, enter: wc text This results in the following output: 27 185 722 text The numbers 27, 185, and 722 are the number of lines, words, and bytes, respectively, in the file text. To display only one or two of the three counts include the appropriate options. For example, the following command displays only line and byte counts: wc -cl text 27 722 text To count lines, words, and bytes in more than one file, use wc with more than one input file or with a file name pat- tern. For example, the following command can be issued in a directory containing the files text, text1, and text2: wc -l text* 27 text 112 text1 5 text2 144 total The numbers 27, 112, and 5 are the numbers of lines in the files text, text1, and text2, respectively, and 144 is the total number of lines in the three files. The file name is always appended to the output. To obtain a pure number for things like reporting purposes, pipe all input to the wc command using cat. For example, the following command will report the total count of characters in all files in a directory. echo There are `cat *.c | wc -c` characters in *.c files There are 1869 characters in *.c files ENVIRONMENT VARIABLES
The following environment variables affect the execution of wc: Provides a default value for the internationalization variables that are unset or null. If LANG is unset or null, the corresponding value from the default locale is used. If any of the internationalization vari- ables contain an invalid setting, the utility behaves as if none of the variables had been defined. If set to a non-empty string value, overrides the values of all the other internationalization variables. Determines the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multibyte characters in arguments and input files) and which characters are defined as white space characters. Determines the locale for the format and contents of diagnostic messages written to standard error and informative messages written to standard output. Determines the location of message catalogues for the processing of LC_MESSAGES. SEE ALSO
Commands: cksum(1), ls(1) Standards: standards(5) wc(1)
All times are GMT -4. The time now is 03:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy