Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Total record count of all the file present in a directory

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Total record count of all the file present in a directory
# 1  
Old 10-24-2017
Total record count of all the file present in a directory
Hi All ,
We need one help on the below requirement.We have multiple pipe delimited .txt file(around 100 .txt files) present on one directory.We need the total record count of all the files present in that directory without header.File format as below :

Code:
 COUPON_CODE|FRIENDLY_CODE|BRAND|EVENT_NAME|EVENT_ID|MASTER_CAMPAIGN_NAME|CAMPAIGN_COMMON_NAME|WEB_PROMO_ID|POS_PROMO_ID|MAX_STORE_USAGE_LIMIT|MAX_ECOMMERCE_USAGE_LIMIT|IS_REDEEMED|IS_EXPIRED|BARCODE_URL|WEB_REDEMPTION_COUNT|STORE_REDEMPTION_COUNT|CREATED_DATE|UPDATED_AT|START_DATE|EXPIRY_DATE|DO_NOT_EXPIRE|SUBCUSTOMER_ID|SUBCUSTOMER_SC_ID|IS_TEST_CODE|IS_CSR_CODE|REDEMPTION_AMOUNT_THRESHOLD|COUPON_AMOUNT|JOB_ID|IS_REWARDS_CERT|REWARDS_CERT_POINTS_APPLIED|REWARDS_CERT_AMOUNT_ISSUED|ISSUANCE_DATE|WEB_PROMOTION_NAME|STORE_PROMOTION_NAME|EVENT_CODE
12W51PS2PK98387|GWIN17DOL25|Gymboree|Gymboree Holiday 2 Winter DM $25 off $100|||Gym Winter 2 DM|||1|1|N|N||0|0|09/26/2017|09/26/2017|11/20/2017|12/24/2017|N||||N||||N|||11/13/2017|||98387
12W51QRB2N98387|GWIN17DOL25|Gymboree|Gymboree Holiday 2 Winter DM $25 off $100|||Gym Winter 2 DM|||1|1|N|N||0|0|09/26/2017|09/26/2017|11/20/2017|12/24/2017|N||||N||||N|||11/13/2017|||98387
12W51QZV5T98387|GWIN17DOL25|Gymboree|Gymboree Holiday 2 Winter DM $25 off $100|||Gym Winter 2 DM|||1|1|N|N||0|0|09/26/2017|09/26/2017|11/20/2017|12/24/2017|N||||N||||N|||11/13/2017|||98387

for one file ,we can get the record count like below :
Code:
 wc -l file1.txt

Can anyone kindly help me how to get total record count of all the .txt files present in that directory without header.Any help on this regard will be appreciated.Thanks !
# 2  
Old 10-24-2017
Hello STCET22,

Could you please try following and let me know if this helps you(not tested though).
Code:
awk 'END{print NR+1-length(ARGV)}' *.txt

Thanks,
R. Singh
These 2 Users Gave Thanks to RavinderSingh13 For This Post:
RudiC STCET22
# 3  
Old 10-24-2017
Would wc -l *.txt work? It will give you output for each file then a total.

If you just want the total just do this:-
Code:
total=$(wc -l *.txt|tail -1)                # Get the long information.  Adjust the wild-card to match all relevant files
read total junk < <(echo $total)            # Parse the line to get the value only
echo "The total number of lines is ${total}"


Another way might be simply:-
Code:
total=$(cat *.txt | wc -l)                  # All in one line



I hope that these help,
Robin
# 4  
Old 10-24-2017
Hello rbatte1,

Apologies if I have missed anything here, I think OP has requested to remove the very first line(header) from the count. That is why I removed it in END section of my code.

Thanks,
R. Singh
# 5  
Old 10-24-2017
@RaviderSingh13: Nice approach! Why not
Code:
awk 'END{print NR+1-ARGC}' *.txt

These 2 Users Gave Thanks to RudiC For This Post:
RavinderSingh13 STCET22
# 6  
Old 10-24-2017
Yes, I missed the extra requirement to ignore the header record.

Can we be sure that either:-
  • the files all have the same header?
  • no files will be zero bytes?
For the former, we could:-
Code:
total=$(egrep -v "^COUPON_CODE" *.txt | wc -l)                  # All in one line

For the latter (if the headers are different) we could perhaps:-
Code:
((total=$(cat *.txt | wc -l) - $(ls -1|wc -l)))

Would either of these work? I'm not sure on it's performance versus awk though.


Robin
This User Gave Thanks to rbatte1 For This Post:
STCET22
# 7  
Old 10-24-2017
Hi All,
Thank you all for your help.
@Ravinder,
As I'm new to awk part ,could you kindly explain the code for my understanding.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Speed : awk command to count the occurrences of fields from one file present in the other file

Hi, file1.txt AAA BBB CCC DDD file2.txt abc|AAA|AAAabcbcs|fnwufnq bca|nwruqf|AAA|fwfwwefwef fmimwe|BBB|fnqwufw|wufbqw wcdbi|CCC|wefnwin|wfwwf DDD|wabvfav|wqef|fwbwqfwfe i need the count of rows of file1.txt present in the file2.txt required output: AAA 2 (10 Replies)
Discussion started by: mdkm
10 Replies

2. UNIX for Dummies Questions & Answers

Count total image in directory

Dear all, I have a directory consisted of files in .jpg, .jpeg etc..all of them are image 20140411030143_62811159403_92886.jpg 5/11/2014 15:01 197K 20140415024737_62811618747_116460.jpg 4/15/2014 14:47 17K 20140415031003_62811618747_109192.jpg 4/17/2014 15:10 17K... (4 Replies)
Discussion started by: radius
4 Replies

3. Shell Programming and Scripting

Need unix commands to delete records from one file if the same record present in another file...

Need unix commands to delete records from one file if the same record present in another file... just like join ... if the record present in both files.. delete from first file or delete the particular record and write the unmatched records to new file.. tried with grep and while... (6 Replies)
Discussion started by: msathees
6 Replies

4. Shell Programming and Scripting

Help with sum total number of record and total number of record problem asking

Input file SFSQW 5192.56 HNRNPK 611.486 QEQW 1202.15 ASDR 568.627 QWET 6382.11 SFSQW 4386.3 HNRNPK 100 SFSQW 500 Desired output file SFSQW 10078.86 3 QWET 6382.11 1 QEQW 1202.15 1 HNRNPK 711.49 2 ASDR 568.63 1 The way I tried: (2 Replies)
Discussion started by: patrick87
2 Replies

5. Shell Programming and Scripting

csv file - adding total to a trailer record

Hi, I have a script which creates and modifies a csv file. I have managed to do everything I need to do apart from 1 thing. I need to append a trailer record to the file. I need this line to hold the total of an entire column of the csv file (skipping the 1st line which is a header). Can... (2 Replies)
Discussion started by: mcclunyboy
2 Replies

6. Shell Programming and Scripting

perl script on how to count the total number of lines of all the files under a directory

how to count the total number of lines of all the files under a directory using perl script.. I mean if I have 10 files under a directory then I want to count the total number of lines of all the 10 files contain. Please help me in writing a perl script on this. (5 Replies)
Discussion started by: adityam
5 Replies

7. UNIX for Advanced & Expert Users

Count total file downloaded using FTP

Hi All, I'm developing a FTP script as below: ftp -v -n <IP_ADDRESS> << EOF user avery jSqaqUU2 lcd /directory/folder/ ascii prompt mget * bye EOF I would like to enhance the script to count the total file downloaded. For example, once the script run i want the message "Total <n>... (1 Reply)
Discussion started by: cas553
1 Replies

8. Shell Programming and Scripting

To find the count of records from tables present inside a file.

hi gurus, I am having a file containing a list of tables.i want to find the count of records inside thes tables. for this i have to connect into database and i have to put the count for all the tables inside another file i used the following loop once all the tablenames are inside the file. ... (1 Reply)
Discussion started by: navojit dutta
1 Replies

9. UNIX for Dummies Questions & Answers

How to count the record count in an EBCDIC file.

How do I get the record count in an EBCDIC file on a Linux Box. :confused: (1 Reply)
Discussion started by: oracle8
1 Replies
Login or Register to Ask a Question