Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help with Getting distinct record count from a .dat file using UNIX command

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with Getting distinct record count from a .dat file using UNIX command
# 1  
Old 06-07-2016
Help with Getting distinct record count from a .dat file using UNIX command
Hi,
I have a .dat file with contents like the below:

Input file

Code:
 ============SEQ NO-1: COLUMN1==========
 9835619
 7152815
============SEQ NO-2: COLUMN2 ==========
  7615348
 7015548
  9373086
============SEQ NO-3: COLUMN3===========
 9373086


Expected Output: (I just need the total distinct count of records under a separate header-as like below)
Code:
 ============SEQ NO-1: COLUMN1==========
2
============SEQ NO-2: COLUMN2 ==========
 3
 ============SEQ NO-3: COLUMN3===========
1

Please help me with the unix command for the same.
Last edited by Don Cragun; 06-07-2016 at 05:04 AM.. Reason: Add CODE tags.
# 2  
Old 06-07-2016
Hello MS06,

Welcome to forums, request you to please use code tags as per forum rules for commands/codes/Inputs which you are using into your posts. For your requirement, could you please try following.
Code:
awk '/^==/{if(count){print count;count=""};print;next} {count++} END{print count}'   Input_file 
OR
awk '/^==/{if(count){print count;count=""};print;next} !/^$/{count++} END{print count}'  Input_file

Second solution will leave all empty lines and will not count them. So output will be as follows.
Code:
============SEQ NO-1: COLUMN1==========
2
============SEQ NO-2: COLUMN2 ==========
3
============SEQ NO-3: COLUMN3===========
1

Thanks,
R. Singh
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Homework & Coursework Questions

awk command to retrieve record 23 and 89 from UNIX file

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: I am looking for awk command to retrieve only the record number 23 and record number 89 from a unix file?... (6 Replies)
Discussion started by: rakeshp
6 Replies

2. UNIX for Beginners Questions & Answers

awk command to retrieve record 23 and 89 from UNIX file

Hi Everyone, I am looking for awk command to retrieve only the record number 23 and record number 89 from a unix file? Please let me know what is the awk command for this? Regards Rakesh (1 Reply)
Discussion started by: rakeshp
1 Replies

3. Shell Programming and Scripting

Finding total distinct count from multiple csv files through UNIX script

Hi All , I have multiple pipe delimited csv files are present in a directory.I need to find out distinct count on a column on those files and need the total distinct count on all files. We can't merge all the files here as file size are huge in millions.I have tried in below way for each... (9 Replies)
Discussion started by: STCET22
9 Replies

4. Shell Programming and Scripting

How to use 'ls' command to list files like *.dat, not *.*.dat?

How to use 'ls' command to list files like *.dat, not *.*.dat (5 Replies)
Discussion started by: pmcginni777
5 Replies

5. Shell Programming and Scripting

FASTEN count line of dat file and compare with the CTRL file

Hi All, I thinking on how to accelerate the speed on calculate the dat file against the number of records CTRL file. There are about 300 to 400 folder directories that contains both DAT and CTL files. DAT contain all the flat files records CTL is the reference check file for the... (3 Replies)
Discussion started by: ckwan
3 Replies

6. UNIX for Dummies Questions & Answers

Count rows in .dat file

Could you please tell e how to find the total number of rows in a .dat file. edit by bakunin: this is exactly why in "advanced and expert" section?? I transfer this thread to "Unix for Dummies Questions and Answers" (2 Replies)
Discussion started by: Deeptanshu
2 Replies

7. UNIX for Dummies Questions & Answers

Record count in Unix servers

Hi All, when i ftp the file from windows machine to unix server the record count in the file is decreased by 1. example: Say a text file has three lines in it and when you place this file in unix server from your local machine and run a below command cat filename | wc -l the output is... (2 Replies)
Discussion started by: Umeshrm
2 Replies

8. Shell Programming and Scripting

Getting Sum, Count and Distinct Count of a file

Hi all this is a UNIX question. I have a large flat file with millions of records. col1|col2|col3 1|a|b 2|c|d 3|e|f 3|g|h footer**** I am supposed to calculate the sum of col1 1+2+3+3=9, count of col1 1,2,3,3=4, and distinct count of col1 1,2,3=c3 I would like it if you avoid... (4 Replies)
Discussion started by: singhabhijit
4 Replies

9. UNIX for Dummies Questions & Answers

Record count problem using sed command

Hi, I have a script which removes 2 header records and 1 trailer record in a list of files. The commands doing the actions are sed '1,2d' $file > tempfile1.dat sed '$d' < tempfile1.dat > $output.txt Its working fine for all records except a file having size=1445509814 and number of... (2 Replies)
Discussion started by: ayanbiswas
2 Replies

10. UNIX for Dummies Questions & Answers

How to count the record count in an EBCDIC file.

How do I get the record count in an EBCDIC file on a Linux Box. :confused: (1 Reply)
Discussion started by: oracle8
1 Replies
Login or Register to Ask a Question