Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Shell script for field wise record count for different Files .csv files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script for field wise record count for different Files .csv files
# 8  
Old 01-21-2016
Hello Kirands,

Thank you for posting the sample data, request you to please use code tags like I am using as follows for commands into your posts as per forum rules. If you need to show the columns which have NULL values too then following may help.
Code:
awk 'NR==1 || NR==2{n=NF;OFS=",";for(i=1;i<=NF;i++){Q=Q?Q OFS $i:$i}print Q;Q="";next} {for(i=1;i<=NF;i++){array[i]++}} END{for(j=1;j<=n;j++){A=A?A OFS array[j]:array[j]};if(A){print A;A=""}}' OFS=,   Input_file

Output will be as follows.
Code:
STATUS,USERID,USERNAME,FIRSTNAME,NICKNAME,MI,LASTNAME,SUFFIX,TITLE
STATUS,USERID,Username,FirstName,Nickname,MiddleName,LastName,Suffix,Title
4,4,4,3,1,1,,,

If you don't need the columns which have entirely NULL values in Input_file then following may help you in same.
Code:
awk 'NR==1 || NR==2{n=NF;OFS=",";for(i=1;i<=NF;i++){Q=Q?Q OFS $i:$i}print Q;Q="";next} {for(i=1;i<=NF;i++){array[i]++}} END{for(j=1;j<=n;j++){if(array[j]){A=A?A OFS array[j]:array[j]}};print A;A=""}' OFS=,   Input_file

Output will be as follows.
Code:
STATUS,USERID,USERNAME,FIRSTNAME,NICKNAME,MI,LASTNAME,SUFFIX,TITLE
STATUS,USERID,Username,FirstName,Nickname,MiddleName,LastName,Suffix,Title
4,4,4,3,1,1

Also you could redirect output of these above commands to a Output_file.csv file too, hope this helps you.

Thanks,
R. Singh
# 9  
Old 01-21-2016
As the sequence of records is undefind for the (j in array) construct, and, as empty/blank fields should not be counted, this might be better fit:
Code:
awk '
NR<=2   {print
         next
        }
        {for (i=1; i<=NF; i++) array[i] += ($i !~ /^ *$/)
         if (NF > MX) MX = NF
        }
END     {for (i=1; i<=MX; i++)  printf "%s%s", array[i]+0, (i!=MX)?OFS:ORS
        }
' FS=, OFS=, /tmp/RecoundCountTest.csv

# 10  
Old 01-21-2016
Hi Ravindra,

Thanks for your response.

Below is the source data:
Code:
STATUS,USERID,USERNAME,FIRSTNAME,NICKNAME,MI,LASTNAME,SUFFIX,TITLE
STATUS,USERID,Username,FirstName,Nickname,MiddleName,LastName,Suffix,Title
ACTIVE,80001001,itria1,Carlos Guillermo, , ,Itria, , 
ACTIVE,80001002,lafournere1,Jean-Paul, , ,, , 
ACTIVE,80001003,,Wade Peter, , ,, , 
ACTIVE,80001004,,Olivier, , ,, ,

Could you please have a look on this and please provide me the code to get the record count:
I have checked the latest code provided, but could not give me correct results. The above source data is now separated with the columns with the data having nulls with the commas (,)..

My output should be as below :
Code:
STATUS,USERID,USERNAME,FIRSTNAME,NICKNAME,MI,LASTNAME,SUFFIX,TITLE
STATUS,USERID,Username,FirstName,Nickname,MiddleName,LastName,Suffix,Title
4,4,2,4,0,0,1,0,0

Please help to resolve.. Thank you for your support.

Regards,
Kiran.

---------- Post updated at 09:04 AM ---------- Previous update was at 08:22 AM ----------

Hi Rudic,

Thank you for your response.

The code provide by you is giving the correct but the last column output result is not giving the correct record count.

Source file is below :
Code:
STATUS,USERID,USERNAME,FIRSTNAME,NICKNAME,MI,LASTNAME,SUFFIX,TITLE
STATUS,USERID,Username,FirstName,Nickname,MiddleName,LastName,Suffix,Title
ACTIVE,80001001,itria1,Carlos Guillermo, , ,Itria, , 
ACTIVE,80001002,lafournere1,Jean-Paul, , ,, , 
ACTIVE,80001003,,Wade Peter, , ,, , 
ACTIVE,80001004,,Olivier, , ,, ,

Target file after running the code provided by you is giving as below :
Code:
STATUS,USERID,USERNAME,FIRSTNAME,NICKNAME,MI,LASTNAME,SUFFIX,TITLE
STATUS,USERID,Username,FirstName,Nickname,MiddleName,LastName,Suffix,Title
4,4,2,4,0,0,1,0,4

The last columns count is giving as 4 which is incorrect.

Expected result is as below : (last column should be 0 as count)
Code:
STATUS,USERID,USERNAME,FIRSTNAME,NICKNAME,MI,LASTNAME,SUFFIX,TITLE
STATUS,USERID,Username,FirstName,Nickname,MiddleName,LastName,Suffix,Title
4,4,2,4,0,0,1,0,0

Please help to resolve this issues.

Thanks you,
Kiran.
Last edited by vbe; 01-21-2016 at 10:23 AM..
# 11  
Old 01-21-2016
Reminder:
We are not here to do the work for you, so far in this thread you have not shown what you have tried before opening the thread...
So before posting
Quote:
Please help to resolve this issues
You should try to understand what has been posted and maybe ask what bit isnt clear for you as you are trying to adapt but face an issue
This User Gave Thanks to vbe For This Post:
Kirands
# 12  
Old 01-21-2016
Quote:
Originally Posted by Kirands
.
.
.
The last columns count is giving as 4 which is incorrect.
.
.
.
The last columns count is giving as 4 which is absolutely correct as your file has DOS line terminators 0x0D (or <CR> or \r) that result in a non-empty field which is to be counted. Remove those to make it a correct *nix text file, and it will fly.
This User Gave Thanks to RudiC For This Post:
Kirands
# 13  
Old 01-21-2016
Dear Vbe,

I really tried and did more analysis and executing the code given by expertise, going forward i will sure mention my analysis and understanding to the expertise.

Thank you, Kiran.
# 14  
Old 01-21-2016
In addition to CR/LF line terminators instead of <newline> line terminators, note also that when the field separator is a comma, a <space> in a field is not an empty field. So, for instance, the line in the sample input:
Code:
ACTIVE,80001002,lafournere1,Jean-Paul, , ,, ,

when expanded so we can see all of the spaces:
Code:
printf '%s' 'ACTIVE,80001002,lafournere1,Jean-Paul, , ,, , '|od -bc 
0000000   101 103 124 111 126 105 054 070 060 060 060 061 060 060 062 054
           A   C   T   I   V   E   ,   8   0   0   0   1   0   0   2   ,
0000020   154 141 146 157 165 162 156 145 162 145 061 054 112 145 141 156
           l   a   f   o   u   r   n   e   r   e   1   ,   J   e   a   n
0000040   055 120 141 165 154 054 040 054 040 054 054 040 054 040        
           -   P   a   u   l   ,       ,       ,   ,       ,            
0000056

only has one empty field (noted by the 2 adjacent commas marked in red). The last field on that line contains a <space> character and is, therefore, not empty.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding total distinct count from multiple csv files through UNIX script

Hi All , I have multiple pipe delimited csv files are present in a directory.I need to find out distinct count on a column on those files and need the total distinct count on all files. We can't merge all the files here as file size are huge in millions.I have tried in below way for each... (9 Replies)
Discussion started by: STCET22
9 Replies

2. Shell Programming and Scripting

Extract count of string in all files and display on date wise

Hi All, hope you all are doing well! I kindly ask you for shell scripting help, here is the description: I have huge number of files shown below on date wise, which contains different strings(numbers you can say) including 505001 and 602001. ... (14 Replies)
Discussion started by: VasuKukkapalli
14 Replies

3. Shell Programming and Scripting

Match columns from two csv files and update field in one of the csv file

Hi, I have a file of csv data, which looks like this: file1: 1AA,LGV_PONCEY_LES_ATHEE,1,\N,1,00020460E1,0,\N,\N,\N,\N,2,00.22335321,0.00466628 2BB,LES_POUGES_ASF,\N,200,200,00006298G1,0,\N,\N,\N,\N,1,00.30887539,0.00050312... (10 Replies)
Discussion started by: djoseph
10 Replies

4. Shell Programming and Scripting

How to create or convert to pdf files from csv files using shell script?

Hi, Can anyone help me how to convert a .csv file to a .pdf file using shell script Thanks (2 Replies)
Discussion started by: ssk250
2 Replies

5. Shell Programming and Scripting

Record count checking for multiple files through for-loop

Hi Friends, I wrote one shell script to check the record count in two files and that will send us the notification activity if found zero record count. What i did is I created for loop and checking the count for both of the files but what is happening is for first file has data then it's... (13 Replies)
Discussion started by: victory
13 Replies

6. Shell Programming and Scripting

Script to compare count of two csv files

Hi Guys, I need to write a script to compare the count of two csv files each having 5 columns. Everyday a csv file is recived. Now we need to compare the count of todays csv file with yesterday's csv file and if the total count of records is same in todays csv file and yesterday csv file out... (3 Replies)
Discussion started by: Vivekit82
3 Replies

7. Shell Programming and Scripting

count of record in files

Hi all, I have written a scripts which count number of lines in all the files in a directory and write in a text file. I don't know how to format it while writing. OS suns solaris 10 my scripts is as below for i in /ersdg3/ERS/ERS_INPUT_LOGS/RIO/LOGS/RIO_02-Aug-2012/ *.LOG do echo... (11 Replies)
Discussion started by: guddu_12
11 Replies

8. Shell Programming and Scripting

Combine Multiple text or csv files column-wise

Hi All I am trying to combine columns from multiple text files into a single file using paste command but the record length being unequal in the different files the data is running over to the closest empty cell on the left. Please see below. What can i do to resolve this ? File 1 File... (15 Replies)
Discussion started by: venky_ibm
15 Replies

9. Shell Programming and Scripting

Comparison of record count of two files

Hi, I have one text file and zip file in UNIX directory.The Zip file contains another text file. For Ex: Text File Name = Req_file.txt Zip file Name= Response_file.txt_04072009_121548.gz Zip file contains one text file that name is Response_file.txt I want to compare the record... (1 Reply)
Discussion started by: praka
1 Replies
Login or Register to Ask a Question