Need a script to parse data and output to csv


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need a script to parse data and output to csv
# 1  
Old 11-02-2017
Need a script to parse data and output to csv

I am not too savvy with arrays and am assuming that what I am looking for needs arrays. This is my requirement.



So I have the raw data that gets updated to a log as shown below

Code:
[20171102 10:00:00] StudentInfo:
FullInfo = {
    Address = Newark
    Age = 20
    Name= John
}

[20171102 10:00:00] StudentInfo:
FullInfo = {
    Address = NYC
    Age = 22
    Name = Mary
}


There are blocks of data in the sense, a block starts from '[' and closes at '}'. So from the above there are 2 blocks.

The fields are what appears between '{' and '}'. These fields are not fixed, in the sense, i will run the script and supply the fields, sometimes I will supply say 3 fields - Address, Age and Name and other times more or less.

Can a script be made to parse this data and output the data into csv?

So the output for the above would look as

Code:
DateTime                  Address   Age     Name
20171102 10:00:00  Newark    20        John
20171102 10:00:00  NYC         22        Mary


If i had say another field supplied say Subject, then the output would be


Code:
[20171102 10:00:00] StudentInfo:
FullInfo = {
    Address = Newark
    Age = 20
    Name = John
    Subject = Math
}

[20171102 10:00:00] StudentInfo:
FullInfo = {
    Address = NYC
    Age = 22
    Name = Mary
    Subject = Literature
}

I would the output in csv as

Code:
DateTime                  Address   Age     Name    Subject
20171102 10:00:00  Newark    20        John     Math
20171102 10:00:00  NYC         22        Mary     Literature


Please can some one advise. The farthest I have got is doing awk and parsing the data, but I dont know how to call it by block and the align it in columns
# 2  
Old 11-02-2017
Hello sidnow,

Could you please try following and let me know if this helps you.
Code:
awk 'BEGIN{
  print "DateTime                  Address   Age     Name"
}
/^}/{
  print date1,add,age,name;
  next
}
/StudentInfo/{
  gsub(/\[|\]/,"");
  date1=$1 $2;
  next
}
/Address/{
  add=$3;
  next
}
/Age/{
  age=$3;
next
}
/Name/{
  name=NF==2?$2:$3;
  next
}'   Input_file

Thanks,
R. Singh
# 3  
Old 11-02-2017
Try also
Code:
awk  -F"[= ]*"  '
/^[[]/  {V = $1 " " $2 " " ++CNT
         gsub (/[][]/, "", V)
         next
        }

NF < 3  {next
        }

        {LN[V]
         HD[$2]
         MX[V,$2] = $3
        }

END     {H = "DateTime"
                        printf "%20s", H; for (i in HD) printf "%10.10s", i; print "";
         for (j in LN) {printf "%20s", j; for (i in HD) printf "%10.10s", MX[j,i]; print ""}
        }
' file
            DateTime       Age      Name   Subject   Address
 20171102 10:00:00 4        22      MaryLiterature       NYC
 20171102 10:00:00 3        20      John      Math    Newark
 20171102 10:00:00 2        22      Mary                 NYC
 20171102 10:00:00 1        20      John              Newark

We need the running count CNT to differentiate between single records, small additional code can suppress it if need be.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Python script to run multiple command and append data in output csv file

Experts, I am writing a script and able to write only small piece of code and not able to collect logic to complete this task. In input file have to look for name like like this (BGL_HSR_901_1AG_A_CR9KTR10) before sh iss neors. Record this (BGL_HSR_901_1AG_A_CR9KTR10) in csv file Now have to... (0 Replies)
Discussion started by: as7951
0 Replies

2. Shell Programming and Scripting

Copy data to CSV file from txt output

Hi Everyone , Below is output from a TL1 , I want just the NE Name: and beside that the Temperature and the voltages in a csv file , Is this possible? > act-user:AB1S2a:ArshadFO:493::**********; AB1S2a 2016-02-07 10:13:24 M 493 COMPLD "ArshadFO:2016-02-07 10-04-55,0" ;... (11 Replies)
Discussion started by: adgjmpt
11 Replies

3. Shell Programming and Scripting

How to parse this file using awk and output in CSV format?

My source file looks like this: Cust-Number = "101" Cust-Name="Joe" Cust-Town="London" Cust-hobby="tennis" Cust-purchase="200" Cust-Number = "102" Cust-Name="Mary" Cust-Town="Newyork" Cust-hobby="reading" Cust-purchase="125" Now I want to parse this file (leaving out hobby) and... (10 Replies)
Discussion started by: Balav
10 Replies

4. Shell Programming and Scripting

Output large volume of data to CSV file

I have a program that output the ownership and permission on each directory and file on the server to a csv file. I am getting error message when I run the program. The program is not outputting to the csv file. Error: the file access permissions do not allow the specified action cannot... (2 Replies)
Discussion started by: dellanicholson
2 Replies

5. Shell Programming and Scripting

BASH script to parse XML and generate CSV

Hi All, Hope all you are doing good! Need your help. I have an XML file which needs to be converted CSV file. I am not an expert of awk/sed so your help is highly appreciated!! XML file looks like this: <l:event dateTime="2013-03-13 07:15:54.713" layerName="OSB" processName="ABC"... (2 Replies)
Discussion started by: bhaskar_m
2 Replies

6. UNIX for Dummies Questions & Answers

Help to parse csv file with shell script

Hello ! I am very aware that this is not the first time this question is asked here, because I have already read a lot of previous answers, but none of them worked, so... As said in the title, I want to read a csv file with a bash script. Here is a sample of the file: ... (4 Replies)
Discussion started by: Grhyll
4 Replies

7. Shell Programming and Scripting

Extract and parse XML data (statistic value) to csv

Hi All, I need to parse some statistic data from the "measInfo" -eg. 25250000 (as highlighted) and return the result into line by line, and erasing all other unnecessary info/tag. Thought of starting with grep "measInfoID="25250000" but this only returns 1 line. How do I get all the output... (8 Replies)
Discussion started by: jackma
8 Replies

8. Shell Programming and Scripting

select data from oracle table and save the output as csv file

Hi I need to execute a select statement in a solaris environment with oracle database. The select statement returns number of rows of data. I need the data to be inserted into a CSV file with proper format. For that we normally use "You have to select all your columns as one big string,... (2 Replies)
Discussion started by: rdhanek
2 Replies

9. Shell Programming and Scripting

parse csv file, sha1 hash and output

I have a file, not really a csv, but containing delineated data just the same. Lets call that file "raw_data.txt". It contains data in the format of company name:fein number like this: first company name:123456789 second company name:987654321 what i need to do is read this file, apply... (11 Replies)
Discussion started by: FreddyG
11 Replies

10. Shell Programming and Scripting

Unix Script to parse a CSV

I am writing a unix script that will parse a CSV and edit the values. My CSV looks like this 0,0,0,0,1,0,1,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0 10,11,7,0,4,12,2,3,7,0,11,3,12,4,0,5,5,4,5,0,8,6,12,0,9,3,3,0,2,7,8... (16 Replies)
Discussion started by: RJ17
16 Replies
Login or Register to Ask a Question