Script to generate .csv file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to generate .csv file
# 1  
Old 04-17-2016
Script to generate .csv file

Dears,I need your help in this, I have to create a report based on the output file generated by another program. I want to write a shell script for this. The output file generated every 15 minutes but i can’t open it until the end of day so the script will get the file as an input the file will be as follows:
Code:
PEr;ST100000;ET101500,s1;Si1010;xw129;tr54;vyy87;Gk67;kl888
PEr;ST100000;ET101500,s1;Si1020;xw129;tr54;vyy87;ANf876;Gk67
PEr;ST100000;ET101500,s1;Si1030;xw129;tr54;vyy87;ANf876;Gk67;kl888
PEr;ST100000;ET101500,s1;Si1040;tr54;ANf876;kl888
PEr;ST100000;ET101500,s1;Si1060;xw129;tr54;vyy87;ANf876;Gk67;kl888
PEr;ST100000;ET101500,s1;Si1060;tr54;vyy87;ANf876;Gk67
PEr;ST100000;ET101500,s1;Si1070;xw129;tr54;vyy87;ANf876;Gk67;kl888
PEr;ST100000;ET101500,s1;Si1080;xw129;vyy87;Gk67;kl888
PEr;ET101500;ET103000,s1;Si1010;xw134;tr54;vyy87;ANf845;kl882
PEr;ET101500;ET103000,s1;Si1020;xw169;tr34;vyy87;ANf576;Gk62;kl883
PEr;ET101500;ET103000,s1;Si1030;xw179;tr24;vyy87;ANf676;Gk62;kl884
PEr;ET101500;ET103000,s1;Si1040;xw189;tr94;vyy87;ANf764;Gk63;kl885
PEr;ET101500;ET103000,s1;Si1050;xw134;tr54;vyy87;Gk63;kl882
PEr;ET101500;ET103000,s1;Si1060;tr34;vyy87;ANf576;Gk62;kl883
PEr;ET101500;ET103000,s1;Si1070;xw179;tr24;vyy87;kl884
PEr;ET101500;ET103000,s1;Si1080;tr94;ANf764;Gk63
PEr;ET103000;ET104500,s1;Si1010;tr84;vyy87;ANf976;Gk67;kl888
PEr;ET103000;ET104500,s1;Si1020;xw129;tr74;vyy87;Gk67;kl888
PEr;ET103000;ET104500,s1;Si1030;xw139;tr64;vyy87;ANf876;Gk67;kl888
PEr;ET103000;ET104500,s1;Si1040;xw125;tr54;vyy87;ANf876;kl888

the expected output

Time:101500
Item_Nu;xw;tr;vyy;ANf;Gk;kl
1010;129;54;87;;67;888
1020;129;54;87;876;67;
1030;129;54;87;876;67;888
1040;;54;;876;;888
1060;129;54;87;876;67;888
1060;;54;87;876;67;
1070;129;54;87;876;67;888
1080;129;;87;;67;888

Time:103000
Item_Nu;xw;tr;vyy;ANf;Gk;kl
1010;134;54;87;845;;882
1020;169;34;87;576;62;883
1030;179;24;87;676;Gk62;884
1040;189;94;87;764;Gk63;885
1050;134;54;;87;;63;882
1060;;34;87;576;62;883
1070;179;24;87;;;884
1080;;94;;764;63;

# 2  
Old 04-17-2016
What's the difference between this question, and the one you asked last week?
# 3  
Old 04-17-2016
One requirment was missed which is some of the tags might not be updated in the next 15 minutes 😢
# 4  
Old 04-17-2016
Hello abdul2020,

Could you please try following and let me know if this helps you.
Code:
awk -F";" '{gsub(/[[:alpha:]]/,X,$0)} FNR==NR{A[$2]=A[$2]?A[$2] ORS $4 FS $5 FS $6 FS $7 FS $8 FS $9:$4 FS $5 FS $6 FS $7 FS $8 FS $9;next} ($2 in A){print "Time:"$2 ORS "Item_Nu;xw;tr;vyy;ANf;Gk;kl" OFS A[$2];delete A[$2]}'  Input_file   Input_file

Output will be as follows.
Code:
Time:100000
Item_Nu;xw;tr;vyy;ANf;Gk;kl 1010;129;54;87;67;888
1020;129;54;87;876;67
1030;129;54;87;876;67
1040;54;876;888;;
1060;129;54;87;876;67
1060;54;87;876;67;
1070;129;54;87;876;67
1080;129;87;67;888;
Time:101500
Item_Nu;xw;tr;vyy;ANf;Gk;kl 1010;134;54;87;845;882
1020;169;34;87;576;62
1030;179;24;87;676;62
1040;189;94;87;764;63
1050;134;54;87;63;882
1060;34;87;576;62;883
1070;179;24;87;884;
1080;94;764;63;;
Time:103000
Item_Nu;xw;tr;vyy;ANf;Gk;kl 1010;84;87;976;67;888
1020;129;74;87;67;888
1030;139;64;87;876;67
1040;125;54;87;876;888

If above doesn't match your requirement completely, then please show us Input_file(more samples) with expected output and all your conditions too, hope this helps.
EDIT: Adding a non-one liner form on same.
Code:
awk -F";" '{
                gsub(/[[:alpha:]]/,X,$0)
           }
                FNR==NR{
                                A[$2]=A[$2]?A[$2] ORS $4 FS $5 FS $6 FS $7 FS $8 FS $9:$4 FS $5 FS $6 FS $7 FS $8 FS $9;
                                next
                       }
           ($2 in A)   {
                                print "Time:"$2 ORS "Item_Nu;xw;tr;vyy;ANf;Gk;kl" OFS A[$2];
                                delete A[$2]
                       }
          '  Input_file  Input_file

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
# 5  
Old 04-17-2016
Please give a clear and detailed specification of your problem: input and output samples and a description of the logics and or algorithms connecting them, instead of hoping that someone out in them there hills will undergo the ordeal of scrutinizing your samples field by field, trying to find out WHAT you really need!

What in the solutions given to your other (nearly) identical problem does not fit? Where are you stuck? Any attempts from your side? Which "tags might not be updated"? What are the tags?
# 6  
Old 04-17-2016
Quote:
Originally Posted by RudiC
Please give a clear and detailed specification of your problem: input and output samples and a description of the logics and or algorithms connecting them, instead of hoping that someone out in them there hills will undergo the ordeal of scrutinizing your samples field by field, trying to find out WHAT you really need!

What in the solutions given to your other (nearly) identical problem does not fit? Where are you stuck? Any attempts from your side? Which "tags might not be updated"? What are the tags?
First of all thank you for your help, I came up with this

Code:
cat file | cut -d";" -f-3 | uniq | while read l
    do echo $l | cut -d';' -f3 | sed -e 's/[^0-9]*\([0-9]*\),.*/Time:\1/g'
     echo 'Item_Nu;xw;tr;vyy;ANf;Gk;kl'
     cat  file |  grep "^$l" |  cut -d';' -f4- | sed 's/[a-z|A-Z]//g'
     echo
    done

but didn't work probably once I ran it with the real data. the file input file thousand of lines. I got stuck where some of the tags will not updated which led to not map the headers
Code:
'Item_Nu;xw;tr;vyy;ANf;Gk;kl'

with the values because the above script prints the hears and then prints the values without mapping them.

I found this new requirement yesterday once i have tried with the real data.

I'm sorry for this confusion and many thanks for help.

Moderator's Comments:
Mod Comment Please use code tags as required by forum rules!

Last edited by RudiC; 04-17-2016 at 04:54 PM.. Reason: Added code tags.
# 7  
Old 04-17-2016
Please consider a precise and detailed specification next time, like
- tags (or field names) are reflected in the header except for the first header field that is NOT found back in the data records but should be filled with data field 4
- fields are separated by ";"
- data field 3 is the time stamp and should be printed with every change after separation of its field name
- data field 4 needs to lose its field name and trailer and then to be printed as first field per line
- alpha only field name of arbitrary length is leading the fields, numerical field values follow without separator
- fields should be printed according to their position in the header.
etc. etc. etc.

Wildly guessing on what MIGHT BE required, I came up with
Code:
awk -F";" '
BEGIN           {HDStr = "Item_Nu;xw;tr;vyy;ANf;Gk;kl"
                 HDCNT = split (HDStr, HD)
                }

$3 != LAST      {LAST = $3
                 gsub (/[^0-9]|.$/, _, $3)
                 print "Time: " $3
                 print HDStr
                }

                {for (i=5; i<=NF; i++)  {IX = match ($i, /[0-9]/)
                                         RES[substr ($i, 1, IX-1)] = substr ($i, IX)
                                        }
                 gsub (/[^0-9]/, _, $4)
                 printf "%s", $4
                 for (i=2; i<=HDCNT; i++)       {printf ";%s", RES[HD[i]]
                                                }
                 printf "\n"
                 split ("", RES)
                }

' file
Time: 101500
Item_Nu;xw;tr;vyy;ANf;Gk;kl
1010;129;54;87;;67;888
1020;129;54;87;876;67;
1030;129;54;87;876;67;888
1040;;54;;876;;888
1060;129;54;87;876;67;888
1060;;54;87;876;67;
1070;129;54;87;876;67;888
1080;129;;87;;67;888
Time: 103000
Item_Nu;xw;tr;vyy;ANf;Gk;kl
1010;134;54;87;845;;882
1020;169;34;87;576;62;883
1030;179;24;87;676;62;884
1040;189;94;87;764;63;885
1050;134;54;87;;63;882
1060;;34;87;576;62;883
1070;179;24;87;;;884
1080;;94;;764;63;
Time: 104500
Item_Nu;xw;tr;vyy;ANf;Gk;kl
1010;;84;87;976;67;888
1020;129;74;87;;67;888
1030;139;64;87;876;67;888
1040;125;54;87;876;;888

This User Gave Thanks to RudiC For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Generate .csv/ xls file report

There can be thousand of .ksh in a specific directory where sql files are called from ksh. Requirement is to loop through all the files content and generate a report like below: Jobname Type type sqlname gemd1970 sql daily tran01 gemw1971 sql weekly ... (6 Replies)
Discussion started by: vedanta
6 Replies

2. Shell Programming and Scripting

Script to generate csv file

Dears, I am new in shell world and I need your help in this, I have to create a report based on the output file generated by another program. I want to write a shell script for this. The output file generated every 15 minutes but i can’t open it until the end of day so the script will get the... (3 Replies)
Discussion started by: abdul2020
3 Replies

3. Shell Programming and Scripting

BASH script to parse XML and generate CSV

Hi All, Hope all you are doing good! Need your help. I have an XML file which needs to be converted CSV file. I am not an expert of awk/sed so your help is highly appreciated!! XML file looks like this: <l:event dateTime="2013-03-13 07:15:54.713" layerName="OSB" processName="ABC"... (2 Replies)
Discussion started by: bhaskar_m
2 Replies

4. Shell Programming and Scripting

Script to generate csv file

Hello; I need to generate a csv file that contains a list of all the files in a particular server (from the root directory ie: \) that have a permission stamp of 777. I would like to create the csv so that it contains the following: server name, file name, full path name where file exists,... (17 Replies)
Discussion started by: gvolpini
17 Replies

5. Shell Programming and Scripting

Read a CSV file and generate SQL output

Friends, This is what I need: I will pass a CSV file as an input, and I want my shell to be reading that CSV file, and based on the parameters it should generate SQLs and write those SQL in a different file in the same location. I'm new to Shell scripting. I'm currently working on a... (25 Replies)
Discussion started by: Ram.Math
25 Replies

6. Shell Programming and Scripting

to read a CSV file and generate SQL output

Friends, This is what I need: I will pass a CSV file as an input, and I want my shell to be reading that CSV file, and based on the parameters it should generate SQLs and write those SQL in a different file in the same location. I'm new to Shell scripting. I'm currently working on a... (1 Reply)
Discussion started by: Ram.Math
1 Replies

7. Shell Programming and Scripting

Need to generate .csv file

I have a csv file with the following data Please find the attachment - zip ... (6 Replies)
Discussion started by: vaas
6 Replies

8. Shell Programming and Scripting

need help in Parsing a CSV file and generate a new output file

Hi Scripting Gurus, I am trying to parse a csv file and generate a new output file. The input file will be a variable length in turns of rows and columns. output file will have 8 columns. we have three columns from the header for each set. just to give little bit more clarification each row... (15 Replies)
Discussion started by: vkr
15 Replies

9. UNIX for Dummies Questions & Answers

generate CSV file using AWK script

Hi guys I have a text report that consists of text in some parts and data in some parts. e.g Report for changes in cashflows No changes were found Report for changes in Bills deal_num deal_date trader maturity log_creator DF_234 20-5-2008 tman 20-5-2009 tman... (2 Replies)
Discussion started by: magikminox
2 Replies

10. Shell Programming and Scripting

Generate csv file

I have a file which has some thousand records in the following format File: input.txt -> <option value="14333">VISWANADH VELAMURI</option> <option value="17020">VISWANADHA RAMA KRISHNA</option> I want to generate a csv file from the above file as follows File: output.txt -> ... (4 Replies)
Discussion started by: rahulrathod
4 Replies
Login or Register to Ask a Question