Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

I want to append data to same .csv file.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting I want to append data to same .csv file.
# 1  
Old 09-05-2010
I want to append data to same .csv file.
I have a script which has to be scheduled to run 3 times a day.
My script picks the required fields from logfile and stores the data in a.csv file.
Sample data.

my logfile contain:

Code:
0097A,0374D,100903,1519,00000606191
0097A,C88RA,100903,0724,00000606105

So the output of first execution will be stored in a.csv file:
Code:
0097A,0374D,100903,1519,00000606191
0097A,C88RA,100903,0724,00000606105

Before second execution of script now logfile contain:
Code:
0097A,0374D,100903,1519,00000606191
0097A,C88RA,100903,0724,00000606105
0097A,C88RA,100903,0726,00000606106
0100A,0374A,100903,1358,00000355348
0115A,6652A,100903,1103,00000489132
0118A,6652A,100903,1301,00000339704

So my output should be appended to a.csv file keeping old data and appending new data to it with some line and time in between as below:
I want my output like this.

Code:
0097A,0374D,100903,1519,00000606191
0097A,C88RA,100903,0724,00000606105
-------------------2nd Phase $time--------------
0097A,C88RA,100903,0726,00000606106
0100A,0374A,100903,1358,00000355348
0115A,6652A,100903,1103,00000489132
0118A,6652A,100903,1301,00000339704

---------- Post updated at 05:55 AM ---------- Previous update was at 05:48 AM ----------

Code:
#!/bin/sh
time="`date '+%H%M%S'`"
date=`date +%y%m%d -d"1 day ago"`
inbound_dir=/vist/logfiles/to_solmis
mkdir -p /vist/sumit/in_ASN.$date.$time
cp `grep -il ST~856~ $inbound_dir/*$date*` /vist/sumit/in_ASN.$date.$time/
        for i in /vist/sumit/in_ASN.$date.$time/*
        do
            #cp `echo $i` /vis/sumit/inbound.$date
            tr -c '[A-Z] [0-9] [a-z] ~' '[ *]' <$i > b
            awk -F "DTM~" '{print $1}' b > c
            awk -F "~SH~" '{print $2}' c > d
            awk -F "~" '{print $1","$2","$3","$4","$11}' d >> a.csv.$date.$time
       done
            rm b
            rm c
            rm d

Moderator's Comments:
Mod Comment I don't know where you copy and paste your output from, but could you please stop using it! Every single line contains font and size tags, and it's a nightmare to edit. Thank you.
Last edited by Scott; 09-05-2010 at 08:09 AM.. Reason: Code tags, please...
# 2  
Old 09-05-2010
Not sure what you are trying to do with all those awk invocations in your shell script, but if you want to pick up the delta from an incrementing logfile then you could do something like this:

Code:
diff logfile csvfile | awk 'BEGIN{system("date '+%H%M%S'")} /^</{print $2}'

The diff command displays the delta records in the logfile with a preceding "<"; you filter those and redirect to your csvfile with the date on top.

tyler_durden

In case you want to embellish the delta header like so -

Code:
--- Phase 2 HH24MISS ---

then you'll have to figure out the last "phase" number first, increment it and then append it followed by the date and delta records.

This awk one-liner returns 2 if a phase isn't present in the csvfile, otherwise it returns the incremented phase number.

Code:
awk '/^--/{x=$3} END{print x==0?2:x+1}' csvfile

So you could assign it to a shell variable and pass that to your awk script like so -

Code:
PHASE=$(awk '/^--/{x=$3} END{print x==0?x+2:x+1}' csvfile)
diff logfile csvfile | awk -v P=$PHASE 'BEGIN{printf "--- Phase " P; system("printf \" %s ---\n\" `date '+%H%M%S'` ")} /^</{print $2}' >>csvfile

And finally, if the logfile doesn't exist then you simply copy csvfile to logfile. (This is the simplest part, and you should be able to do this on your own.)
For all other cases, the script posted above should work.

HTH,
tyler_durden
Last edited by durden_tyler; 09-05-2010 at 03:40 PM..
# 3  
Old 09-07-2010
I have to append the new data received from logfile to csvfile
Actually my logfile contain a file,so using awk command I am picking the required data and storing in csvfile.
Logfile is not the same as csvfile.
csvfile contain some fields that are reuired.

So I cant differentiate logfile and csvfile.
As below mentioned in my script.

I want append data to a.csvfile
# 4  
Old 09-07-2010
Quote:
Originally Posted by shrima.pratima
Actually my logfile contain a file,so using awk command I am picking the required data and storing in csvfile.
Logfile is not the same as csvfile.
csvfile contain some fields that are reuired.

So I cant differentiate logfile and csvfile.
As below mentioned in my script.

I want append data to a.csvfile
Sorry, the following line in my post:

Code:
And finally, if the logfile doesn't exist then you simply copy csvfile to logfile.

should read:

Code:
And finally, if the csvfile doesn't exist then you simply copy logfile to csvfile.

For the data you've provided, I believe diff should work.
If the data you've provided doesn't match your actual data (or the one that your script works on), then modify your example and show us exactly what you are trying to pick up from your logfile.

Finally, csvfile in my post should be considered a.csv (which was "a csv file" in your first post).

tyler_durden
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Python script to run multiple command and append data in output csv file

Experts, I am writing a script and able to write only small piece of code and not able to collect logic to complete this task. In input file have to look for name like like this (BGL_HSR_901_1AG_A_CR9KTR10) before sh iss neors. Record this (BGL_HSR_901_1AG_A_CR9KTR10) in csv file Now have to... (0 Replies)
Discussion started by: as7951
0 Replies

2. Shell Programming and Scripting

Matching two fields in two csv files, create new file and append match

I am trying to parse two csv files and make a match in one column then print the entire file to a new file and append an additional column that gives description from the match to the new file. If a match is not made, I would like to add "NA" to the end of the file Command that Ive been using... (6 Replies)
Discussion started by: dis0wned
6 Replies

3. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07... (8 Replies)
Discussion started by: refrain
8 Replies

4. Shell Programming and Scripting

Append data to new row in CSV file every day

Hi All I will run the same script every day in corn and output should go to same CSV file but in different row with dates on it. Below is my example in attached format. Script i am using to collect switch port online DATE=`date '+%d-%m-%y'` for f in `cat... (1 Reply)
Discussion started by: ranjancom2000
1 Replies

5. Shell Programming and Scripting

want to append the data in one file to the another

Hi , i have two log files, i need to combine this as a one log file. i need to do this by SED , test1.log sadadadaadfsaf test2.log adadadadadada i need this in a single file from test 1 to test2.log test2.log(expected result) adadadadadada (7 Replies)
Discussion started by: mhdmehraj
7 Replies

6. Shell Programming and Scripting

How to append value at first line of CSV file using shell script?

I have an issue where I need to append a value at the last of the csv, I have created a shell script and it is appending the columns at the last but it is appending at all lines, and my requirement is specific to just append at the 1st line. Have a look and suggest, (7 Replies)
Discussion started by: anujrichhariya
7 Replies

7. Shell Programming and Scripting

Append Header in CSV file

Hi, I create a csv file and the output looks like below Arun,E001 Sathish,E003 Now i need to include the below header and the output should like below Name,Number Arun,E001 Sathish,E003 Please guide me. Thanks (4 Replies)
Discussion started by: Sekar1
4 Replies

8. Shell Programming and Scripting

how to append line of of data to file

hai..i am new to unix..and i've currently learn shell script.. i have this small problem where i would like to save every data from log file into user directory if the data is equal to the name of the user.. i manage to do that with below script.. i would like to ask if there is any solutions so... (1 Reply)
Discussion started by: meggae
1 Replies

9. UNIX for Dummies Questions & Answers

append data to file

i want to develop a script newdata that writes new data to a file called items the file items has the following headings columns separated by tabs: channel date time programe if i type executable file newdata on the command line with parameters, it should append it to the items files the... (1 Reply)
Discussion started by: fletcher
1 Replies

10. Shell Programming and Scripting

Need to append file name to all rows in in .csv file

Hi , Can some one help in appending the file name to all the rows in .csv files the current work is like this. This is adding a new line for file name, I need to append file name to all lines in .csv for i in `ls $filename*.csv` do echo "$i" > ./tmpfile cat "$i" >> ./tmpfile mv... (3 Replies)
Discussion started by: Satyagiri
3 Replies
Login or Register to Ask a Question