Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to Gather data from logs and export to a CSV file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to Gather data from logs and export to a CSV file
# 1  
Old 11-21-2016
Question Script to Gather data from logs and export to a CSV file
Greetings,

After a few hours of trial and error, I decide to ask for some help.

I am new to AWK and shell script, so please don't laugh Smilie

I made the below script, to gather data from some logs and have the output into a CSV file :

Code:
#!/bin/sh
#Script to collect Errors

file="/Path/Errors_Report_$(date +%Y%m%d).csv"

DAY=`date +%d -d "Yesterday"`

#Header of the CSV file

echo -e "User_ID,Host_name,Date,Time,Code_302,Code_304,Code_305,Code_1,Code_2,Code_3,Code_7,Code_10,Code_18,Code_101,Code_201,Code_300,Code_301,Code_303,Code_306,Code_14,Code_9999,Code_19,Code_9,Code_11" > /path/Error_Report_$(date +%Y%m%d).csv

grep 'Pattern' Path/File_$(date +%Y%m%d)* |awk -F "|" '{if ($28~/ Pattern /) print substr($1,74,10)","substr($1,85,2)","$28; else if($29~/ Pattern /) print substr($1,74,10)","substr($1,85,2)","$29; else if($30~/ Pattern /) print substr($1,74,10)","substr($1,85,2)","$30}' | cut -d, -f1,2,10 | awk -F "," '{if ($3~/ Error1/) print $1","$2","302; else if ($3~/ Error2/) print $1","$2","304; else if ($3~/ Error3/) print $1","$2","305; else if ($3~/ Error4/) print $1","$2","1; else if ($3~/ Error5/) print $1","$2","2; else if ($3~/ Error6./) print $1","$2","3; else if ($3~/ Error7./) print $1","$2","7; else if ($3~/ Error 8/) print $1","$2","10; else if ($3~/ Error./) print $1","$2","18; else if ($3~/ Error 9/) print $1","$2","101; else if ($3~/ Error 10/) print $1","$2","201; else if ($3~/ Error11/) print $1","$2","300; else if ($3~/ Error12/) print $1","$2","301; else if ($3~/ Error13/) print $1","$2","303; else if ($3~/ Error14/) print $1","$2","306; else if ($3~/ Error15/) print $1","$2","14; else if ($3~/ Error16/) print $1","$2","9999; else if ($3~/ Error17./) print $1","$2","19; else if ($3~/ Error18/) print $1","$2","9; else if ($3~/ Error19/) print $1","$2","11}' | sort | uniq -c | awk -F" " '{print $1","$2}' |  awk -F "," '{if($4==302) print $2","$3","$1",,,,,,,,,,,,,,,,,,,"; else if($4==304) print $2","$3",,"$1",,,,,,,,,,,,,,,,,,"; else if($4==305) print $2","$3",,,"$1",,,,,,,,,,,,,,,,,"; else if($4==1) print $2","$3",,,,"$1",,,,,,,,,,,,,,,,"; else if($4==2) print $2","$3",,,,,"$1",,,,,,,,,,,,,,,"; else if($4==3) print $2","$3",,,,,,"$1",,,,,,,,,,,,,,"; else if($4==7) print $2","$3",,,,,,,"$1",,,,,,,,,,,,,"; else if($4==10) print $2","$3",,,,,,,,"$1",,,,,,,,,,,,"; else if($4==18) print $2","$3",,,,,,,,,"$1",,,,,,,,,,,"; else if($4==101) print $2","$3",,,,,,,,,,"$1",,,,,,,,,,"; else if($4==201) print $2","$3",,,,,,,,,,,"$1",,,,,,,,,"; else if($4==300) print $2","$3",,,,,,,,,,,,"$1",,,,,,,,"; else if($4==301) print $2","$3",,,,,,,,,,,,,"$1",,,,,,,"; else if($4==303) print $2","$3",,,,,,,,,,,,,,"$1",,,,,,"; else if($4==306) print $2","$3",,,,,,,,,,,,,,,"$1",,,,,"; else if($4==14) print $2","$3",,,,,,,,,,,,,,,,"$1",,,,"; else if($4==9999) print $2","$3",,,,,,,,,,,,,,,,,"$1",,,"; else if($4==19) print $2","$3",,,,,,,,,,,,,,,,,,"$1","; else if($4==9) print $2","$3",,,,,,,,,,,,,,,,,,,"$1","; else if($4==11) print $2","$3",,,,,,,,,,,,,,,,,,,,"$1}' | awk -F"," '{print "2001,Server1,"$1","$2":00:00,"$3","$4","$5","$6","$7","$8","$9","$10","$11","$12","$13","$14","$15","$16","$17","$18","$19","$20","$21","$22}' >> /Path/Error_Report_$(date +%Y%m%d).csv

The output is like below :

Code:
User_ID,Host_name,Date,Time,Code_302,Code_304,Code_305,Code_1,Code_2,Code_3,Code_7,Code_10,Code_18,Code_101,Code_201,Code_300,Code_301,Code_303,Code_306,Code_14,Code_9999,Code_19,Code_9,Code_11
2001,Server1,2016-11-18,00:00:00,,,,,,,,,,,,,,,,,,,,5
2001,Server1,2016-11-18,00:00:00,1,,,,,,,,,,,,,,,,,,,
2001,Server1,2016-11-18,00:00:00,,,,,,,,,,,,,,1,,,,,,
2001,Server1,2016-11-18,00:00:00,,,,,,,,,,,,,,,,,,,42,
2001,Server1,2016-11-18,01:00:00,,,,,,,,,,,,,,,,,,,,3
2001,Server1,2016-11-18,01:00:00,2,,,,,,,,,,,,,,,,,,,
2001,Server1,2016-11-18,01:00:00,,,,,,,,,,,,,,3,,,,,,
2001,Server1,2016-11-18,01:00:00,,,,,,,,,,,,,,,,,,,21,

The plan was to gather all data and have the amount of errors found from a given time in just one line like below, grouping by hours all errors in just one line :

Code:
User_ID,Host_name,Date,Time,Code_302,Code_304,Code_305,Code_1,Code_2,Code_3,Code_7,Code_10,Code_18,Code_101,Code_201,Code_300,Code_301,Code_303,Code_306,Code_14,Code_9999,Code_19,Code_9,Code_11
2001,smom01brb.rio.oi.ombr,2016-11-18,00:00:00,1,,,,,,,,,,,,1,,,,,,42,5

And that is what I am not able to achieve with my knowledge, which is very limited by the way.

How could I make the script output the information in the desired format ? All errors from a time set, just in one line, instead of one line from each errors counted ?

Any sugestion to fix the script I did are welcome and any other script which accomplish that is more than welcome.

Thanks for you time in reading my doubt / request and thanks in advanced for all sugestions.
# 2  
Old 11-21-2016
That has to be close to the record for the longest 1-liner posted on this site.

I've formatted your posted code below to make it a little more readable.

Some questions:

1) Can you supply some example input.
2) Is " Pattern " fixed text or should this be an argument to your script?
3) Should "2001" and "Server1" be hard-coded in the final awk script, extracted from the input, sourced from a configuration file or passed as arguments to the script?


I feel the error strings you are searching for in the 3rd awk script in the process chain (eg "Error 10", "Error17.",etc) should be setup in a configuration file with pattern and output position. Perhaps something like this:

Code:
ERROR_PATTERN,OUTPUT_POSITION
Error1,1
Error 10,11
Error16,18

Code:
echo -e "User_ID,Host_name,Date,Time,Code_302,Code_304,Code_305,"\
"Code_1,Code_2,Code_3,Code_7,Code_10,Code_18,Code_101,Code_201,"\
"Code_300,Code_301,Code_303,Code_306,Code_14,Code_9999,Code_19,"\
"Code_9,Code_11" > /path/Error_Report_$(date +%Y%m%d).csv

grep 'Pattern' Path/File_$(date +%Y%m%d)* |awk -F "|" '{
     if($28~/ Pattern /) print substr($1,74,10)","substr($1,85,2)","$28;
else if($29~/ Pattern /) print substr($1,74,10)","substr($1,85,2)","$29;
else if($30~/ Pattern /) print substr($1,74,10)","substr($1,85,2)","$30}' |
cut -d, -f1,2,10 |
awk -F "," '{
     if ($3~/ Error1/)   print $1","$2","302;
else if ($3~/ Error2/)   print $1","$2","304;
else if ($3~/ Error3/)   print $1","$2","305;
else if ($3~/ Error4/)   print $1","$2","1;
else if ($3~/ Error5/)   print $1","$2","2;
else if ($3~/ Error6./)  print $1","$2","3;
else if ($3~/ Error7./)  print $1","$2","7;
else if ($3~/ Error 8/)  print $1","$2","10;
else if ($3~/ Error./)   print $1","$2","18;
else if ($3~/ Error 9/)  print $1","$2","101;
else if ($3~/ Error 10/) print $1","$2","201;
else if ($3~/ Error11/)  print $1","$2","300;
else if ($3~/ Error12/)  print $1","$2","301;
else if ($3~/ Error13/)  print $1","$2","303;
else if ($3~/ Error14/)  print $1","$2","306;
else if ($3~/ Error15/)  print $1","$2","14;
else if ($3~/ Error16/)  print $1","$2","9999;
else if ($3~/ Error17./) print $1","$2","19;
else if ($3~/ Error18/)  print $1","$2","9;
else if ($3~/ Error19/)  print $1","$2","11}' | sort | uniq -c |
awk -F" " '{print $1","$2}' |
awk -F "," '{
     if($4==302)  print $2","$3","$1",,,,,,,,,,,,,,,,,,,";
else if($4==304)  print $2","$3",,"$1",,,,,,,,,,,,,,,,,,";
else if($4==305)  print $2","$3",,,"$1",,,,,,,,,,,,,,,,,";
else if($4==1)    print $2","$3",,,,"$1",,,,,,,,,,,,,,,,";
else if($4==2)    print $2","$3",,,,,"$1",,,,,,,,,,,,,,,";
else if($4==3)    print $2","$3",,,,,,"$1",,,,,,,,,,,,,,";
else if($4==7)    print $2","$3",,,,,,,"$1",,,,,,,,,,,,,";
else if($4==10)   print $2","$3",,,,,,,,"$1",,,,,,,,,,,,";
else if($4==18)   print $2","$3",,,,,,,,,"$1",,,,,,,,,,,";
else if($4==101)  print $2","$3",,,,,,,,,,"$1",,,,,,,,,,";
else if($4==201)  print $2","$3",,,,,,,,,,,"$1",,,,,,,,,";
else if($4==300)  print $2","$3",,,,,,,,,,,,"$1",,,,,,,,";
else if($4==301)  print $2","$3",,,,,,,,,,,,,"$1",,,,,,,";
else if($4==303)  print $2","$3",,,,,,,,,,,,,,"$1",,,,,,";
else if($4==306)  print $2","$3",,,,,,,,,,,,,,,"$1",,,,,";
else if($4==14)   print $2","$3",,,,,,,,,,,,,,,,"$1",,,,";
else if($4==9999) print $2","$3",,,,,,,,,,,,,,,,,"$1",,,";
else if($4==19)   print $2","$3",,,,,,,,,,,,,,,,,,"$1",";
else if($4==9)    print $2","$3",,,,,,,,,,,,,,,,,,,"$1",";
else if($4==11)   print $2","$3",,,,,,,,,,,,,,,,,,,,"$1}' |
awk -F"," '{
  print "2001,Server1,"$1","$2":00:00,"$3","$4","$5","$6","$7",
  "$8","$9","$10","$11","$12","$13","$14","$15","$16","$17",
  "$18","$19","$20","$21","$22}' >>
  /Path/Error_Report_$(date +%Y%m%d).csv

This User Gave Thanks to Chubler_XL For This Post:
Yagami_Sama
# 3  
Old 11-21-2016
Thanks for replying back.

Answering your questions :

Some questions:

1) Can you supply some example input.

The information is gathered from a log entry similar to the one below, and there are millions of it per day :

Code:
2016-11-18 14:44:31,728:        |Thread_180|1|CUSTOMER_ID|B|UNIQUE_ID|ACT|A|null|1.99|MODE|mmp|2016-11-18 14:12:12.0|2016-11-18 14:12:12.0|null|0|R|0|1|null|N|-|-1|||null|SITE_NAME|[TID=3,1.99,,-,,,-,-ERROR_CODE,1,0.0,,,]|[CBCK=96,-,,,-#not required]|[BAL=100,1.99,,-,,,-,-#Continue,1,0.0,,,]|[CONTENT_ID=actinfo=MODE;null;null;IP;USER/protocolnumber;0000000000000000/,cosid:1]|[CAMPAIGN_ID=null]|[TOTAL_AMNT=null]|[RECO:refId=00000000-0000-0000-0000-000000000000,chrg_refid=000000000000000000]|[TSK = Q,0,00,2016-11-19 14:12:12.0,2016-11-18 14:12:12.0]

2) Is " Pattern " fixed text or should this be an argument to your script?

"Pattern" is what I want to grep, it is the "TID" I marked in red on the log example above.

3) Should "2001" and "Server1" be hard-coded in the final awk script, extracted from the input, sourced from a configuration file or passed as arguments to the script?


Yes. They are fixed value. This script will run in multiple servers, and in each server this information is different. In this case I pass this information here :

print "2001,Server1...


Talking about the code.

The first part the one starting with the :

Code:
"Echo -e"

is the header of the CSV file.

So I must keep this.

The first part of the script starting here :

Code:
grep 'Pattern' Path/File_$(date +%Y%m%d)*

...

Is where I grep all varitions of this :

Code:
[CHG=3,1.99,,-,,,-,-ERROR_CODE,1,0.0,,,]


ERROR_CODE will vary from case to case, there are several error, some of short description and other with a long description.

The Second Part :

Code:
 awk -F "," '{
     if ($3~/ Error1/)

Going from Error1 to Error19 is where I count the errors, based on the description.


And third part :

Code:
awk -F "," '{
     if($4==302)

Is where my doubts start. Here is where I pass the format to a CSV file . here :

Code:
else if($4==11)   print $2","$3",,,,,,,,,,,,,,,,,,,,"$1}' |

Is where I pass the amount of "Error 11" to the last position of the output file.


And the last part :

Code:
awk -F"," '{
  print "2001,Server1,"$1","$2":00:00...

Is where I count all errors I gathered, considering that 2001, server1 are fixed and the :00:00 is the time.

I will upload one sample of the report.

And sorry for the long post. I just tried to put as much information as I could.
ERROR_Report.rar (436 Bytes) 
# 4  
Old 11-23-2016
This might get you started.

It uses a file called cfg which will contain your error codes along with their output position and column heading.

For testing I used this:

Code:
TEST,2,Code_304
THIS,3,Code_305
ERROR_CODE,12,Code_300
THAT,20,Code_11

Program is a follows:
Code:
PAT="TID"
SERV="2001,Server1"

awk -v PAT="$PAT" -v SERV="$SERV" -F "," '
FNR==NR{ P[$2]=$1 ; Heading[$2]=$3; FldMax=$2>FldMax?$2:FldMax ; next }
$0 ~ PAT {

   DTM=$1
   #Truncate Mins and Secs from DTM - we want totals per hour
   gsub(/:.*/,":00:00",DTM)

   #replace space between date and time with comma
   gsub(/ /,",",DTM)

   #build list of date/times we found in the log
   if (!(DTM in D)) {
      D[DTM];Times[++L]=DTM
   }
   # Test fields 28 thru 30 for PAT match and count per DTM value
   for(i=28;i<30;i++)
     if ($i ~ PAT) {
        split(substr($1,74,10)","substr($1,85,2)","$i, V, ",");
        for(j in P) if (V[10] ~ P[j]) {
             C[DTM,j]++
             next
        }
        next
     }
}
END {
  printf "%s","User_ID,Host_name,Date,Time"
  for(field=1;field<=FldMax;field++)
     printf ",%s", Heading[field]
  printf "\n"
  for(line=1;line<=L;line++) {
    printf "%s,%s",SERV,Times[line];
    for(field=1;field<=FldMax;field++)
       printf ",%s",C[Times[line],field]
       printf "\n"
  }
}' cfg FS="|" ./File_$(date +%Y%m%d)* > ./Error_Report_$(date +%Y%m%d).csv

With this input:
Code:
2016-11-18 14:44:31,728:        |Thread_180|1|CUSTOMER_ID|B|UNIQUE_ID|ACT|A|null|1.99|MODE|mmp|2016-11-18 14:12:12.0|2016-11-18 14:12:12.0|null|0|R|0|1|null|N|-|-1|||null|SITE_NAME|[TID=3,1.99,,-,,,-,-ERROR_CODE,1,0.0,,,]|[CBCK=96,-,,,-#not required]|[BAL=100,1.99,,-,,,-,-#Continue,1,0.0,,,]|[CONTENT_ID=actinfo=MODE;null;null;IP;USER/protocolnumber;0000000000000000/,cosid:1]|[CAMPAIGN_ID=null]|[TOTAL_AMNT=null]|[RECO:refId=00000000-0000-0000-0000-000000000000,chrg_refid=000000000000000000]|[TSK = Q,0,00,2016-11-19 14:12:12.0,2016-11-18 14:12:12.0]
2016-11-18 14:50:31,728:        |Thread_180|1|CUSTOMER_ID|B|UNIQUE_ID|ACT|A|null|1.99|MODE|mmp|2016-11-18 14:12:12.0|2016-11-18 14:12:12.0|null|0|R|0|1|null|N|-|-1|||null|SITE_NAME|[TID=3,1.99,,-,,,-,-ERROR_CODE,1,0.0,,,]|[CBCK=96,-,,,-#not required]|[BAL=100,1.99,,-,,,-,-#Continue,1,0.0,,,]|[CONTENT_ID=actinfo=MODE;null;null;IP;USER/protocolnumber;0000000000000000/,cosid:1]|[CAMPAIGN_ID=null]|[TOTAL_AMNT=null]|[RECO:refId=00000000-0000-0000-0000-000000000000,chrg_refid=000000000000000000]|[TSK = Q,0,00,2016-11-19 14:12:12.0,2016-11-18 14:12:12.0]
2016-11-18 14:54:31,728:        |Thread_180|1|CUSTOMER_ID|B|UNIQUE_ID|ACT|A|null|1.99|MODE|mmp|2016-11-18 14:12:12.0|2016-11-18 14:12:12.0|null|0|R|0|1|null|N|-|-1|||null|SITE_NAME|[TID=3,1.99,,-,,,-,-THIS,1,0.0,,,]|[CBCK=96,-,,,-#not required]|[BAL=100,1.99,,-,,,-,-#Continue,1,0.0,,,]|[CONTENT_ID=actinfo=MODE;null;null;IP;USER/protocolnumber;0000000000000000/,cosid:1]|[CAMPAIGN_ID=null]|[TOTAL_AMNT=null]|[RECO:refId=00000000-0000-0000-0000-000000000000,chrg_refid=000000000000000000]|[TSK = Q,0,00,2016-11-19 14:12:12.0,2016-11-18 14:12:12.0]
2016-11-18 14:55:31,728:        |Thread_180|1|CUSTOMER_ID|B|UNIQUE_ID|ACT|A|null|1.99|MODE|mmp|2016-11-18 14:12:12.0|2016-11-18 14:12:12.0|null|0|R|0|1|null|N|-|-1|||null|SITE_NAME|[TID=3,1.99,,-,,,-,-THAT,1,0.0,,,]|[CBCK=96,-,,,-#not required]|[BAL=100,1.99,,-,,,-,-#Continue,1,0.0,,,]|[CONTENT_ID=actinfo=MODE;null;null;IP;USER/protocolnumber;0000000000000000/,cosid:1]|[CAMPAIGN_ID=null]|[TOTAL_AMNT=null]|[RECO:refId=00000000-0000-0000-0000-000000000000,chrg_refid=000000000000000000]|[TSK = Q,0,00,2016-11-19 14:12:12.0,2016-11-18 14:12:12.0]
2016-11-18 16:44:31,728:        |Thread_180|1|CUSTOMER_ID|B|UNIQUE_ID|ACT|A|null|1.99|MODE|mmp|2016-11-18 14:12:12.0|2016-11-18 14:12:12.0|null|0|R|0|1|null|N|-|-1|||null|SITE_NAME|[TID=3,1.99,,-,,,-,-THAT,1,0.0,,,]|[CBCK=96,-,,,-#not required]|[BAL=100,1.99,,-,,,-,-#Continue,1,0.0,,,]|[CONTENT_ID=actinfo=MODE;null;null;IP;USER/protocolnumber;0000000000000000/,cosid:1]|[CAMPAIGN_ID=null]|[TOTAL_AMNT=null]|[RECO:refId=00000000-0000-0000-0000-000000000000,chrg_refid=000000000000000000]|[TSK = Q,0,00,2016-11-19 14:12:12.0,2016-11-18 14:12:12.0]

I got this output

Code:
User_ID,Host_name,Date,Time,,Code_304,Code_305,,,,,,,,,Code_300,,,,,,,,Code_11
2001,Server1,2016-11-18,14:00:00,,,1,,,,,,,,,2,,,,,,,,1
2001,Server1,2016-11-18,16:00:00,,,,,,,,,,,,,,,,,,,,1

Last edited by Chubler_XL; 11-23-2016 at 04:59 AM.. Reason: Moved headings into cfg file
This User Gave Thanks to Chubler_XL For This Post:
Yagami_Sama
# 5  
Old 11-23-2016
Thank you very much for the code and sorry for the delay .

Your code is light years ahead of anything I did in awk, actually my awk knowledge is limited to “print $”.

And after a few hours trying to make your code work, unfortunately I it did not run here.
Probably I did not understand you code properly, but I am getting this :


Code:
awk: billing_errors.awk:10: awk -v PAT="$PAT" -v SERV="$SERV" -F "," '
awk: billing_errors.awk:10:           ^ syntax error
awk: billing_errors.awk:10: awk -v PAT="$PAT" -v SERV="$SERV" -F "," '
awk: billing_errors.awk:10:                          ^ syntax error
awk: billing_errors.awk:10: awk -v PAT="$PAT" -v SERV="$SERV" -F "," '

removing the ‘ from here :

Code:
awk -v PAT="$PAT" -v SERV="$SERV" -F "," '

gave another errors :

Code:
awk: billing_errors.awk:10: awk -v PAT="$PAT" -v SERV="$SERV" -F ","
awk: billing_errors.awk:10:           ^ syntax error
awk: billing_errors.awk:10: awk -v PAT="$PAT" -v SERV="$SERV" -F ","
awk: billing_errors.awk:10:                          ^ syntax error
awk: billing_errors.awk:47: } cfg FS="|" /var/log/ERROR..log > /var/report/Billing_Errors_$(date +%Y%m%d).csv
awk: billing_errors.awk:47:    ^ syntax error
awk: billing_errors.awk:47: } cfg FS="|" /var/log/ERROR..log > /var/report/Billing_Errors_$(date +%Y%m%d).csv
awk: billing_errors.awk:47:                  ^ syntax error
awk: billing_errors.awk:47: } cfg FS="|" /var/log/ERROR..log > /var/report/Billing_Errors_$(date +%Y%m%d).csv
awk: billing_errors.awk:47:                             ^ syntax error
awk: billing_errors.awk:47: } cfg FS="|" /var/log/ERROR..log > /var/report/Billing_Errors_$(date +%Y%m%d).csv
awk: billing_errors.awk:47:                                                                               ^ syntax error
awk: billing_errors.awk:47: } cfg FS="|" /var/log/ERROR..log > /var/report/Billing_Errors_$(date +%Y%m%d).csv
awk: billing_errors.awk:47:                                                                       ^ syntax error
awk: billing_errors.awk:47: } cfg FS="|" /var/log/ERROR..log > /var/report/Billing_Errors_$(date +%Y%m%d).csv
awk: billing_errors.awk:47:                                                                             ^ syntax error

I created a cfg file , following your instructions and the file is in the same path as the script.

Probably I did something wrong, but could you please advise what I did wrong and where I will inform to use the cfg file and from where it will read.

Thanks again.
# 6  
Old 11-23-2016
What I guess you have done is to copy and past this shell script into a file you named billing_errors.awk and then ran it with awk -f or #!/usr/bin/awk

The code block under "Program is as follows" in my post above is a shell script that invokes awk from line 4. Sorry for the confusion, I should have started it with #!/bin/sh to make that clear.


Save the script as billing_errors.sh and run it within a shell.
This User Gave Thanks to Chubler_XL For This Post:
Yagami_Sama
# 7  
Old 11-23-2016
Quote:
Originally Posted by Chubler_XL
What I guess you have done is to copy and past this shell script into a file you named billing_errors.awk and then ran it with awk -f or #!/usr/bin/awk

The code block under "Program is as follows" in my post above is a shell script that invokes awk from line 4. Sorry for the confusion, I should have started it with #!/bin/sh to make that clear.


Save the script as billing_errors.sh and run it within a shell.
Now it ran smoothly but it did not bring the output, actually it found nothing ...

Probably I messed with files, so if you could be kind and explain one more thing :

Here :

Code:
}' cfg FS="|" ./File_$(date +%Y%m%d)* > ./Error_Report_$(date +%Y%m%d).csv

The cfg is the file in which I put the errors right and the ./File_$(date +%Y%m%d)* will be the log files in which it will look for the parameter ?

Because it is taking forever to run or is not bringing any result when depending of the file I mentioned there.

Thanks again !
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Linux

Parsing - export html table data as .csv file?

Hi all, Is there any out there have a brilliant idea on how to export html table data as .csv or write to txt file with separated comma and also get the filename of link from every table and put one line per rows each table. Please see the attached html and PNG of what it looks like. ... (7 Replies)
Discussion started by: lxdorney
7 Replies

2. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07... (8 Replies)
Discussion started by: refrain
8 Replies

3. Shell Programming and Scripting

Shell script to extract data from csv file

Hi everyone, I have a csv file which has data with different heading and column names as below. Static Data Ingested ,,,,,,,,,,,,Known Explained Rejections Column_1,column_2,Column_3,Column_4,,Column_6,Column_7,,% Column_8,,Column_9 ,Column_10 ,... (14 Replies)
Discussion started by: Vivekit82
14 Replies

4. UNIX for Dummies Questions & Answers

Shell script to extract data from csv file

Hi Guys, I am new to shell script.I need your help to write a shell script. I need to write a shell script to extract data from a .csv file where columns are ',' separated. The file has 7 columns having values say column 1,column 2.....column 7 as below along with their values. Name, Address,... (7 Replies)
Discussion started by: Vivekit82
7 Replies

5. Shell Programming and Scripting

Run script to export the data to ixf file in loop

Hi, I am trying to export the data to an .ixf file. I have read the table names from a .dat file and those table name should be passed to the select * from schema.TABLENAME query . I am trying the below loop while read TABLE; do db2 EXPORT TO ~/data_export/$TABLE.ixf OF IXF MESSAGES... (5 Replies)
Discussion started by: vikyalex4
5 Replies

6. Shell Programming and Scripting

Exporting data as a CSV file from Unix shell script

Friends...This is the first time i am trying the report generation using shell script... any suggestions are welcome. Is there a way to set the font size & color when i am exporting the data from unix shell script as a CSV file ? The following sample data is saved as a .csv file in the... (2 Replies)
Discussion started by: appu2176
2 Replies

7. Web Development

script to load data from csv file

hello i want a script to load the data line by line from a csv file into a mysql table (3 Replies)
Discussion started by: srpa01red
3 Replies

8. Shell Programming and Scripting

FTP script to gather logs.

Hi, I am currently working on a FTP script which would ftp to extranet servers from intranet server and gather logs based on timestamps in archive logs and by using a wildcard character in the present logs. I have the following in place AAA="Extranet Server 1:Instance Extranet Server... (2 Replies)
Discussion started by: openspark
2 Replies

9. Shell Programming and Scripting

Shell Script to Load data into the database using a .csv file and .ctl file

Since i'm new to scripting i'm findind it difficult to code a script. The script has to be an executable with 2 paramters passed to it.The Parameters are 1. The Control file name(.ctl file) 2. The Data file name(.csv file) Does anybody have an idea about it? :confused: (3 Replies)
Discussion started by: Csmani
3 Replies

10. Shell Programming and Scripting

unix script to export data from csv file to oracle database

Hello people, Need favour. The problem I have is that, I need to develop a unix shell script that performs recurring exports of data from a csv file to an oracle database. Basically, the csv file contains just the first name and last name will be dumped to an Unix server. The data from these... (3 Replies)
Discussion started by: vinayagan
3 Replies
Login or Register to Ask a Question