Parse log file to insert into database


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parse log file to insert into database
# 8  
Old 08-16-2015
Here's what I have so far for the entire process.

crontab -e
Code:
0 0 * * * /scripts/collect_cpu.sh
0 * * * * /scripts/process_cpu.sh


collect_cpu.sh
Code:
#!/bin/bash

sar -u 300 288 >> /var/log/mylogs/$(date "+%Y-%m-%d")_$(hostname)_cpuusage.log

process_cpu.sh
Code:
#!/bin/bash

# Change directory to HNP logs
cd /var/log/mylogs/

# Loop over each file
for logFile in *_cpuusage.log
  do
    awk '
      BEGIN { OFS = "," } FNR == 1 {
        split(FILENAME,F,/_/)
        d=F[1]
        h=F[2]
      }
      $2 ~ /^[AP]M$/ && $3 != "CPU" {
        split($1, HMS, /:/)
	if(HMS[1] == 12)HMS[1] = 0
	if($2 == "PM")	HMS[1] += 12
	t = sprintf("%02d:%s:%s", HMS[1], HMS[2], HMS[3])
        print d, t, h, $4+$5, $6, $7+$8+$9
      }
    ' $logFile > /var/log/mylogs/$logFile.processed

  # load into mysql
  mysql -e "load data local infile '"$logFile.processed"' into table cpuusage fields TERMINATED BY ',' (hidden)

  # rename file done
  mv $logFile.processed $logFile.processed.done

  # mark .done for previous days
  if d < $(date "+%Y-%m-%d")
    mv $logFile $logFile.done
  endif

done

I can manage this process if I run the cpu process job for any day before the current day. However, I want to process this every 60 minutes.

Is there any scripting I can do to better manage this? I'm thinking it may be easier to just rename the current log file as .done and let the existing cron job recreate the cpu log file when it runs again the next 5 minutes, however, what if my process happens the same time as the cron? Seems i'll run into a file lock issue.
# 9  
Old 08-16-2015
Why don't you do that in one go, i.e. combine the two scripts into one? That would entirely decouple two runs, and the DB's locking mechanisms would certainly ensure two processes don't interact.

What be the execution times of collect_cpu.sh and process_cpu.sh.
# 10  
Old 08-16-2015
The first script kicks off once per day and runs every 5 minutes for 288 times. How can I combine both?

From what I read, if I just run 'sar' it gives me an average CPU since last reboot. Not since the last time it ran.

By logging every 5 minutes, I can get an idea of what the CPU was at a point in time and correlate it to the same 5 minute window my other web apps are executing. I'm also tracking CPU load, but that's a separate script.
# 11  
Old 08-16-2015
Please be more specific in what you want to change from what you are currently doing.

Does this code inprocess_cpu.sh:
Code:
  # mark .done for previous days
  if d < $(date "+%Y-%m-%d")
    mv $logFile $logFile.done
  endif

really work for you? In addition to missing a then, what does the d utility on your system do and what is in the file named 2015-08-16 when you run this command today?

If you want to update the database hourly, why are you creating one log file per day instead of one log file per hour? Why not run:
Code:
sar -u 300 12 > ($date '+%H.log')

in your hourly script and run the awk script on the resulting log file instead of reprocessing data being accumulated continuously during the day into a single file.
# 12  
Old 08-17-2015
Again, why don't you process the data immediately after creating/logging them?
# 13  
Old 08-17-2015
Quote:
Originally Posted by Don Cragun
Please be more specific in what you want to change from what you are currently doing.

Does this code inprocess_cpu.sh:
Code:
  # mark .done for previous days
  if d < $(date "+%Y-%m-%d")
    mv $logFile $logFile.done
  endif

really work for you? In addition to missing a then, what does the d utility on your system do and what is in the file named 2015-08-16 when you run this command today?

If you want to update the database hourly, why are you creating one log file per day instead of one log file per hour? Why not run:
Code:
sar -u 300 12 > ($date '+%H.log')

in your hourly script and run the awk script on the resulting log file instead of reprocessing data being accumulated continuously during the day into a single file.
Good point. I'll run both hourly. Is there a way to merge the hourly files into one per day?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read latest log files and perform database insert

Hi Experts, I have a situation where I need to write a shell script to continuously monitor a log directory with multiple log files and perform following: 1. Read the latest log file continuously and grep "Success" OR "Failure" 2. As it capture either Success or Failure, it has to perform a... (1 Reply)
Discussion started by: rish_max
1 Replies

2. Shell Programming and Scripting

How to read a text file line by line and insert into a database table?

I have a test file that I want to read and insert only certain lines into the the table based on a filter. 1. Rread the log file 12 Hours back Getdate() -12 Hours 2. Extract the following information on for lines that say "DUMP is complete" A. Date B. Database Name C.... (2 Replies)
Discussion started by: JolietJake
2 Replies

3. Shell Programming and Scripting

Parse through ~21,000 Database DDL statements -- Fastest way to perform search, replace and insert

Hello All: We are looking to search through 2000 files with around 21,000 statements where we have to search, replace and insert a pattern based on the following: 1) Parse through the file and check for CREATE MULTISET TABLE or CREATE SET TABLE statements.....and they always end with ON... (5 Replies)
Discussion started by: madhunk
5 Replies

4. Shell Programming and Scripting

Korn shell program to parse CSV text file and insert values into Oracle database

Enclosed is comma separated text file. I need to write a korn shell program that will parse the text file and insert the values into Oracle database. I need to write the korn shell program on Red Hat Enterprise Linux server. Oracle database is 10g. (15 Replies)
Discussion started by: shellguy
15 Replies

5. Web Development

INSERT data to a Database Table from a text file

If you have a text file and if you want to Insert data to your Database Table, You can do it with these queries LOAD DATA LOCAL INFILE '/path/yourTextFile.txt' INTO TABLE yourTableName FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n' (0 Replies)
Discussion started by: sitex
0 Replies

6. Shell Programming and Scripting

To Read a File and Insert a part of the lines into the database

Hi Guys I need to have a shell script which reads a log file and insert a part of each line into the database. Some sample lines in the file are as below. 20091112091359 MED_RQACK : user_data=60173054304,100232120,20091112091359,;ask_status=0;ask_reason=OK;msg_id=20091112091319... (5 Replies)
Discussion started by: Somanadh
5 Replies

7. Shell Programming and Scripting

Shell script to parse a line and insert a word

Hi All, I have a file like this, data1,data2,,,data5,data6. i want to write a shell script to replace data3 with "/example/string". which means my data file should look like this . data1,data2,example/string],,data5,data6. Could you guys help me to get a sed command or any other command... (8 Replies)
Discussion started by: girish.raos
8 Replies

8. Shell Programming and Scripting

how to insert data into database by reading it from a text file??

Hi....can you guys help me out in this script?? Below is a text file and it contains these: GEF001 000093625 MKL002510 000001 000000 000000 000000 000000 000000 000001 GEF001 000093625 MKL003604 000001 000000 000000 000000 000000 000000 000001 GEF001 000093625 MKL005675 000001... (4 Replies)
Discussion started by: pallavishetty
4 Replies

9. Shell Programming and Scripting

How to insert data into MYSql database from a text file

Hi, Need to get help from you guys about this issue. I need to insert data into MySql database from a text file which is located in other server. The text file is something look like below: Date | SubscriberNo | Call Duration 20/7/07 | 123456788 | 20 20/7/07 | 123412344 | 30 The... (4 Replies)
Discussion started by: shirleyeow
4 Replies

10. Shell Programming and Scripting

how to insert data in database based on text file?

Hi....can you guys help me out in this script?? Below is a text file script....called Bukom.txt and it contains these: BUKOM 20060101 2.5 2.6 2.7 2.8 2.9 2.3 2.1 BUKOM 20060102 2.4 2.5 2.6 2.7 2.7 2.6 2.4 BUKOM 20060103 2.1 ... (9 Replies)
Discussion started by: forevercalz
9 Replies
Login or Register to Ask a Question