Sponsored Content
Top Forums Shell Programming and Scripting Parse log file to insert into database Post 302952234 by unplugme71 on Sunday 16th of August 2015 01:54:33 AM
Old 08-16-2015
Parse log file to insert into database

I have a log file that's created daily by this command:

Code:
sar -u 300 288 >> /var/log/usage/$(date "+%Y-%m-%d")_$(hostname)_cpu.log

It that contains data like this:

Code:
Linux 3.16.0-4-amd64 (myhostname)       08/15/2015      _x86_64_        (1 CPU)

11:34:17 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:39:17 PM     all      0.09      0.00      0.07      0.09      0.00     99.75
11:44:17 PM     all      0.04      0.00      0.03      0.00      0.00     99.92
11:49:17 PM     all      1.49      0.00      0.49      0.06      0.00     97.96
11:54:17 PM     all     23.27      0.00      0.51      0.05      0.03     76.14


11:56:12 PM     all      0.17      0.00      0.13      0.01      0.01     99.69
Average:        all      5.69      0.00      0.26      0.05      0.01     93.99


I'm not sure if there's a way to strip out the data I don't need before it logs to the file such as %nice, %iowait, and %steal. As well as the header and footer information. It would make the rest of this post irrelevant.

I need to ignore the first few rows and the last row. I also need to ignore any blank lines.

The only data I need to insert into the database is the time, %user, %system, and %idle data.

The table looks like this:

Code:
row_id (int) | date (date) | time (time) | hostname (varchar) | user (decimal) | system (decimal) | idle (decimal)

The log file name is YYYY-MM-DD_hostname_cpu.log format. I need to extract the date of the filename and insert into the table as well.

So far I have

Code:
cat YYYY-MM-DD_hostname_cpu.log | awk 'NR printf "%s,%s,%s\n", $4,$6,$9'

I then needed it inserted into a mysql database. I'm assuming some type of while loop would be needed here.

Last edited by unplugme71; 08-16-2015 at 01:30 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to insert data in database based on text file?

Hi....can you guys help me out in this script?? Below is a text file script....called Bukom.txt and it contains these: BUKOM 20060101 2.5 2.6 2.7 2.8 2.9 2.3 2.1 BUKOM 20060102 2.4 2.5 2.6 2.7 2.7 2.6 2.4 BUKOM 20060103 2.1 ... (9 Replies)
Discussion started by: forevercalz
9 Replies

2. Shell Programming and Scripting

How to insert data into MYSql database from a text file

Hi, Need to get help from you guys about this issue. I need to insert data into MySql database from a text file which is located in other server. The text file is something look like below: Date | SubscriberNo | Call Duration 20/7/07 | 123456788 | 20 20/7/07 | 123412344 | 30 The... (4 Replies)
Discussion started by: shirleyeow
4 Replies

3. Shell Programming and Scripting

how to insert data into database by reading it from a text file??

Hi....can you guys help me out in this script?? Below is a text file and it contains these: GEF001 000093625 MKL002510 000001 000000 000000 000000 000000 000000 000001 GEF001 000093625 MKL003604 000001 000000 000000 000000 000000 000000 000001 GEF001 000093625 MKL005675 000001... (4 Replies)
Discussion started by: pallavishetty
4 Replies

4. Shell Programming and Scripting

Shell script to parse a line and insert a word

Hi All, I have a file like this, data1,data2,,,data5,data6. i want to write a shell script to replace data3 with "/example/string". which means my data file should look like this . data1,data2,example/string],,data5,data6. Could you guys help me to get a sed command or any other command... (8 Replies)
Discussion started by: girish.raos
8 Replies

5. Shell Programming and Scripting

To Read a File and Insert a part of the lines into the database

Hi Guys I need to have a shell script which reads a log file and insert a part of each line into the database. Some sample lines in the file are as below. 20091112091359 MED_RQACK : user_data=60173054304,100232120,20091112091359,;ask_status=0;ask_reason=OK;msg_id=20091112091319... (5 Replies)
Discussion started by: Somanadh
5 Replies

6. Web Development

INSERT data to a Database Table from a text file

If you have a text file and if you want to Insert data to your Database Table, You can do it with these queries LOAD DATA LOCAL INFILE '/path/yourTextFile.txt' INTO TABLE yourTableName FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n' (0 Replies)
Discussion started by: sitex
0 Replies

7. Shell Programming and Scripting

Korn shell program to parse CSV text file and insert values into Oracle database

Enclosed is comma separated text file. I need to write a korn shell program that will parse the text file and insert the values into Oracle database. I need to write the korn shell program on Red Hat Enterprise Linux server. Oracle database is 10g. (15 Replies)
Discussion started by: shellguy
15 Replies

8. Shell Programming and Scripting

Parse through ~21,000 Database DDL statements -- Fastest way to perform search, replace and insert

Hello All: We are looking to search through 2000 files with around 21,000 statements where we have to search, replace and insert a pattern based on the following: 1) Parse through the file and check for CREATE MULTISET TABLE or CREATE SET TABLE statements.....and they always end with ON... (5 Replies)
Discussion started by: madhunk
5 Replies

9. Shell Programming and Scripting

How to read a text file line by line and insert into a database table?

I have a test file that I want to read and insert only certain lines into the the table based on a filter. 1. Rread the log file 12 Hours back Getdate() -12 Hours 2. Extract the following information on for lines that say "DUMP is complete" A. Date B. Database Name C.... (2 Replies)
Discussion started by: JolietJake
2 Replies

10. Shell Programming and Scripting

Read latest log files and perform database insert

Hi Experts, I have a situation where I need to write a shell script to continuously monitor a log directory with multiple log files and perform following: 1. Read the latest log file continuously and grep "Success" OR "Failure" 2. As it capture either Success or Failure, it has to perform a... (1 Reply)
Discussion started by: rish_max
1 Replies
LAST, LASTB(1)							   User Commands						    LAST, LASTB(1)

NAME
last, lastb - show a listing of last logged in users SYNOPSIS
last [options] [username...] [tty...] lastb [options] [username...] [tty...] DESCRIPTION
last searches back through the /var/log/wtmp file (or the file designated by the -f option) and displays a list of all users logged in (and out) since that file was created. One or more usernames and/or ttys can be given, in which case last will show only the entries matching those arguments. Names of ttys can be abbreviated, thus last 0 is the same as last tty0. When catching a SIGINT signal (generated by the interrupt key, usually control-C) or a SIGQUIT signal, last will show how far it has searched through the file; in the case of the SIGINT signal last will then terminate. The pseudo user reboot logs in each time the system is rebooted. Thus last reboot will show a log of all the reboots since the log file was created. lastb is the same as last, except that by default it shows a log of the /var/log/btmp file, which contains all the bad login attempts. OPTIONS
-a, --hostlast Display the hostname in the last column. Useful in combination with the --dns option. -d, --dns For non-local logins, Linux stores not only the host name of the remote host, but its IP number as well. This option translates the IP number back into a hostname. -f, --file file Tell last to use a specific file instead of /var/log/wtmp. The --file option can be given multiple times, and all of the specified files will be processed. -F, --fulltimes Print full login and logout times and dates. -i, --ip Like --dns , but displays the host's IP number instead of the name. -number -n, --limit number Tell last how many lines to show. -p, --present time Display the users who were present at the specified time. This is like using the options --since and --until together with the same time. -R, --nohostname Suppresses the display of the hostname field. -s, --since time Display the state of logins since the specified time. This is useful, e.g., to easily determine who was logged in at a particular time. The option is often combined with --until. -t, --until time Display the state of logins until the specified time. --time-format format Define the output timestamp format to be one of notime, short, full, or iso. The notime variant will not print any timestamps at all, short is the default, and full is the same as the --fulltimes option. The iso variant will display the timestamp in ISO-8601 format. The ISO format contains timezone information, making it preferable when printouts are investigated outside of the system. -w, --fullnames Display full user names and domain names in the output. -x, --system Display the system shutdown entries and run level changes. TIME FORMATS
The options that take the time argument understand the following formats: YYYYMMDDhhmmss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm (seconds will be set to 00) YYYY-MM-DD (time will be set to 00:00:00) hh:mm:ss (date will be set to today) hh:mm (date will be set to today, seconds to 00) now yesterday (time is set to 00:00:00) today (time is set to 00:00:00) tomorrow (time is set to 00:00:00) +5min -5days NOTES
The files wtmp and btmp might not be found. The system only logs information in these files if they are present. This is a local configu- ration issue. If you want the files to be used, they can be created with a simple touch(1) command (for example, touch /var/log/wtmp). FILES
/var/log/wtmp /var/log/btmp AUTHOR
Miquel van Smoorenburg <miquels@cistron.nl> AVAILABILITY
The last command is part of the util-linux package and is available from Linux Kernel Archive <https://www.kernel.org/pub/linux/utils/util- linux/>. SEE ALSO
login(1), wtmp(5), init(8), shutdown(8) util-linux October 2013 LAST, LASTB(1)
All times are GMT -4. The time now is 03:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy