Visit The New, Modern Unix Linux Community


Pulling information from a data file by date


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Pulling information from a data file by date
# 1  
Pulling information from a data file by date

Code:
awk -v now="$(date +%s)" -v tDiff="${USERMINUTES}" '
   BEGIN {
      FS="="
      if (!now) now=systime()
      if (!tDiff) tDiff=60*60
      p=1
  }
   /{/ {rec=$0;p=1;next}
   /}/ && rec && p {print rec ORS $0;next}
   $1=="entry_time" { if (now-$2>tDiff)p=0 }
   {rec=rec ORS $0}' "${1}"


the below code is very fast. it was built for something else but i'd like to be able to tweak it to do what i want to do.

what i need to do is a read a system log file which is about 40MB huge. i was to pull out the last 10 minutes worth of a information from the log.

my problem is, for a log file that big, records may be in there which may be a year or more old.

for instance, if i wanted to grab the last 10 minutes from a log. a variation of the following command can be used:

Code:
awk '/Jan 16 10:20/,0' /var/log/mail.log

however, if the log is a year old. then this awk statement will grab the very first occurence of "Jan 16 10:20", which may be a year ago, as opposed to 10 minutes ago.

any help will be much appreciated.

Last edited by SkySmart; 01-20-2015 at 02:28 PM..
# 2  
Is the year included in the timestamp in the log file you're searching?

If not, it may be difficult to determine which set of lines you want.
# 3  
Quote:
Originally Posted by Don Cragun
Is the year included in the timestamp in the log file you're searching?

If not, it may be difficult to determine which set of lines you want.
unfortunately, the year is not included. and i agree, that makes it difficult.
# 4  
Why not test the first line and only continue to search the log if date & time is within the last 10 minutes?
# 5  
I admit that it may be lengthy with a huge file, but why not tac it until Jan 16 is found? Btw, tac is smart and uses seek_set to read the records in chunks from the rear. If killed by e.g. a broken pipe due to the receiver quits, the rest of blocks is not read.

Last edited by RudiC; 01-17-2015 at 11:50 AM..
This User Gave Thanks to RudiC For This Post:
# 6  
Quote:
Originally Posted by RudiC
I admit that it may be lengthy with a huge file, but why not tac it until Jan 16 is found? Btw, tac is smart and uses seek_set to read the records in chunks from the rear. If killed by e.g. a broken pipe due to the receiver quits, the rest of blocks is not read.
tac is only available on linux. and i need something that will work on any unix host.

so the code below seems to work:

Code:
awk '{a[NR]=$0} END {while (NR) print a[NR--]}'  log.data

the only thing is, it reads the entire file backwards. if the file is huge, you can imagine it will take a long time to complete.

so what i want to do is, i want this code to be modified so that it aborts immediately after it finds the date and time matching the time specified by the user.

so if i wanted to scan the last 3 days of a log. that would be Jan 17. so when this awk code is run, and it is reading the file backwards, once it finds a date that starts with "Jan 17", it aborts and does not continue scanning back any further.

can anyone please help me modify this code to do that?
# 7  
How about using dd skip=N to read some blocks at the end of file?

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #570
Difficulty: Medium
The C programming language allows you to pass a parameter to a function by providing its memory address instead of the value stored in it.
True or False?

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Get information from one files, based on data from other file

Hello. I am trying to get some info from log file. I have fileA , which contains all the country prefixes (the file contains one column and "n" rows ). And i have fileB, which contains huge data of phone numbers (the file contains one column and "n" rows). What i want to do is, to count... (7 Replies)
Discussion started by: dragonfly85
7 Replies

2. Shell Programming and Scripting

Pulling Data, Then Moving to the Next File

I'm scanning a list of emails- I need to pull 2 pieces of data, then move to the next file: Sender's Email Address Email Date I need these to be outputted into a single column- separated by a ",". Like this: Email1's Address, Email1's Date Stamp Email2's Address, Email2's Date Stamp... (4 Replies)
Discussion started by: sudo
4 Replies

3. Shell Programming and Scripting

BASH- Need help pulling data from .emlx

Hello, fellow computer junkies. First time poster! My boss wrote an application (Mavericks 10.9, Mountain Lion 10.8) that checks a user's security settings. The user runs the application, then it spits out an email that is sent back to our inbox showing the results. On our end, we have a mail rule... (5 Replies)
Discussion started by: sudo
5 Replies

4. Shell Programming and Scripting

Pulling data from xml

Hi there, Please could anyone help with this. I have an xml file that contains repeating values eg <Rule name> AAAAA <Action> BBBBB </Action> <Data> CCCCC </Data> <Type> DDDDD </Type> </Rule name> <Rule name> A1A1A1A1 <Action> B1B1B1B1 </Action> <Data> C1C1C1C </Data> <Type>... (4 Replies)
Discussion started by: ssideel
4 Replies

5. Shell Programming and Scripting

Pulling data by GPS coordinates from text file

Hi there, I'm having a problem trying to extract data from within a text file. I'm trying to extract this manually for a lack of better words. I need any items that fall within latitude 36.5 to 39.5 and long -75.3 to -83.9 I have been doing this using cat neta.txt | grep '!38' and working... (6 Replies)
Discussion started by: Mikey
6 Replies

6. Shell Programming and Scripting

Pulling the file date from remote server

Hi, Can anyone please tell me whether we can pull the file from the remote server tru FTP, with the same date the file is available in remote server? Thanks, Punitha (1 Reply)
Discussion started by: puni
1 Replies

7. Shell Programming and Scripting

Help with pulling / filtering data from a .csv

Good day Gurus, I have a csv file that contains an inventory of active servers. This csv file contains a well over a hundred systems (IBM, SUN, HP). It also contains those systems details. See below for an example hostA,invver,1.02,20100430 hostA,date,08/30/2010,06:18 hostA,use,"Unknown... (4 Replies)
Discussion started by: LuffyDMonkey
4 Replies

8. Shell Programming and Scripting

SFTP to server, pulling data and removing the data

Hi all, I have the following script, but are not too sure about the syntax to complete the script. In essence, the script must connect to a SFTP server at a client site with username and password located in a file on my server. Then change to the appropriate directory. Pull the data to the... (1 Reply)
Discussion started by: codenjanod
1 Replies

9. Shell Programming and Scripting

Pulling data and following lines from file

I saw a few posts close to what i want to do, but they didn't look like they would work exactly.. or I need to think out of the box on this. I have a file that I keep server stats in for my own performance analysis. this file has the output from many commands in it (uptime, vmstats, ps, swap... (2 Replies)
Discussion started by: MizzGail
2 Replies

Featured Tech Videos