Splitting files using awk and reading filename value from input data

Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Old 06-23-2016
Splitting files using awk and reading filename value from input data

I have a process that requires me to read data from huge log files and find the most recent entry on a per-user basis. The number of users may fluctuate wildly month to month, so I can't code for it with names or a set number of variables to capture the data, and the files are large so I don't want to read the it several times.

The entries of interest have a particular string so I can extract just them from the overall log file and I have a way to split the output into separate files on a per-user basis, my plan being to then just read the last line of each files created with tail -1 and the filename giving me the user account in question.

My boss, however, worries about false-positive data matches for my expression (by chance or maliciously) that might try to overwrite a critical file.


My data has a syslog-type date in it which means doing a sort -u is proving tricky too. I've got this far with splitting the data out to files under /tmp/logs as splitlog.rbatte1 or similar but if field 11 were ever */../../etc/passwd then potentially I would be in trouble.

The date is the first three fields and 'as far as I am aware' a valid user name would be in field 11, but ........

A simplified part of the code would be:-
Code:
grep "Active transaction started" /var/log/qapplog | awk "{print \$1, \$2, \$3, \$11> \"/tmp/logs/splitlog.\"\$11}"
for userfile in /tmp/logs/splitlog.*
do
   lastrecord=$(tail -1 $userfile)
   printf "User %s last record is %s\n" "$userfile" "$lastrecord"
   .... whatever else here ....
done

I have considered adding tr -d "\/" to strip out the characters, but now that it's been raised, I'm concerned that there may be other things I'm not considering.

Is there a better way to work here, potentially with awk getting the equivalent of basename "$11" or variable substitution in the shell of "${{11}##*/}"?


Any suggestions welcome. Perhaps there is a better design overall that will find the last entry on a per-user basis. The log is thankfully written in time order, so the last in the file by user name is the last by time already.

Kind regards,
Robin
# 2  
Old 06-23-2016
How about posting a few lines from the input file? Some with, and some without the string of interest.

Not sure if I understood the requirement correctly, but, taking my syslog for a sample, wouldn't this fulfill your needs (redirect printout to a "user" file in /tmp/log if need be):
Code:
awk '{split ($5, U, /[[]/); T[U[1]] = $1 " " $2 " " $3} END {for (t in T) print t, T[t]}' /var/log/syslog
systemd-tmpfiles Jun 23 08:47:54
anacron Jun 23 08:46:57
dhclient: Jun 23 18:28:50
dbus Jun 23 12:03:34
CRON Jun 23 21:17:01
kernel: Jun 23 21:18:13
mtp-probe: Jun 23 11:40:12

# 4  
Old 06-25-2016
Just add your Active transaction started as an awk "pattern", and it should fly, shouldn't it? You can even refine the pattern to fulfill your boss' proclivities.
This User Gave Thanks to RudiC For This Post:
rbatte1 (06-27-2016)
# 5  
Old 06-25-2016
Quote:
Originally Posted by RudiC
How about posting a few lines from the input file? Some with, and some without the string of interest.

Not sure if I understood the requirement correctly, but, taking my syslog for a sample, wouldn't this fulfill your needs (redirect printout to a "user" file in /tmp/log if need be):
Code:
awk '{split ($5, U, /[[]/); T[U[1]] = $1 " " $2 " " $3} END {for (t in T) print t, T[t]}' /var/log/syslog
systemd-tmpfiles Jun 23 08:47:54
anacron Jun 23 08:46:57
dhclient: Jun 23 18:28:50
dbus Jun 23 12:03:34
CRON Jun 23 21:17:01
kernel: Jun 23 21:18:13
mtp-probe: Jun 23 11:40:12

Quote:
Originally Posted by rbatte1
I think you've got it just right, but it will take me a bit to decipher your code!


Many thanks,
Robin
Hello, rbatte1.

Without having an actual input or output representative example from you, I do think RudiC has shown you a valid indication of what it is possible. Definitively, you do not have the need to grep for it.
If is of any help to decipher what it has been shown, allow me to break it in pieces.
Essentially, there are two pieces to process each line, based on the format structure of /var/log/syslog
Code:
split ($5, U, /[[]/); # it divides field 5 into smaller parts ([ is the separator) and creates an array to capture the pieces using a variable named U.
T[U[1]] = $1 " " $2 " " $3 # builds an index using the variable T where the index is the first element of U, the split done before.

Once all lines has been processed, display the result:
Code:
END {for (t in T) print t, T[t]} # iterate over the index and display what's recorded.

Since, this is done for each line, you'll get multiple hits.
If you want to narrow it down to a particular criteria, let's say: "Active transaction started"
Code:
awk '/Active transaction started/ {split ($5, U, /[[]/); T[U[1]] = $1 " " $2 " " $3} END {for (t in T) print t, T[t]}' /var/log/syslog

Limiting the index to only those lines that contain the match.

If you were to post a representative example, I am quite sure we could provide other alternatives, as well.
Also, you might be interested in knowing about an open source project named: ELK
Which it does an amazing job at dealing with all sort of information on logs.

Last edited by Aia; 06-25-2016 at 06:53 PM..
This User Gave Thanks to Aia For This Post:
rbatte1 (06-27-2016)
# 6  
Old 06-27-2016
A compromise perhaps bolting several things together.

From the suggestions given and further digging, I've ended up with:-
Code:
awk "/Active transaction started/ && /${userregex}/ {print \$1, \$2, \$3 \$11 \
> \"/tmp/logs/splitlog.\"gensub(/[^[:alnum:]._-]/, \"\", \"g\", \$11)}" /var/log/qapplog

for userfile in /tmp/logs/splitlog.*
do
   lastrecord=$(tail -1 $userfile)
   printf "User %s last record is %s\n" "$userfile" "$lastrecord"
   .... whatever else here ....
done

To explain it a bit more:-
  • The ${userregex} is a regular expression for all the users we are interested in, so we can exclude testing messages which sadly get written to the same log.

  • The text output is just for checking we've got it, there would be further processing (and that's all fine) This is finding a way to get the last logged entry for each user, so the printf line is just for debug.

  • We removed the grep with the two expressions (both must be satisfied) see this thread

  • The output file(s) are redirected to /tmp/logs/splitlog. appended by the output from the gensub on field 11. The gensub removes all characters from field 11 that are not matched by being alphanumeric, full-stop (period for American English), underscore or hyphen, those all being acceptable characters to build a filename from and sensibly allowed in user account names. We could possibly have apostrophes too, but these have been excluded.

It seems to work for me in testing, but I'd appreciate another few sets of eyes to validate I'm not doing something daft and leaving a gaping hole somewhere.


Robin
# 7  
Old 06-27-2016
You won't get a user name from your for loop but a full path to a splitlog.usernamemod file, and you run multiple processes to get at the last entries.

I'm not sure I understand why you abstain from packing as much processing as possible into the awk script, piping its output (= last entry per user) into a while loop.

And, maybe the "further processing" could - to a large? extent - be included into the awk as well?

Last edited by RudiC; 06-27-2016 at 01:46 PM.. Reason: typo
This User Gave Thanks to RudiC For This Post:
rbatte1 (06-28-2016)
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Splitting the Data using awk karumudi7 Shell Programming and Scripting 3 04-15-2015 11:37 PM
Call a awk script with variable and input filename Onkar Banerjee Shell Programming and Scripting 0 04-19-2013 04:55 AM
Splitting input CSV file into 3 files kedrick Shell Programming and Scripting 7 10-04-2012 08:36 AM
Splitting record into multiple records by appending values from an input field (AWK) imtiaz99 Shell Programming and Scripting 4 04-24-2012 08:18 AM
Help with reading two input files in awk joseamck Shell Programming and Scripting 5 11-14-2011 10:04 PM
Data Splitting into two files from one file karumudi7 Shell Programming and Scripting 3 09-14-2011 06:31 AM
awk- reading input file twice acsg Shell Programming and Scripting 14 04-26-2011 08:57 AM
Reading from standard input with awk rany1 Shell Programming and Scripting 3 04-29-2010 01:13 PM
Filename from splitting files to have the same filename of the original file with counter value natalie23 Shell Programming and Scripting 3 11-13-2009 06:15 AM
Reading specific contents from 1 input files and appending it to another input file sksahu Shell Programming and Scripting 5 01-14-2009 06:09 AM
Splitting input files into multiple files through AWK command arund_01 Shell Programming and Scripting 3 05-13-2008 10:17 AM
awk reading 2 input files but not getting expected value pdtak Shell Programming and Scripting 1 04-29-2008 11:03 PM
AWK set FILENAME via user input timj123 Shell Programming and Scripting 2 02-24-2008 03:05 PM
Reading in data sets into arrays from an input file. sidamin810 Shell Programming and Scripting 5 07-06-2005 08:54 AM
Moving files by splitting the path embedded in the filename rahulrathod UNIX for Dummies Questions & Answers 7 04-14-2005 03:43 AM