Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Splitting files using awk and reading filename value from input data
Top Forums Programming Open Source Splitting files using awk and reading filename value from input data Post 302976078 by rbatte1 on Thursday 23rd of June 2016 01:35:45 PM
Old 06-23-2016
Splitting files using awk and reading filename value from input data
I have a process that requires me to read data from huge log files and find the most recent entry on a per-user basis. The number of users may fluctuate wildly month to month, so I can't code for it with names or a set number of variables to capture the data, and the files are large so I don't want to read the it several times.

The entries of interest have a particular string so I can extract just them from the overall log file and I have a way to split the output into separate files on a per-user basis, my plan being to then just read the last line of each files created with tail -1 and the filename giving me the user account in question.

My boss, however, worries about false-positive data matches for my expression (by chance or maliciously) that might try to overwrite a critical file.


My data has a syslog-type date in it which means doing a sort -u is proving tricky too. I've got this far with splitting the data out to files under /tmp/logs as splitlog.rbatte1 or similar but if field 11 were ever */../../etc/passwd then potentially I would be in trouble.

The date is the first three fields and 'as far as I am aware' a valid user name would be in field 11, but ........

A simplified part of the code would be:-
Code:
grep "Active transaction started" /var/log/qapplog | awk "{print \$1, \$2, \$3, \$11> \"/tmp/logs/splitlog.\"\$11}"
for userfile in /tmp/logs/splitlog.*
do
   lastrecord=$(tail -1 $userfile)
   printf "User %s last record is %s\n" "$userfile" "$lastrecord"
   .... whatever else here ....
done

I have considered adding tr -d "\/" to strip out the characters, but now that it's been raised, I'm concerned that there may be other things I'm not considering.

Is there a better way to work here, potentially with awk getting the equivalent of basename "$11" or variable substitution in the shell of "${{11}##*/}"?


Any suggestions welcome. Perhaps there is a better design overall that will find the last entry on a per-user basis. The log is thankfully written in time order, so the last in the file by user name is the last by time already.

Kind regards,
Robin
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Moving files by splitting the path embedded in the filename

Hello All. I am having a directory /tmp/rahul which contains many files in the format @#home@#rahul@#programs@#script.pl where /home/rahul/programs is the directory where the script.pl file is to be placed. I have many files in this format. What i want is a script which read these... (7 Replies)
Discussion started by: rahulrathod
7 Replies

2. Shell Programming and Scripting

Reading in data sets into arrays from an input file.

Hye all, I would like some help with reading in a file in which the data is seperated by commas. for instance: input.dat: 1,2,34,/test for the above case, the fn. will store the values into an array -> data as follows: data = 1 data = 2 data = 34 data = /test I am trying to write... (5 Replies)
Discussion started by: sidamin810
5 Replies

3. Shell Programming and Scripting

awk reading 2 input files but not getting expected value

I'm reading 2 input files but not getting expected value. I should get an alpha value on file_1_data but not getting any. Please help. >cat test6.sh awk ' FILENAME==ARGV { file_1_data=$0; print "----- 1 Line " NR " -----" $1; next } FILENAME==ARGV { file_2_data=$0; print "----- 2... (1 Reply)
Discussion started by: pdtak
1 Replies

4. Shell Programming and Scripting

Splitting input files into multiple files through AWK command

Hi, I needs to split *.txt files from single directory depends on the some mutltiple input values. i have wrote the code like below for file in *.txt do grep -i -h "value1|value2" $file > $file; done. My requirment is more input values needs to be given in grep; let us say 50... (3 Replies)
Discussion started by: arund_01
3 Replies

5. Shell Programming and Scripting

Reading specific contents from 1 input files and appending it to another input file

Hi guys, I am new to AWK and unix scripting. Please see below my problem and let me know if anyone you can help. I have 2 input files (example given below) Input file 2 is a standard file (it will not change) and we have to get the name (second column after comma) from it and append it... (5 Replies)
Discussion started by: sksahu
5 Replies

6. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ... (3 Replies)
Discussion started by: natalie23
3 Replies

7. Shell Programming and Scripting

Help with reading two input files in awk

Hello, I'm trying to write an awk program that reads two files inputs. example, file 1: 0.00017835 0.000176738 0.00018811 0.000189504 0.000188155 0.000180065 0.000178991 0.000178252 0.000182513 file 2: 1.7871769E-05 1.5139576E-16 1.5140196E-16 1.5139874E-16 1.7827407E-04 ... (5 Replies)
Discussion started by: joseamck
5 Replies

8. Shell Programming and Scripting

Splitting input CSV file into 3 files

Hi , I am receiving a CSV file that can vary in number of rows each time. I am supposed to split this file into 3 separate files like this: 1. create a file named 'File1.csv' that will contain first 3 rows of the input file 2. create file named 'File2.csv' that will contain last 3 rows of the... (7 Replies)
Discussion started by: kedrick
7 Replies

9. Shell Programming and Scripting

Splitting the Data using awk

Hello All, I have a comma delimiter file with 10 columns. I took the desired data but from $4 I need to split into two columns as 3+7 bytes. awk -F"," -v OFS=',' '{print $2,$3,$4}' foo.txt 42366,11/10/2014,5012418769 42366,11/10/2014,2046955672 42366,11/10/2014,2076802951 ... (3 Replies)
Discussion started by: karumudi7
3 Replies

10. Shell Programming and Scripting

How to embed data instead of reading user input from an array?

Hello, I am running under ubuntu1 14.04 and I have a script which is sending given process names to vanish so that I'd see less output when I run most popular tools like top etc in terminal window. In usual method it works. Whenever I restart the system, I have to enter the same data from... (2 Replies)
Discussion started by: baris35
2 Replies

LEARN ABOUT OSF1

faxcron

FAXCRON(8)						      System Manager's Manual							FAXCRON(8)

NAME

       faxcron - HylaFAX routine maintenance script

SYNOPSIS

       /usr/sbin/faxcron [ -n ] [ -l lastrun ]

DESCRIPTION

       faxcron	is  a  command	script	that  does  routine upkeep tasks in a HylaFAX spooling directory hierarchy.  This script is intended to be
       invoked from cron(8) on behalf of the fax user (often the ``fax'' user is uucp) once a day, with the standard output sent by  mail  to  the
       HylaFAX administrator.  For example, the following crontab might be set up for the fax user:
	 25   23    *	 *    *   sh /usr/sbin/faxcron | mail FaxMaster

       The faxcron script does the following work:

       o  report statistics for facsimile transmitted since the last time faxcron was run.

       o  report statistics for facsimile transmitted in the past week.

       o  report statistics for facsimile received since the last time faxcron was run.

       o  report statistics for facsimile received in the past week.

       o  report any recent calls that failed suspiciously and return the associated trace log;

       o  purge data that is 30 days or older from the remote machine information directory;

       o  delete information that is 30 days or older from session trace log files;

       o  remove files from the tmp directory that are older than 1 day;

       o  remove files in the received facsimile queue that are 7 days or older;

       o  report sites that are currently having jobs rejected; and

       o  force all session log files to be owned by the fax user and kept with protection mode 644.

OPTIONS

       -n	 Forces faxcron to run without updating any files.

       -l lastrun
		 Forces  the  notion  of  the last time that faxcron was run to be lastrun.  The value of lastrun is a date and time string of the
		 form ``MM/DD/YY HH:MM'' (the date(1) format string ``%D %H:%M'').

       -info n	 Set the expiration time for data in the info database to be n days.

       -log n	 Set the expiration time for session log data to be n days.

       -rcv n	 Set the expiration time for files in the received facsimile queue to be n days.

       -tmp n	 Set the expiration time for files in the tmp directory to be n days.

       -mode m	 Set the file protection mode for session logs to m (a command line argument to chmod(1)).

NOTES

       This script requires nawk(1) or gawk(1) and a date (1) program that supports the ``+format'' option.

       If session logs are to be kept private this script must be modified to filter out sensitive information such as calling card numbers.   (It
       should also be run with ``-mode 0600'' so that session log files are not publicly readable.)

FILES

       /var/spool/hylafax		      spooling area
       /var/spool/hylafax/status/lastrun      timestamp of last run
       /usr/sbin/xferfaxstats		      for generating transmit statistics
       /usr/sbin/recvstats		      for generating receive statistics
       /var/spool/hylafax/tmp/faxcronXXXX     temporary file for truncating session logs
       /var/spool/hylafax/tmp/faxlogXXXX      temporary file for logs of failed calls

SEE ALSO

       cron(8), faxq(8), faxgetty(8), faxsend(8), xferfaxstats(8), recvstats(8), hylafax-server(5)

								   May 12, 1996 							FAXCRON(8)
All times are GMT -4. The time now is 10:18 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy