Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Splitting files using awk and reading filename value from input data
Top Forums Programming Open Source Splitting files using awk and reading filename value from input data Post 302976260 by rbatte1 on Monday 27th of June 2016 06:45:48 AM
Old 06-27-2016
A compromise perhaps bolting several things together.

From the suggestions given and further digging, I've ended up with:-
Code:
awk "/Active transaction started/ && /${userregex}/ {print \$1, \$2, \$3 \$11 \
> \"/tmp/logs/splitlog.\"gensub(/[^[:alnum:]._-]/, \"\", \"g\", \$11)}" /var/log/qapplog

for userfile in /tmp/logs/splitlog.*
do
   lastrecord=$(tail -1 $userfile)
   printf "User %s last record is %s\n" "$userfile" "$lastrecord"
   .... whatever else here ....
done

To explain it a bit more:-
  • The ${userregex} is a regular expression for all the users we are interested in, so we can exclude testing messages which sadly get written to the same log.

  • The text output is just for checking we've got it, there would be further processing (and that's all fine) This is finding a way to get the last logged entry for each user, so the printf line is just for debug.

  • We removed the grep with the two expressions (both must be satisfied) see this thread

  • The output file(s) are redirected to /tmp/logs/splitlog. appended by the output from the gensub on field 11. The gensub removes all characters from field 11 that are not matched by being alphanumeric, full-stop (period for American English), underscore or hyphen, those all being acceptable characters to build a filename from and sensibly allowed in user account names. We could possibly have apostrophes too, but these have been excluded.

It seems to work for me in testing, but I'd appreciate another few sets of eyes to validate I'm not doing something daft and leaving a gaping hole somewhere.


Robin
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Moving files by splitting the path embedded in the filename

Hello All. I am having a directory /tmp/rahul which contains many files in the format @#home@#rahul@#programs@#script.pl where /home/rahul/programs is the directory where the script.pl file is to be placed. I have many files in this format. What i want is a script which read these... (7 Replies)
Discussion started by: rahulrathod
7 Replies

2. Shell Programming and Scripting

Reading in data sets into arrays from an input file.

Hye all, I would like some help with reading in a file in which the data is seperated by commas. for instance: input.dat: 1,2,34,/test for the above case, the fn. will store the values into an array -> data as follows: data = 1 data = 2 data = 34 data = /test I am trying to write... (5 Replies)
Discussion started by: sidamin810
5 Replies

3. Shell Programming and Scripting

awk reading 2 input files but not getting expected value

I'm reading 2 input files but not getting expected value. I should get an alpha value on file_1_data but not getting any. Please help. >cat test6.sh awk ' FILENAME==ARGV { file_1_data=$0; print "----- 1 Line " NR " -----" $1; next } FILENAME==ARGV { file_2_data=$0; print "----- 2... (1 Reply)
Discussion started by: pdtak
1 Replies

4. Shell Programming and Scripting

Splitting input files into multiple files through AWK command

Hi, I needs to split *.txt files from single directory depends on the some mutltiple input values. i have wrote the code like below for file in *.txt do grep -i -h "value1|value2" $file > $file; done. My requirment is more input values needs to be given in grep; let us say 50... (3 Replies)
Discussion started by: arund_01
3 Replies

5. Shell Programming and Scripting

Reading specific contents from 1 input files and appending it to another input file

Hi guys, I am new to AWK and unix scripting. Please see below my problem and let me know if anyone you can help. I have 2 input files (example given below) Input file 2 is a standard file (it will not change) and we have to get the name (second column after comma) from it and append it... (5 Replies)
Discussion started by: sksahu
5 Replies

6. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ... (3 Replies)
Discussion started by: natalie23
3 Replies

7. Shell Programming and Scripting

Help with reading two input files in awk

Hello, I'm trying to write an awk program that reads two files inputs. example, file 1: 0.00017835 0.000176738 0.00018811 0.000189504 0.000188155 0.000180065 0.000178991 0.000178252 0.000182513 file 2: 1.7871769E-05 1.5139576E-16 1.5140196E-16 1.5139874E-16 1.7827407E-04 ... (5 Replies)
Discussion started by: joseamck
5 Replies

8. Shell Programming and Scripting

Splitting input CSV file into 3 files

Hi , I am receiving a CSV file that can vary in number of rows each time. I am supposed to split this file into 3 separate files like this: 1. create a file named 'File1.csv' that will contain first 3 rows of the input file 2. create file named 'File2.csv' that will contain last 3 rows of the... (7 Replies)
Discussion started by: kedrick
7 Replies

9. Shell Programming and Scripting

Splitting the Data using awk

Hello All, I have a comma delimiter file with 10 columns. I took the desired data but from $4 I need to split into two columns as 3+7 bytes. awk -F"," -v OFS=',' '{print $2,$3,$4}' foo.txt 42366,11/10/2014,5012418769 42366,11/10/2014,2046955672 42366,11/10/2014,2076802951 ... (3 Replies)
Discussion started by: karumudi7
3 Replies

10. Shell Programming and Scripting

How to embed data instead of reading user input from an array?

Hello, I am running under ubuntu1 14.04 and I have a script which is sending given process names to vanish so that I'd see less output when I run most popular tools like top etc in terminal window. In usual method it works. Whenever I restart the system, I have to enter the same data from... (2 Replies)
Discussion started by: baris35
2 Replies

LEARN ABOUT OSF1

log

log(8)							      System Manager's Manual							    log(8)

NAME

       log - Records input and output from a program

SYNOPSIS

       /usr/sbin/log <logfile> <command>

OPERANDS

       The file in which to record the interaction being logged.  The command to execute.

DESCRIPTION

       The  log  program  runs <command> and logs the input to and output from <command> to the <logfile> file.  Input and output are logged until
       <command> exits, the log program exits, and the exit status of <command> is returned.

       The log program is used by the system  installation  procedure  and  the  it(8)	command  to  create  the  /var/adm/smlogs/install.log  and
       /var/adm/smlogs/it.log installation log files.

RESTRICTIONS

       Because	the  log  program is used in the installation standalone environment, program size was the greatest concern in its implementation.
       The log program does not search for the PATH variable to locate <command> and error messages are terse.

       The log program causes <command> to take standard input from and write standard output and standard error to  UNIX  pipes.   Some  commands
       will  not  be able to operate in this environment; therefore, it is suggested that you use the script(1) command instead.  UNIX shells will
       not issue prompts when run from log unless the shell is started with an explicit interactive switch (-i for most shells).  For example, log
       foo.tmp /sbin/sh -i

       In the previous example, foo.tmp is the name of <logfile>.

       The  log  program  intercepts  end-of-file (usually Ctrl/d).  Therefore programs which normally receive end-of-file as an exit command must
       exit by some other means.

ERRORS

       Log open error

   Explanation:
       The log program was unable to open <logfile>. Verify that the directory exists and that ownerships and permissions are set correctly.

       Exec Error

   Explanation:
       The log program was unable to execute <command>. Verify that you specified a full pathname for <command> and  that  <command>  is  an  exe-
       cutable file.

       Fork Error

   Explanation:
       The log program was unable to create one of the processes it requires to log data.

SEE ALSO

       Commands: it(8), script(1)

																	    log(8)
All times are GMT -4. The time now is 02:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy