Help with awk for a newbie Post: 302321425

Sponsored Content

Top Forums Shell Programming and Scripting Help with awk for a newbie Post 302321425 by capnino on Monday 1st of June 2009 06:01:14 AM

06-01-2009

Registered User

Help with awk for a newbie

Hi there, greetingt to everybody.
I have configured syslog-ng to get messages over UDP saving logs into a text file, it works fine.
I need to store the content of this file in several files depending from some criteria that I try to explain you with some examples :
Suppose the content of my log file (source.txt) looks like this :

1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;4#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx#xxxx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;1#xxx;x;x;x;xxxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

As you can see I have 3 different kind of separators ";" "|" "#", there is a fix number of columns ( first 7 columns ) while the lenght of each row depends from value reported in the column n. 7. Depending from the value reported in column 7 I have several blocks of 5 columns separated by "#".
The result must be something like this :

File1.txt :
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;4#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx#xxxx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;1#xxx;x;x;x;xxxx

File2.txt
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

I think could be done with 'awk' somethin similar to :
awk -F";" '$1 == "1"' source.txt > File1.txt
awk -F";" '$1 == "2"' source.txt > File2.txt
I do not know if it's possible to create both files with just one awk command line but I am quite sure it is possible.
Than I need to create other 2 files from File1.txt and File2.txt.
In this case what I need is to split every line into several lines depending from the number reported into the column n. 7, in each line of this new file must be report just 2 columns of the original file as reported in the example below :

Suppose this is the content of the ordinal file (File2.txt)
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

I need this in the new file :
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxx;xx;x;x;xxx

I hope the above example was self explaining for you, I need to create from one row as many rows depending from number that can be found in the column n. 7, than each row must have 2 columns where the first is the content of the column n.6 of the original file and the second is the content of the block of 5 columns found in the original file separated by "#".

Any idea about how to solve this problem, thanks in advance.
Greetings

capnino

View Public Profile for capnino

Find all posts by capnino

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk newbie

I would like to parse a file and use each record to create a unique file where the name of the file created is taken from the 3rd field. Example: here are 3 records. 000007 0110 07-0001583 20060802132213 20060801112601 000007 0110 07-0001584 ...

2. Shell Programming and Scripting

Newbie Awk data sorting

Hi guys, just started using awk here. I've got a file called a.txt which contains a load of numbers on each line e.g. 35 232 654 1 9 4 I want to learn how to do three things: 1. Find the minimum 2. Find the maximum 3. Find the average I want to learn how to do this using awk....

3. Shell Programming and Scripting

Need Help with AWK or SED (Newbie)

Hello, I have a text file in below format, how do I put a header and assign field names to the file with either AWK or SED STRT~ VA ~23606 ~TM14~8506~1485 (page 1) STRT~ VA ~23662 ~TM17~8362~1783 (page 2) STRT~ VA ~23662 ~TM17~8362~1783 STRT~ VA ~23662 ~TM17~8362~1783 STRT~ VA ~23662...

4. Shell Programming and Scripting

Need help with AWK (newbie)

Hello all I have a file with below data, I need to add a third column as XXXX if the first coloumn (H180620081) last number is "1" and YYYY if the first coloumn last number is "2" Can it be done with AWK, I would appreciate if anyone could help me ou with this. H180620081 31310 ...

5. Shell Programming and Scripting

Help a newbie please with awk if else statements

Hi, Despite reading the Conditional Statements chapter in the O'Reilly Sed & Awk book several times and looking at numerous examples, I cannot for the life of me get any kind of if ... else statement to work in my awk scripts! My scripts work perfectly (as they are written at least) and do what...

6. UNIX for Dummies Questions & Answers

Newbie Help with Grep or Awk .. Easy one ...

I have this output: uniquemember=uid=315kthatch,ou=people,ou=client315,dc=paisleyhosting,dc=com and i want the output to be just this: 315kthatch I need it to be generic tho, because I have hundreds of lines of output, and the preceding numbers are not always 315. So I would need...

7. Shell Programming and Scripting

Newbie AWK Question

$ awk -f awk1.awk Where awk1.awk reads: BEGIN { printf ("Name = Smith, Richard" > "awk1" \n) printf ("Name = Johnson, Waylan" > "awk1" \n) printf ("Name = Brown, Pat" > "awk1" \n) } If I get rid of "\n", awk1 file gets created but all the names...

8. Shell Programming and Scripting

Need help with awk to get values from input (newbie here)

Hello everyone! I have a small text file called processes.txt which contains a few lines in this fashion: ID: 35; Arrival_Time: 0; Total_Exec_Time: 4; ID: 36; Arrival_Time: 1; Total_Exec_Time: 6; I am trying to figure out how I can get the values from between the delimiters ';' and ':' per...

9. Shell Programming and Scripting

newbie needs help batching awk, tabitha

Hi guys, I need a little help learning to batch an awk script. All the examples I found on line are too complicated for me. Here's the awk command that I want to run on lots of files. awk 'NR==FNR{a=$0;next$3 in a{print $0 a " " a}' inputfile_A_1.out inputfile_B_1.out > outputfile1.txt ...

10. UNIX for Dummies Questions & Answers

Basic awk...newbie quetion

Hi, I was trying to change the value of the 4th column (put '1' in the 4th column of each row). My awk command is: awk -F, '{$3=1;}1' OFS= input.txt > ./test_out.txt My input file is: a 1 2 31 b 4 5 61 c 7 8 91 My output file (test_out.txt)is: a 1 2 31 b 4 5 61 c 7 8 91 What...

LEARN ABOUT DEBIAN

newslog

NEWSLOG(5)							File Formats Manual							NEWSLOG(5)

NAME

       newslog - description of Usenet log files

DESCRIPTION

       Most  log  files created by Usenet programs reside in the /var/log/news directory and have a ``.log'' extension.  Several versions are usu-
       ally kept with an additional extension such as ``.1'', ``.2'', etc. -- the higher the number, the older the log.  The  older  versions  are
       compressed.

       The scanlogs script and related utilities (see newslog(8)) are responsible for rotating and compressing these files.

       Some  log  files always have data, others only have data if there is a problem, and others are only created if a particular program is used
       or configuration parameter is set.  The innstat script (see newslog(8)) monitors the size of all log files.

       The following files will only accumulate data under the direction of control.ctl(5):
	      control.log miscctl.log newgroup.log rmgroup.log unwanted.log
       In order to create these files, the ``message'' and ``action'' fields of control.ctl should be chosen from the following table:
	      Message	  Action	 Meaning
	      all	  log=miscctl	 Log all messages by default
	      default	  log=miscctl	 Log unknown messages
	      newgroup	  doit=newgroup  Create group and log message
	      newgroup	  log=newgroup	 Log message
	      rmgroup	  doit=rmgroup	 Remove group and log message
	      rmgroup	  log=rmgroup	 Log message
	      ``other''   doit=miscctl	 log and process the message
	      ``other''   log=miscctl	 Log message
       Here, ``other'' refers to any other control message such as:
	      checkgroups ihave sendme sendsys senduuname version

       The following is a list of log files.

       control.log
	      This file maintains a count of the number of newgroup and rmgroup control messages seen for each newsgroup.  The	count  is  of  the
	      number  of  control messages with identical arguments, regardless if they were actually processed.  All control arguments, including
	      invalid ones, are counted.  This file is updated by tally.control, which is invoked by scanlogs if either the  newgroup  or  rmgroup
	      logs exist.  This file is not rotated.

       errlog This  file contains the standard output and standard error of any program spawned by innd(8).  The most common programs are the con-
	      trol-message handlers found in /usr/lib/news/control.  This file should be empty.  Scanlogs will print the entire contents  of  this
	      log file if it is non-empty.

       expire.log
	      By  default,  when  news.daily  is  going  to expire old news articles, it writes the date to this file, followed by any output from
	      expire(8) and the ending date.  All lines but the first are indented four spaces.

       miscctl.log
	      When control.ctl is configured as described above, all control messages except newgroup and rmgroup are appended	to  this  file	by
	      writelog.   There  will be a summary line describing the message and the action taken, followed by the article indented by four spa-
	      ces, and a blank line.

       newgroup.log
	      When control.ctl is configured as described above, all newgroup messages are appended to this file using the same format as for mis-
	      cctl.log.

       news   This file logs articles received by innd.  Scanlogs summarizes the rejected articles reported in this file.

       news.crit
	      All  critical  error messages issued by innd are appended to this file via syslog(3).  This log file should be empty.  Scanlogs will
	      print the entire contents of this log file if it is non-empty.  You should have the following line in your syslog.conf(5) file:
		     news.crit	/var/log/news/news.crit

       news.err
	      All major error messages issued by innd are appended to this file via syslog.  This log file should be empty.  Scanlogs  will  print
	      the entire contents of this log file if it is non-empty.	You should have the following line in your syslog.conf file:
		     news.err  /var/log/news/news.err

       news.notice
	      All  standard  error  messages  and  status  messages issued by innd are appended to this file via syslog.  Scanlogs uses the awk(1)
	      script innlog.awk to summarize this file.  You should have the following line in your syslog.conf file:
		     news.notice  /var/log/news/news.notice

       nntpsend.log
	      The nntpsend(8) programs appends all status messages to this file.

       rmgroup.log
	      When control.ctl is configured as described above, all rmgroup messages are appended to this file using the same format as for misc-
	      ctl.log.

       unwanted.log
	      This  log maintains a count of the number of articles that were rejected because they were posted to newsgroups that do not exist at
	      the local site.  This file is updated by tally.unwanted and maintained in reverse numeric order (the  most  popular  rejected  group
	      first).  This file is not rotated.

HISTORY

       Written by Landon Curt Noll <chongo@toad.com> and Rich $alz <rsalz@uunet.uu.net> for InterNetNews.  This is revision 1.6, dated 1996/09/10.

SEE ALSO

       control.ctl(5), ctlinnd(8), expire(8), innd(8), news.daily(8), nntpsend(8), newslog(8).

																	NEWSLOG(5)