Sponsored Content
Full Discussion: Help with awk for a newbie
Top Forums Shell Programming and Scripting Help with awk for a newbie Post 302321425 by capnino on Monday 1st of June 2009 06:01:14 AM
Old 06-01-2009
Help with awk for a newbie

Hi there, greetingt to everybody.
I have configured syslog-ng to get messages over UDP saving logs into a text file, it works fine.
I need to store the content of this file in several files depending from some criteria that I try to explain you with some examples :
Suppose the content of my log file (source.txt) looks like this :

1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;4#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx#xxxx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;1#xxx;x;x;x;xxxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

As you can see I have 3 different kind of separators ";" "|" "#", there is a fix number of columns ( first 7 columns ) while the lenght of each row depends from value reported in the column n. 7. Depending from the value reported in column 7 I have several blocks of 5 columns separated by "#".
The result must be something like this :

File1.txt :
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;4#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx#xxxx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;1#xxx;x;x;x;xxxx

File2.txt
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

I think could be done with 'awk' somethin similar to :
awk -F";" '$1 == "1"' source.txt > File1.txt
awk -F";" '$1 == "2"' source.txt > File2.txt
I do not know if it's possible to create both files with just one awk command line but I am quite sure it is possible.
Than I need to create other 2 files from File1.txt and File2.txt.
In this case what I need is to split every line into several lines depending from the number reported into the column n. 7, in each line of this new file must be report just 2 columns of the original file as reported in the example below :


Suppose this is the content of the ordinal file (File2.txt)
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

I need this in the new file :
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxx;xx;x;x;xxx

I hope the above example was self explaining for you, I need to create from one row as many rows depending from number that can be found in the column n. 7, than each row must have 2 columns where the first is the content of the column n.6 of the original file and the second is the content of the block of 5 columns found in the original file separated by "#".

Any idea about how to solve this problem, thanks in advance.
Greetings
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk newbie

I would like to parse a file and use each record to create a unique file where the name of the file created is taken from the 3rd field. Example: here are 3 records. 000007 0110 07-0001583 20060802132213 20060801112601 000007 0110 07-0001584 ... (4 Replies)
Discussion started by: zoo591
4 Replies

2. Shell Programming and Scripting

Newbie Awk data sorting

Hi guys, just started using awk here. I've got a file called a.txt which contains a load of numbers on each line e.g. 35 232 654 1 9 4 I want to learn how to do three things: 1. Find the minimum 2. Find the maximum 3. Find the average I want to learn how to do this using awk.... (5 Replies)
Discussion started by: i_am_a_robot
5 Replies

3. Shell Programming and Scripting

Need Help with AWK or SED (Newbie)

Hello, I have a text file in below format, how do I put a header and assign field names to the file with either AWK or SED STRT~ VA ~23606 ~TM14~8506~1485 (page 1) STRT~ VA ~23662 ~TM17~8362~1783 (page 2) STRT~ VA ~23662 ~TM17~8362~1783 STRT~ VA ~23662 ~TM17~8362~1783 STRT~ VA ~23662... (1 Reply)
Discussion started by: udaybo
1 Replies

4. Shell Programming and Scripting

Need help with AWK (newbie)

Hello all I have a file with below data, I need to add a third column as XXXX if the first coloumn (H180620081) last number is "1" and YYYY if the first coloumn last number is "2" Can it be done with AWK, I would appreciate if anyone could help me ou with this. H180620081 31310 ... (11 Replies)
Discussion started by: udaybo
11 Replies

5. Shell Programming and Scripting

Help a newbie please with awk if else statements

Hi, Despite reading the Conditional Statements chapter in the O'Reilly Sed & Awk book several times and looking at numerous examples, I cannot for the life of me get any kind of if ... else statement to work in my awk scripts! My scripts work perfectly (as they are written at least) and do what... (4 Replies)
Discussion started by: jonathanm
4 Replies

6. UNIX for Dummies Questions & Answers

Newbie Help with Grep or Awk .. Easy one ...

I have this output: uniquemember=uid=315kthatch,ou=people,ou=client315,dc=paisleyhosting,dc=com and i want the output to be just this: 315kthatch I need it to be generic tho, because I have hundreds of lines of output, and the preceding numbers are not always 315. So I would need... (3 Replies)
Discussion started by: kthatch
3 Replies

7. Shell Programming and Scripting

Newbie AWK Question

$ awk -f awk1.awk Where awk1.awk reads: BEGIN { printf ("Name = Smith, Richard" > "awk1" \n) printf ("Name = Johnson, Waylan" > "awk1" \n) printf ("Name = Brown, Pat" > "awk1" \n) } If I get rid of "\n", awk1 file gets created but all the names... (3 Replies)
Discussion started by: ora_umair
3 Replies

8. Shell Programming and Scripting

Need help with awk to get values from input (newbie here)

Hello everyone! I have a small text file called processes.txt which contains a few lines in this fashion: ID: 35; Arrival_Time: 0; Total_Exec_Time: 4; ID: 36; Arrival_Time: 1; Total_Exec_Time: 6; I am trying to figure out how I can get the values from between the delimiters ';' and ':' per... (16 Replies)
Discussion started by: mehungry
16 Replies

9. Shell Programming and Scripting

newbie needs help batching awk, tabitha

Hi guys, I need a little help learning to batch an awk script. All the examples I found on line are too complicated for me. Here's the awk command that I want to run on lots of files. awk 'NR==FNR{a=$0;next$3 in a{print $0 a " " a}' inputfile_A_1.out inputfile_B_1.out > outputfile1.txt ... (11 Replies)
Discussion started by: atjurhs
11 Replies

10. UNIX for Dummies Questions & Answers

Basic awk...newbie quetion

Hi, I was trying to change the value of the 4th column (put '1' in the 4th column of each row). My awk command is: awk -F, '{$3=1;}1' OFS= input.txt > ./test_out.txt My input file is: a 1 2 31 b 4 5 61 c 7 8 91 My output file (test_out.txt)is: a 1 2 31 b 4 5 61 c 7 8 91 What... (4 Replies)
Discussion started by: pc2001
4 Replies
syslogd(1M)                                                                                                                            syslogd(1M)

NAME
syslogd - log system messages SYNOPSIS
/usr/sbin/syslogd [-d] [-f configfile] [-m markinterval] [-p path] [-t | -T] syslogd reads and forwards system messages to the appropriate log files or users, depending upon the priority of a message and the system facility from which it originates. The configuration file /etc/syslog.conf (see syslog.conf(4)) controls where messages are forwarded. sys- logd logs a mark (timestamp) message every markinterval minutes (default 20) at priority LOG_INFO to the facility whose name is given as mark in the syslog.conf file. A system message consists of a single line of text, which may be prefixed with a priority code number enclosed in angle-brackets (<>); pri- orities are defined in <sys/syslog.h>. syslogd reads from the STREAMS log driver, /dev/log, and from any transport provider specified in /etc/netconfig, /etc/net/transport/hosts, and /etc/net/transport/services. syslogd reads the configuration file when it starts up, and again whenever it receives a HUP signal (see signal.h(3HEAD), at which time it also closes all files it has open, re-reads its configuration file, and then opens only the log files that are listed in that file. syslogd exits when it receives a TERM signal. As it starts up, syslogd creates the file /var/run/syslog.pid, if possible, containing its process identifier (PID). If message ID generation is enabled (see log(7D)), each message will be preceded by an identifier in the following format: [ID msgid facil- ity.priority]. msgid is the message's numeric identifier described in msgid(1M). facility and priority are described in syslog.conf(4). [ID 123456 kern.notice] is an example of an identifier when message ID generation is enabled. If the message originated in a loadable kernel module or driver, the kernel module's name (for example, ufs) will be displayed instead of unix. See EXAMPLES for sample output from syslogd with and without message ID generation enabled. In an effort to reduce visual clutter, message IDs are not displayed when writing to the console; message IDs are only written to the log file. See EXAMPLES. The /etc/default/syslogd file contains the following default parameter settings. See FILES. LOG_FROM_REMOTE Specifies whether remote messages are logged. LOG_FROM_REMOTE=NO is equivalent to the -t command-line option. The default value for LOG_FROM_REMOTE is YES. The following options are supported: -d Turn on debugging. This option should only be used interactively in a root shell once the system is in multi-user mode. It should not be used in the system start-up scripts, as this will cause the system to hang at the point where syslogd is started. -f configfile Specify an alternate configuration file. -m markinterval Specify an interval, in minutes, between mark messages. -p path Specify an alternative log device name. The default is /dev/log. -T Enable the syslogd UDP port to turn on logging of remote messages. This is the default behavior. See . -t Disable the syslogd UDP port to turn off logging of remote messages. See . EXAMPLES
Example 1: syslogd Output Without Message ID Generation Enabled The following example shows the output from syslogd when message ID generation is not enabled: Sep 29 21:41:18 cathy unix: alloc /: file system full Example 2: syslogd Output with ID generation Enabled The following example shows the output from syslogd when message ID generation is enabled. The message ID is displayed when writing to log file/var/adm/messages. Sep 29 21:41:18 cathy ufs: [ID 845546 kern.notice] alloc /: file system full Example 3: syslogd Output with ID Generation Enabled The following example shows the output from syslogd when message ID generation is enabled when writing to the console. Even though message ID is enabled, the message ID is not displayed at the console. Sep 29 21:41:18 cathy ufs: alloc /: file system full /etc/syslog.conf Configuration file /var/run/syslog.pid Process ID /etc/default/syslogd Contains default settings. You can override some of the settings by command-line options. /dev/log STREAMS log driver /etc/netconfig Transport providers available on the system /etc/net/transport/hosts Network hosts for each transport /etc/net/transport/services Network services for each transport See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ logger(1), svcs(1), msgid(1M),svcadm(1M), syslog(3C), syslog.conf(4), attributes(5), signal.h(3HEAD), smf(5), log(7D) NOTES
The mark message is a system time stamp, and so it is only defined for the system on which syslogd is running. It can not be forwarded to other systems. When syslogd receives a HUP signal, it attempts to complete outputting pending messages, and close all log files to which it is currently logging messages. If, for some reason, one (or more) of these files does not close within a generous grace period, syslogd discards the pending messages, forcibly closes these files, and starts reconfiguration. If this shutdown procedure is disturbed by an unexpected error and syslogd cannot complete reconfiguration, syslogd sends a mail message to the superuser on the current system stating that it has shut down, and exits. Care should be taken to ensure that each window displaying messages forwarded by syslogd (especially console windows) is run in the system default locale (which is syslogd's locale). If this advice is not followed, it is possible for a syslog message to alter the terminal set- tings for that window, possibly even allowing remote execution of arbitrary commands from that window. The syslogd service is managed by the service management facility, smf(5), under the service identifier: svc:/system/system-log:default Administrative actions on this service, such as enabling, disabling, or requesting restart, can be performed using svcadm(1M). The ser- vice's status can be queried using the svcs(1) command. 31 May 2005 syslogd(1M)
All times are GMT -4. The time now is 05:45 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy