06-01-2009
Help with awk for a newbie
Hi there, greetingt to everybody.
I have configured syslog-ng to get messages over UDP saving logs into a text file, it works fine.
I need to store the content of this file in several files depending from some criteria that I try to explain you with some examples :
Suppose the content of my log file (source.txt) looks like this :
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;4#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx#xxxx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;1#xxx;x;x;x;xxxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
As you can see I have 3 different kind of separators ";" "|" "#", there is a fix number of columns ( first 7 columns ) while the lenght of each row depends from value reported in the column n. 7. Depending from the value reported in column 7 I have several blocks of 5 columns separated by "#".
The result must be something like this :
File1.txt :
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;4#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx#xxxx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;1#xxx;x;x;x;xxxx
File2.txt
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
I think could be done with 'awk' somethin similar to :
awk -F";" '$1 == "1"' source.txt > File1.txt
awk -F";" '$1 == "2"' source.txt > File2.txt
I do not know if it's possible to create both files with just one awk command line but I am quite sure it is possible.
Than I need to create other 2 files from File1.txt and File2.txt.
In this case what I need is to split every line into several lines depending from the number reported into the column n. 7, in each line of this new file must be report just 2 columns of the original file as reported in the example below :
Suppose this is the content of the ordinal file (File2.txt)
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
I need this in the new file :
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxx;xx;x;x;xxx
I hope the above example was self explaining for you, I need to create from one row as many rows depending from number that can be found in the column n. 7, than each row must have 2 columns where the first is the content of the column n.6 of the original file and the second is the content of the block of 5 columns found in the original file separated by "#".
Any idea about how to solve this problem, thanks in advance.
Greetings
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I would like to parse a file and use each record to create a unique file
where the name of the file created is taken from the 3rd field.
Example: here are 3 records.
000007 0110 07-0001583 20060802132213 20060801112601
000007 0110 07-0001584 ... (4 Replies)
Discussion started by: zoo591
4 Replies
2. Shell Programming and Scripting
Hi guys, just started using awk here.
I've got a file called a.txt which contains a load of numbers on each line e.g.
35
232
654
1
9
4
I want to learn how to do three things:
1. Find the minimum
2. Find the maximum
3. Find the average
I want to learn how to do this using awk.... (5 Replies)
Discussion started by: i_am_a_robot
5 Replies
3. Shell Programming and Scripting
Hello,
I have a text file in below format, how do I put a header and assign field names to the file with either AWK or SED
STRT~ VA ~23606 ~TM14~8506~1485 (page 1)
STRT~ VA ~23662 ~TM17~8362~1783 (page 2)
STRT~ VA ~23662 ~TM17~8362~1783
STRT~ VA ~23662 ~TM17~8362~1783
STRT~ VA ~23662... (1 Reply)
Discussion started by: udaybo
1 Replies
4. Shell Programming and Scripting
Hello all
I have a file with below data, I need to add a third column as XXXX if the first coloumn (H180620081) last number is "1"
and YYYY if the first coloumn last number is "2"
Can it be done with AWK, I would appreciate if anyone could help me ou with this.
H180620081 31310 ... (11 Replies)
Discussion started by: udaybo
11 Replies
5. Shell Programming and Scripting
Hi,
Despite reading the Conditional Statements chapter in the O'Reilly Sed & Awk book several times and looking at numerous examples, I cannot for the life of me get any kind of if ... else statement to work in my awk scripts! My scripts work perfectly (as they are written at least) and do what... (4 Replies)
Discussion started by: jonathanm
4 Replies
6. UNIX for Dummies Questions & Answers
I have this output:
uniquemember=uid=315kthatch,ou=people,ou=client315,dc=paisleyhosting,dc=com
and i want the output to be just this:
315kthatch
I need it to be generic tho, because I have hundreds of lines of output, and the preceding numbers are not always 315. So I would need... (3 Replies)
Discussion started by: kthatch
3 Replies
7. Shell Programming and Scripting
$ awk -f awk1.awk
Where awk1.awk reads:
BEGIN {
printf ("Name = Smith, Richard" > "awk1" \n)
printf ("Name = Johnson, Waylan" > "awk1" \n)
printf ("Name = Brown, Pat" > "awk1" \n)
}
If I get rid of "\n", awk1 file gets created but all the names... (3 Replies)
Discussion started by: ora_umair
3 Replies
8. Shell Programming and Scripting
Hello everyone!
I have a small text file called processes.txt which contains a few lines in this fashion:
ID: 35; Arrival_Time: 0; Total_Exec_Time: 4;
ID: 36; Arrival_Time: 1; Total_Exec_Time: 6;
I am trying to figure out how I can get the values from between the delimiters ';' and ':' per... (16 Replies)
Discussion started by: mehungry
16 Replies
9. Shell Programming and Scripting
Hi guys,
I need a little help learning to batch an awk script. All the examples I found on line are too complicated for me. Here's the awk command that I want to run on lots of files.
awk 'NR==FNR{a=$0;next$3 in a{print $0 a " " a}' inputfile_A_1.out inputfile_B_1.out > outputfile1.txt
... (11 Replies)
Discussion started by: atjurhs
11 Replies
10. UNIX for Dummies Questions & Answers
Hi,
I was trying to change the value of the 4th column (put '1' in the 4th column of each row). My awk command is:
awk -F, '{$3=1;}1' OFS= input.txt > ./test_out.txt
My input file is:
a 1 2 31
b 4 5 61
c 7 8 91
My output file (test_out.txt)is:
a 1 2 31
b 4 5 61
c 7 8 91
What... (4 Replies)
Discussion started by: pc2001
4 Replies
LEARN ABOUT CENTOS
syslogd
syslogd(1M) syslogd(1M)
NAME
syslogd - log system messages
SYNOPSIS
/usr/sbin/syslogd [-d] [-f configfile] [-m markinterval] [-p path] [-t | -T]
syslogd reads and forwards system messages to the appropriate log files or users, depending upon the priority of a message and the system
facility from which it originates. The configuration file /etc/syslog.conf (see syslog.conf(4)) controls where messages are forwarded. sys-
logd logs a mark (timestamp) message every markinterval minutes (default 20) at priority LOG_INFO to the facility whose name is given as
mark in the syslog.conf file.
A system message consists of a single line of text, which may be prefixed with a priority code number enclosed in angle-brackets (<>); pri-
orities are defined in <sys/syslog.h>.
syslogd reads from the STREAMS log driver, /dev/log, and from any transport provider specified in /etc/netconfig, /etc/net/transport/hosts,
and /etc/net/transport/services.
syslogd reads the configuration file when it starts up, and again whenever it receives a HUP signal (see signal.h(3HEAD), at which time it
also closes all files it has open, re-reads its configuration file, and then opens only the log files that are listed in that file. syslogd
exits when it receives a TERM signal.
As it starts up, syslogd creates the file /var/run/syslog.pid, if possible, containing its process identifier (PID).
If message ID generation is enabled (see log(7D)), each message will be preceded by an identifier in the following format: [ID msgid facil-
ity.priority]. msgid is the message's numeric identifier described in msgid(1M). facility and priority are described in syslog.conf(4). [ID
123456 kern.notice] is an example of an identifier when message ID generation is enabled.
If the message originated in a loadable kernel module or driver, the kernel module's name (for example, ufs) will be displayed instead of
unix. See EXAMPLES for sample output from syslogd with and without message ID generation enabled.
In an effort to reduce visual clutter, message IDs are not displayed when writing to the console; message IDs are only written to the log
file. See EXAMPLES.
The /etc/default/syslogd file contains the following default parameter settings. See FILES.
LOG_FROM_REMOTE
Specifies whether remote messages are logged. LOG_FROM_REMOTE=NO is equivalent to the -t command-line option. The default value for
LOG_FROM_REMOTE is YES.
The following options are supported:
-d Turn on debugging. This option should only be used interactively in a root shell once the system is in multi-user
mode. It should not be used in the system start-up scripts, as this will cause the system to hang at the point
where syslogd is started.
-f configfile Specify an alternate configuration file.
-m markinterval Specify an interval, in minutes, between mark messages.
-p path Specify an alternative log device name. The default is /dev/log.
-T Enable the syslogd UDP port to turn on logging of remote messages. This is the default behavior. See .
-t Disable the syslogd UDP port to turn off logging of remote messages. See .
EXAMPLES
Example 1: syslogd Output Without Message ID Generation Enabled
The following example shows the output from syslogd when message ID generation is not enabled:
Sep 29 21:41:18 cathy unix: alloc /: file system full
Example 2: syslogd Output with ID generation Enabled
The following example shows the output from syslogd when message ID generation is enabled. The message ID is displayed when writing to log
file/var/adm/messages.
Sep 29 21:41:18 cathy ufs: [ID 845546 kern.notice]
alloc /: file system full
Example 3: syslogd Output with ID Generation Enabled
The following example shows the output from syslogd when message ID generation is enabled when writing to the console. Even though message
ID is enabled, the message ID is not displayed at the console.
Sep 29 21:41:18 cathy ufs: alloc /: file system full
/etc/syslog.conf Configuration file
/var/run/syslog.pid Process ID
/etc/default/syslogd Contains default settings. You can override some of the settings by command-line options.
/dev/log STREAMS log driver
/etc/netconfig Transport providers available on the system
/etc/net/transport/hosts Network hosts for each transport
/etc/net/transport/services Network services for each transport
See attributes(5) for descriptions of the following attributes:
+-----------------------------+-----------------------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+-----------------------------+-----------------------------+
|Availability |SUNWcsu |
+-----------------------------+-----------------------------+
logger(1), svcs(1), msgid(1M),svcadm(1M), syslog(3C), syslog.conf(4), attributes(5), signal.h(3HEAD), smf(5), log(7D)
NOTES
The mark message is a system time stamp, and so it is only defined for the system on which syslogd is running. It can not be forwarded to
other systems.
When syslogd receives a HUP signal, it attempts to complete outputting pending messages, and close all log files to which it is currently
logging messages. If, for some reason, one (or more) of these files does not close within a generous grace period, syslogd discards the
pending messages, forcibly closes these files, and starts reconfiguration. If this shutdown procedure is disturbed by an unexpected error
and syslogd cannot complete reconfiguration, syslogd sends a mail message to the superuser on the current system stating that it has shut
down, and exits.
Care should be taken to ensure that each window displaying messages forwarded by syslogd (especially console windows) is run in the system
default locale (which is syslogd's locale). If this advice is not followed, it is possible for a syslog message to alter the terminal set-
tings for that window, possibly even allowing remote execution of arbitrary commands from that window.
The syslogd service is managed by the service management facility, smf(5), under the service identifier:
svc:/system/system-log:default
Administrative actions on this service, such as enabling, disabling, or requesting restart, can be performed using svcadm(1M). The ser-
vice's status can be queried using the svcs(1) command.
31 May 2005 syslogd(1M)