Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help with awk for a newbie

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with awk for a newbie
# 1  
Old 06-01-2009
Help with awk for a newbie
Hi there, greetingt to everybody.
I have configured syslog-ng to get messages over UDP saving logs into a text file, it works fine.
I need to store the content of this file in several files depending from some criteria that I try to explain you with some examples :
Suppose the content of my log file (source.txt) looks like this :

1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;4#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx#xxxx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;1#xxx;x;x;x;xxxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

As you can see I have 3 different kind of separators ";" "|" "#", there is a fix number of columns ( first 7 columns ) while the lenght of each row depends from value reported in the column n. 7. Depending from the value reported in column 7 I have several blocks of 5 columns separated by "#".
The result must be something like this :

File1.txt :
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;4#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx#xxxx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;1#xxx;x;x;x;xxxx

File2.txt
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

I think could be done with 'awk' somethin similar to :
awk -F";" '$1 == "1"' source.txt > File1.txt
awk -F";" '$1 == "2"' source.txt > File2.txt
I do not know if it's possible to create both files with just one awk command line but I am quite sure it is possible.
Than I need to create other 2 files from File1.txt and File2.txt.
In this case what I need is to split every line into several lines depending from the number reported into the column n. 7, in each line of this new file must be report just 2 columns of the original file as reported in the example below :


Suppose this is the content of the ordinal file (File2.txt)
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

I need this in the new file :
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxx;xx;x;x;xxx

I hope the above example was self explaining for you, I need to create from one row as many rows depending from number that can be found in the column n. 7, than each row must have 2 columns where the first is the content of the column n.6 of the original file and the second is the content of the block of 5 columns found in the original file separated by "#".

Any idea about how to solve this problem, thanks in advance.
Greetings
# 2  
Old 06-01-2009
To separate the files:

Code:
awk -F";" '{print > "File"$1".txt"}' source.txt

To get the end result:

Code:
awk -F ";" '{
  n=split($0,a,"#")
  for(i=2;i<=n;i++){ 
    print $5 "#" a[i]
  }
}' File2.txt

# 3  
Old 06-01-2009
if you have Python
Code:
#!/usr/bin/env python
for line in open("file"):
    open("file_"+line[0]+".txt","a").write(line)  
    f=open(line[0]+".txt","a")    
    line=line.strip().split("#")
    oline = line[0].split(";")
    ordinal=oline[-2]
    for i in line[1:]:
        f.write(ordinal+"#"+i+"\n")
    f.close()

output
Code:
# more file_1.txt  file_2.txt
::::::::::::::
file_1.txt
::::::::::::::
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;4#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx#xxxx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;1#xxx;x;x;x;xxxx
::::::::::::::
file_2.txt
::::::::::::::
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;2#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;xxxxxx;3#xxx;x;x;x;xxxx#xxx;x;xx;x;xxx#xxxx;xx;xxx;x;xxx

# more 1.txt 2.txt
::::::::::::::
1.txt
::::::::::::::
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxxx;xx;xxx;x;xxx
xxxxxx#xxxx;xx;xx;x;x
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxxx;xx;xxx;x;xxx
xxxxxx#xxx;x;x;x;xxxx
::::::::::::::
2.txt
::::::::::::::
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxx;x;x;x;xxxx
xxxxxx#xxx;x;xx;x;xxx
xxxxxx#xxxx;xx;xxx;x;xxx

# 4  
Old 06-01-2009
Code:
cat source.txt
1;xxxxx;xxxxxxxxxx|xxxx;xxx;aaaaaa;4#a1x;x;x;x;xxxx#a2x;x;xx;x;xxx#a3xx;xx;xxx;x;xxx#a4xx;xx;xx;x;x
1;xxxxx;xxxxxxxxxx|xxxx;xxx;bbbbbb;3#b1x;x;x;x;xxxx#b2x;x;xx;x;xxx#b3xx;xx;xxx;x;xxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;cccccc;2#c1x;x;x;x;xxxx#c2x;x;xx;x;xxx
1;xxxxx;xxxxxxxxxx|xxxx;xxx;dddddd;1#d1x;x;x;x;xxxx
2;xxxxx;xxxxxxxxxx|xxxx;xxx;eeeeee;3#e1x;x;x;x;xxxx#e2x;x;xx;x;xxx#e3xx;xx;xxx;x;xxx
 
rm file*.txt

nawk -F\; '{if((f="file"$1".txt")!=of){close(of);of=f};p=$5;oFS=FS;FS="#";$0=$0;for(i=2;i<=NF;i++)print p"#"$i>>f;FS=oFS}' source.txt

cat file1.txt
aaaaaa#a1x;x;x;x;xxxx
aaaaaa#a2x;x;xx;x;xxx
aaaaaa#a3xx;xx;xxx;x;xxx
aaaaaa#a4xx;xx;xx;x;x
bbbbbb#b1x;x;x;x;xxxx
bbbbbb#b2x;x;xx;x;xxx
bbbbbb#b3xx;xx;xxx;x;xxx
dddddd#d1x;x;x;x;xxxx

cat file2.txt
cccccc#c1x;x;x;x;xxxx
cccccc#c2x;x;xx;x;xxx
eeeeee#e1x;x;x;x;xxxx
eeeeee#e2x;x;xx;x;xxx
eeeeee#e3xx;xx;xxx;x;xxx

Un-obscured awk code:
Code:
{
if( (f="file"$1".txt") != of )  {
  close(of) # close current file if we switched to another output file
  of = f
  }
p = $5
oFS = FS # save current FS
FS = "#" # set new FS
$0 = $0 # rebuild fields according to new FS
for (i = 2; i <= NF; i++) print p"#"$i >> f
FS = oFS # restore FS
}

To avoid hitting the max number of open files, the code does a close() just before (re)opening a file. To keep the close-open operations to a minimum it would be smarter to do:
Code:
sort source.txt | nawk -F\; ...

# 5  
Old 06-01-2009
More in deep ...
Thank you Franklin54 and Ghostdog74 .... both configurations are solving my problem ... Thanks a lot .....

Dear Colemar I'll try also you script .... I'll let you now thanks al lot to you .....

I have for you an additional question, how can I transform all sepatators ( "|" ";" "#") in one ";".
After the cration of the files I need to have just ";" as separator and not the originals.
I do not care if this is done before, at mean time or after the creation as described in my first post.

Greetings to everybody
# 6  
Old 06-01-2009
Something like:

Code:
tr '|#' ';' < file > newfile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Basic awk...newbie quetion

Hi, I was trying to change the value of the 4th column (put '1' in the 4th column of each row). My awk command is: awk -F, '{$3=1;}1' OFS= input.txt > ./test_out.txt My input file is: a 1 2 31 b 4 5 61 c 7 8 91 My output file (test_out.txt)is: a 1 2 31 b 4 5 61 c 7 8 91 What... (4 Replies)
Discussion started by: pc2001
4 Replies

2. Shell Programming and Scripting

newbie needs help batching awk, tabitha

Hi guys, I need a little help learning to batch an awk script. All the examples I found on line are too complicated for me. Here's the awk command that I want to run on lots of files. awk 'NR==FNR{a=$0;next$3 in a{print $0 a " " a}' inputfile_A_1.out inputfile_B_1.out > outputfile1.txt ... (11 Replies)
Discussion started by: atjurhs
11 Replies

3. Shell Programming and Scripting

Need help with awk to get values from input (newbie here)

Hello everyone! I have a small text file called processes.txt which contains a few lines in this fashion: ID: 35; Arrival_Time: 0; Total_Exec_Time: 4; ID: 36; Arrival_Time: 1; Total_Exec_Time: 6; I am trying to figure out how I can get the values from between the delimiters ';' and ':' per... (16 Replies)
Discussion started by: mehungry
16 Replies

4. Shell Programming and Scripting

Newbie AWK Question

$ awk -f awk1.awk Where awk1.awk reads: BEGIN { printf ("Name = Smith, Richard" > "awk1" \n) printf ("Name = Johnson, Waylan" > "awk1" \n) printf ("Name = Brown, Pat" > "awk1" \n) } If I get rid of "\n", awk1 file gets created but all the names... (3 Replies)
Discussion started by: ora_umair
3 Replies

5. UNIX for Dummies Questions & Answers

Newbie Help with Grep or Awk .. Easy one ...

I have this output: uniquemember=uid=315kthatch,ou=people,ou=client315,dc=paisleyhosting,dc=com and i want the output to be just this: 315kthatch I need it to be generic tho, because I have hundreds of lines of output, and the preceding numbers are not always 315. So I would need... (3 Replies)
Discussion started by: kthatch
3 Replies

6. Shell Programming and Scripting

Help a newbie please with awk if else statements

Hi, Despite reading the Conditional Statements chapter in the O'Reilly Sed & Awk book several times and looking at numerous examples, I cannot for the life of me get any kind of if ... else statement to work in my awk scripts! My scripts work perfectly (as they are written at least) and do what... (4 Replies)
Discussion started by: jonathanm
4 Replies

7. Shell Programming and Scripting

Need help with AWK (newbie)

Hello all I have a file with below data, I need to add a third column as XXXX if the first coloumn (H180620081) last number is "1" and YYYY if the first coloumn last number is "2" Can it be done with AWK, I would appreciate if anyone could help me ou with this. H180620081 31310 ... (11 Replies)
Discussion started by: udaybo
11 Replies

8. Shell Programming and Scripting

Need Help with AWK or SED (Newbie)

Hello, I have a text file in below format, how do I put a header and assign field names to the file with either AWK or SED STRT~ VA ~23606 ~TM14~8506~1485 (page 1) STRT~ VA ~23662 ~TM17~8362~1783 (page 2) STRT~ VA ~23662 ~TM17~8362~1783 STRT~ VA ~23662 ~TM17~8362~1783 STRT~ VA ~23662... (1 Reply)
Discussion started by: udaybo
1 Replies

9. Shell Programming and Scripting

Newbie Awk data sorting

Hi guys, just started using awk here. I've got a file called a.txt which contains a load of numbers on each line e.g. 35 232 654 1 9 4 I want to learn how to do three things: 1. Find the minimum 2. Find the maximum 3. Find the average I want to learn how to do this using awk.... (5 Replies)
Discussion started by: i_am_a_robot
5 Replies

10. Shell Programming and Scripting

awk newbie

I would like to parse a file and use each record to create a unique file where the name of the file created is taken from the 3rd field. Example: here are 3 records. 000007 0110 07-0001583 20060802132213 20060801112601 000007 0110 07-0001584 ... (4 Replies)
Discussion started by: zoo591
4 Replies
Login or Register to Ask a Question