Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

parsing log files, removing spaces and replace with commas

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting parsing log files, removing spaces and replace with commas
# 1  
Old 06-08-2009
parsing log files, removing spaces and replace with commas
Hello all

i am working on a database to import log files from my systems, but i cannot seem to find the answer. I searched here for a good bit and couldnt peice together what i was looking for.

I know you could do this with awk, i just dont know how. Any help would be greatly appreciated.

ok, so take for instance the /var/adm/messages file:
Jun 5 08:40:02 SunNiprS1 usba: [ID 349649 kern.info] PLEXTOR Corp. USB Storage Adapter 004400947B
Jun 5 08:40:02 SunNiprS1 genunix: [ID 936769 kern.info] scsa2usb0 is /pci@1e,600000/usb@a/cdrom@2
Jun 5 08:40:02 SunNiprS1 genunix: [ID 408114 kern.info] /pci@1e,600000/usb@a/cdrom@2 (scsa2usb0) online
Jun 5 08:41:09 SunNiprS1 scsi: [ID 193665 kern.info] sd16 at scsa2usb0: target 0 lun 0
Jun 5 08:41:09 SunNiprS1 genunix: [ID 936769 kern.info] sd16 is /pci@1e,600000/usb@a/cdrom@2/disk@0,0
Jun 5 08:41:35 SunNiprS1 genunix: [ID 408114 kern.info] /pci@1e,600000/usb@a/cdrom@2/disk@0,0 (sd16) online

all i want to do is seperate each field by a comma, i thought it would be pretty easy...but i am having a difficult time with it. Some sample output would look like this:

Jun 5, 08:40:02, SystemName, usba: [ID 349649 kern.info], PLEXTOR Corp. USB Storage Adapter 004400947B
Jun 5, 08:40:02, SystemName, genunix: [ID 936769 kern.info], scsa2usb0 is /pci@1e,600000/usb@a/cdrom@2

thank you
# 2  
Old 06-08-2009
This would doi it:
Code:
$ sed -e s'/ /, /2' -e 's/ /, /3'  -e 's/ /, /4' -e 's/ /, /8' < /var/adm/messages > /outputlogfile

e.g.:
Code:
$ sed -e s'/ /, /2' -e 's/ /, /3'  -e 's/ /, /4' -e 's/ /, /8' logfile
Jun 5, 08:40:02, SunNiprS1, usba: [ID 349649 kern.info], PLEXTOR Corp. USB Storage Adapter 004400947B
Jun 5, 08:40:02, SunNiprS1, genunix: [ID 936769 kern.info], scsa2usb0 is /pci@1e,600000/usb@a/cdrom@2
Jun 5, 08:40:02, SunNiprS1, genunix: [ID 408114 kern.info], /pci@1e,600000/usb@a/cdrom@2 (scsa2usb0) online
Jun 5, 08:41:09, SunNiprS1, scsi: [ID 193665 kern.info], sd16 at scsa2usb0: target 0 lun 0
Jun 5, 08:41:09, SunNiprS1, genunix: [ID 936769 kern.info], sd16 is /pci@1e,600000/usb@a/cdrom@2/disk@0,0
Jun 5, 08:41:35, SunNiprS1, genunix: [ID 408114 kern.info], /pci@1e,600000/usb@a/cdrom@2/disk@0,0 (sd16) online

Note the device paths have commas in already which you may want to substitute for a different character first.

If you don't want spaces after the commas then add a "-e 's/, /,/g'" to the end.
Last edited by TonyFullerMalv; 06-08-2009 at 07:41 PM.. Reason: Added: If you don't want spaces after the commas line
# 3  
Old 06-08-2009
if you have Python
Code:
#!/usr/bin/env python
for line in open("file"):
    line=line.strip()
    st = line.index("[")
    line=line.replace("]","],")
    o = line[:st].split()
    print ' '.join(o[:2]),','.join(o[2:]),line[st:]

output
Code:
# ./test.py
Jun 5 08:40:02,SunNiprS1,usba: [ID 349649 kern.info], PLEXTOR Corp. USB Storage Adapter 004400947B
Jun 5 08:40:02,SunNiprS1,genunix: [ID 936769 kern.info], scsa2usb0 is /pci@1e,600000/usb@a/cdrom@2
Jun 5 08:40:02,SunNiprS1,genunix: [ID 408114 kern.info], /pci@1e,600000/usb@a/cdrom@2 (scsa2usb0) online
Jun 5 08:41:09,SunNiprS1,scsi: [ID 193665 kern.info], sd16 at scsa2usb0: target 0 lun 0
Jun 5 08:41:09,SunNiprS1,genunix: [ID 936769 kern.info], sd16 is /pci@1e,600000/usb@a/cdrom@2/disk@0,0
Jun 5 08:41:35,SunNiprS1,genunix: [ID 408114 kern.info], /pci@1e,600000/usb@a/cdrom@2/disk@0,0 (sd16) online

Last edited by ghostdog74; 06-09-2009 at 12:47 PM..
# 4  
Old 06-09-2009
ghostdog74, can python be made to not put a comma between the month and date, not after the ":" and put a comma in after the "]"?
# 5  
Old 06-09-2009
You can make use of tr filter here:

echo "a:c:d" | tr ":" ","

This will replace : with ,
# 6  
Old 06-09-2009
of course, see my edits.

-----Post Update-----

Quote:
Originally Posted by dcoolsam
You can make use of tr filter here:

echo "a:c:d" | tr ":" ","

This will replace : with ,
see the requirement again. don't think OP wants to change the date's ":" to ,
# 7  
Old 06-09-2009
wow, thanks to you all!

this should work perfectly, i will update if i had to change anything...

i see where i was messing up, i wasnt using the -e option...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Removing commas from CSV file

Hi I'm creating a sh script to generate a csv file. The CSV contains the values from a sql table. The content looks this: a,b,c,c2,c3,,,,,,,,,,,d,e I have some code that can separate the fields using the comma as delimiter, but some values actually contain commas, such as... (2 Replies)
Discussion started by: preema
2 Replies

2. Shell Programming and Scripting

Removing just the trailing commas :-(

Hi all, I haven't needed to do any shell based editing for nearly 20 years, and no amount of searching around has found me a solution to this very simple problem :-( I have a csv file. Some lines have three commas at the end. This means the invoice hasn't been paid. I'd like to use sed / grep... (4 Replies)
Discussion started by: chardyzulu
4 Replies

3. Shell Programming and Scripting

Help with removing additional commas in string

Hi Experts, I have below strings hello,hi,,,,,,start date age,code,,,,,61,season I am trying to format this string to hello,hi,start date age,code,61,season Can anyone please help me in achieving this? Kind Regards, RB (3 Replies)
Discussion started by: ramakanth_burra
3 Replies

4. UNIX for Dummies Questions & Answers

Removing blank spaces from text files in UNIX

Hello, I am an super newbie, so forgive my sheer ignorance. I have a series of text files formatted as follows (just showing the header and first few lines): mean_geo mean_raw lat lon 0.000 0 -70.616 163.021 0.000 0 -70.620 163.073 0.000 ... (8 Replies)
Discussion started by: vtoniolo
8 Replies

5. Shell Programming and Scripting

Help with sed and replacing white spaces with commas

Dear all, I am in a bit of a quandary. I have 400 text files which I need to edit and output in a very specific way. Here is a sample text file copied from gedit ... The columns will come out a bit messed up but when I cat <file>, it gives a table with six columns (0-28, tot lob vol, vcsf,... (6 Replies)
Discussion started by: antonz
6 Replies

6. Shell Programming and Scripting

Need help in removing commas

i have the below line as output from a script. I want to delete the string "," and get the output without comma, cat D* | grep "bytes free" | awk '{print $3}' | ????? output: 40,966,189,056 Desired O/P: 40966189056 (1 Reply)
Discussion started by: ali560045
1 Replies

7. Linux

How do i remove commas(,) & spaces

Hey guys, I am very much new to shell scripts. So you ppl may feel that i am asking stupid question here. :D 1. I am using command line argument as an input variable. The user gets this value in his mail from client which has commas n spaces (Eg. 12,34,56,789) and the scripts... (5 Replies)
Discussion started by: anushree.a
5 Replies

8. Programming

Removing empty spaces and adding commas

I have a file which contains numbers as follows: 1234 9876 6789 5677 3452 9087 4562 1367 2678 7891 I need to remove the empty spaces and add commas between the numbers like: 1234,9876,6789,5677,3452, 9087,4562,1367,2678,7891 Can anyone tell me the command to do... (4 Replies)
Discussion started by: jazz
4 Replies

9. Shell Programming and Scripting

Removing trailing spaces from delimited files

Hi All I have a file of the following format (delimited by |) this is field 1 | field 2 (lots of blank spaces) | field 3 (lots of blank space) | field 1 | more text (lots of blank spaces) | dhjdsk | Is there a way I can remove... (6 Replies)
Discussion started by: djkane
6 Replies

10. UNIX for Dummies Questions & Answers

removing commas from text file

Dear all I have a file which looks like this xxxxxxxxxxxxxx,xxx,xxxxxxxxxx xxxxxxxxxxxxxx,xxx,xxxxxxxxxx etc basically 14 characters then a comma, three characters, then a comma then 10 characters. We are uploading this file to our mainframe and they want the commas removed, so it... (6 Replies)
Discussion started by: hcclnoodles
6 Replies
Login or Register to Ask a Question