Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Strip log file with multiple tokens

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Strip log file with multiple tokens
# 1  
Old 05-27-2010
Strip log file with multiple tokens
I have app log files that need to be cleansed for readability, and my sed skills are not adequate.

Each line has a long multi-segment header, which I am trying to remove, example below:

Code:
[EL Fine]: 2010.05.26 20:38:00.640--DatabaseSessionImpl(21447570)--Connection(26209441)--Thread(Thread[Thread-37,6,main])--SELECT GETDATE() AS TS

I want to remove first four '--' delimited tockens and see only:

Code:
 
SELECT GETDATE() AS TS

Any sed or awk solutions of hints are very much appreciated.

File sizes are in 200-300 MB range.
# 2  
Old 05-27-2010
Code:
#cat file_name|awk -F- '{print $10}'

# 3  
Old 05-27-2010
Code:
awk -F-- '$0=$NF' file

# 4  
Old 05-27-2010
Try
Code:
sed 's/.*--//' infile

# 5  
Old 05-27-2010
Thank you everybody, great!
Interestingly awk outperforms sed repeatedly, see timing below:

Code:
~$ time sed 's/.*--//' 5M?.log > /dev/null   
real    0m0.036s
user    0m0.030s
sys     0m0.000s
~$ time awk -F-- '$0=$NF' 5M?.log > /dev/null
real    0m0.024s
user    0m0.020s
sys     0m0.000s

Thanks again!
Last edited by radoulov; 05-27-2010 at 06:53 PM.. Reason: Added code tags.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Unzip Multiple zip files and Strip directory

I receive multiple zipped directories with files in them, so the .zip name is the name of the directory containing the files. so i have used a simple loop to unzip all of them but when unzipped i have folders/directories, i wanted to strip these directories and remain with the actual files from... (1 Reply)
Discussion started by: buggzdiq
1 Replies

2. Shell Programming and Scripting

strip csv file

Hi everyone, I hope someone can help me: i am trying to get some info from a csv file, after i awk the column i need , i made a selection and output it in a file. now i need to get a list from this file, but i stuck with some fields. basically i have a text file with next data: 3... (3 Replies)
Discussion started by: lostym
3 Replies

3. Shell Programming and Scripting

Strip time from CSV File?

Hi, I've been trying (and failing miserably) all morning to strip from a CSV file the time from it. Can somebody point me in the right direction on how to do this using sed or awk? The file looks like: "James","07/20/2009-14:40:11" "Steve","08/06/2006-02:34:37"... (5 Replies)
Discussion started by: nmuntz
5 Replies

4. UNIX for Dummies Questions & Answers

Strip time from date in a file

I have a couple of datetime fields in a file with contents like below: ICPBR|6373056085|1|O||||JOHN|SMITH|||H200|706|445668|||123 SMITH ST|LAGRANGE|IL|66666 |||||||N|N|N|N|N|||345676|2009.02.20-13:09:04|257655957|2009.02.20-13:09:04||||N|||||F||||||||||| I want to strip the time off the... (4 Replies)
Discussion started by: ChicagoBlues
4 Replies

5. Shell Programming and Scripting

Shell Script to replace tokens in multiple files

I have multiple script files that I have created, that allow me to simply replace a few tokens at the top of the file, and then not have to go through the actual script and change anything. I have about 10 of them, but I was hoping to find a way to write a small script that would allow me to input... (20 Replies)
Discussion started by: cbo0485
20 Replies

6. Shell Programming and Scripting

how to strip rows from a text file?

Can an expert kindly write an efficient Linux ksh script that will strip rows with no numbers from a text file? Supposing there are three rows that text file called text.txt : "field1","field2","field3",11,22,33,44 "field1","field2","field3",1,2,3,4 "field1","field2","field3",,,, The... (5 Replies)
Discussion started by: ihot
5 Replies

7. UNIX for Dummies Questions & Answers

How to strip the contants from a file

Hi, I have some EDI data which 830, 862 and 997. Here is the sample data: ISA~00~ ~00~ ~ZZ~F159B ~ZZ~U1CAD ~051215~184 3~U~00200~000011432~0~P~< GS~FA~TC11A~U1CAD~051215~1843~000011432~X~002002 ST~997~0001 AK1~SH~1168 AK2~856~11680001 AK5~A... (2 Replies)
Discussion started by: isingh786
2 Replies

8. Shell Programming and Scripting

How to strip apostrophe from a file

I am trying to remove or replace various extraneous characters from a file so that subsequent processes work correctly. The characters that is giving me trouble is the apostrophe '. The command I 'm trying is sed 's/\'//g' ${IN_WRK_DIR}/file1 > ${IN_WRK_DIR}/file2 in a Korn script on HP... (8 Replies)
Discussion started by: aquimby
8 Replies

9. UNIX for Dummies Questions & Answers

trying to strip the first 4 char. of a file out via commandline

im kind alost. i beleave its a sed command. but i cant seem to find it in my book. can someone point me in the write direction. i know this is extreamly sloppy. but this is what i did untill i can figure out how to manipulate the filename namespace. an ls on the directory where this would run... (2 Replies)
Discussion started by: Optimus_P
2 Replies
Login or Register to Ask a Question