Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Can you extract (remove) lines from log files?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Can you extract (remove) lines from log files?
# 1  
Old 08-04-2013
Scissors Can you extract (remove) lines from log files?
I use "MineOS" (a linux distro with python scripts and web ui included for managing a Minecraft Server). The author of the scripts is currently having a problem with the Minecraft server log file being spammed with certain entries. He's working on clearing up the spam.

But in the meantime, I'm wondering if I can run a shell script that will occasionally remove these spam lines (and only these lines) to keep the log file more manageable. Here is an example of the spam:

2013-08-02 20:51:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-02 20:51:30 [INFO] /127.0.0.1:52343 lost connection
2013-08-02 20:51:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-02 20:51:30 [INFO] /127.0.0.1:52344 lost connection


The consistent piece is the "[SEVERE] Reached end of stream for /127.0.0.1". I would like to remove that entire line PLUS the single line that immediately follows each time (that lists 127.0.0.1:##### lost connection). But I don't want to remove other entries from the log file (named, "server.log" btw).

Is there a way to parse this server.log and selectively delete lines?
# 2  
Old 08-04-2013
This deletes the first line and the corresponding line (searched in the next 2 lines in order to allow another log entry in between).
Code:
awk '/\[SEVERE\] Reached end of stream for \/127\.0\.0\.1/ {c=2; next} c && c-- && /\[INFO\] \/127\.0\.0\.1:[0-9]+ lost connection/ {c=0; next} 1' server.log

This User Gave Thanks to MadeInGermany For This Post:
nbsparks
# 3  
Old 08-07-2013
Slight adjustment needed
Hello,

So I guess I need to refine my request.

Here is a real example of the log from recent testing.

Code:
2013-08-07 18:13:13 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:14 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:16 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:17 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:19 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:22 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:24 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:25 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:27 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:28 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:32 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:33 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:35 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:36 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:38 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [INFO] /127.0.0.1:42057 lost connection
2013-08-07 18:13:41 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:42 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:44 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:46 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:47 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:49 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:50 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:52 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:53 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:55 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:57 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:58 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:00 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:01 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:03 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:04 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:14:06 [SEVERE] Reached end of stream for /127.0.0.1

So I guess what I really need is two separate scripts:

1) A script that parses the server.log file and removes ALL instances of lines that contain "[SEVERE] Reached end of stream for /127.0.0.1"

2) A script that parses the server.log file and removes ALL instances of lines that fit this general pattern: "[INFO] /127.0.0.1:????? lost connection" (where ????? represents a random 5-digit port number)

And of course, the scripts have to leave all the remaining logs so that the log file is useful.

Thank you for any help that anyone can provide!
# 4  
Old 08-07-2013
Code:
[root@centosgeek ~]# cat testfile4
2013-08-07 18:13:16 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:17 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:19 [SEVERE] Reached end of stream for /127.0.0.1
SEVERE my stream
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:27 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:28 [SEVERE] Reached end of stream for /127.0.0.1
INFO is flowing but lost
2013-08-07 18:13:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:38 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [INFO] /127.0.0.1:42057 lost connection
2013-08-07 18:13:41 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:42 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:44 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
2013-08-07 18:13:46 [SEVERE] Reached end of stream for /127.0.0.1
SEVERE you forgot the doughnuts
2013-08-07 18:13:47 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:50 [SEVERE] Reached end of stream for /127.0.0.1
[root@centosgeek ~]# awk '(!/SEVERE/ || !/end of stream/) && (!/INFO/ || !/lost connection/)' testfile4
SEVERE my stream
INFO is flowing but lost
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
SEVERE you forgot the doughnuts
[root@centosgeek ~]#

This User Gave Thanks to Just Ice For This Post:
nbsparks
# 5  
Old 08-07-2013
So, if I want to catch "end of stream" and "lost connection" messages for IP addresses other than local host, could I do this for a script?

Code:
awk '(!/SEVERE/ || !/end of stream/ || !/127.0.0.1/) && (!/INFO/ || !/127.0.0.1/ || !/lost connection/)' server.log

Thanks for your help!

---------- Post updated at 02:26 PM ---------- Previous update was at 02:05 PM ----------

I modified a few IP addresses and tested with adding the extra bits to your code.

Here is what I came up with for my script:
Code:
cp server.log server.log.backup
awk '(!/SEVERE/ || !/end of stream/ || !/127.0.0.1/) && (!/INFO/ || !/127.0.0.1/ || !/lost connection/)' server.log.backup >server.log

Thank you!
# 6  
Old 08-08-2013
May be written like this
Code:
awk '!/SEVERE.*end of stream.*127.0.0.1/ && !/INFO.*127.0.0.1.*lost connection/' server.log.backup >server.log

# 7  
Old 08-08-2013
@ nbsparks -- glad it helped ... just remember always that whatever code shows up on this site, you definitely need to test first prior to using in production ...

Quote:
Originally Posted by Jotne
May be written like this
Code:
awk '!/SEVERE.*end of stream.*127.0.0.1/ && !/INFO.*127.0.0.1.*lost connection/' server.log.backup >server.log

@ jotne -- i like the brevity of your code but i seem to be missing something ... i tested on both centos and debian coming out with the same result as posted below ...
Code:
root@debiangeek:~# cat testfile4.1
2013-08-07 18:13:16 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:17 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:19 [SEVERE] Reached end of stream for /127.0.0.1
SEVERE my stream
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /192.168.5.167
2013-08-07 18:13:21 [INFO] /198.162.5.167:42057 lost connection
2013-08-07 18:13:27 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:28 [SEVERE] Reached end of stream for /127.0.0.1
INFO is flowing but lost
2013-08-07 18:13:30 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:38 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:39 [INFO] /127.0.0.1:42057 lost connection
2013-08-07 18:13:41 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:42 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:44 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
2013-08-07 18:13:46 [SEVERE] Reached end of stream for /127.0.0.1
SEVERE you forgot the doughnuts
2013-08-07 18:13:47 [SEVERE] Reached end of stream for /127.0.0.1
2013-08-07 18:13:50 [SEVERE] Reached end of stream for /127.0.0.1
root@debiangeek:~# awk '(!/SEVERE/ || !/end of stream/ || !/127.0/) && (!/INFO/ || !/lost connection/ || !/127.0/)' testfile4.1
SEVERE my stream
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /192.168.5.167
2013-08-07 18:13:21 [INFO] /198.162.5.167:42057 lost connection
INFO is flowing but lost
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
SEVERE you forgot the doughnuts
root@debiangeek:~# awk '!/SEVERE.*end of stream.*127.0.0.1/ && !/INFO.*lost connection.*127.0.0.1/' testfile4.1
SEVERE my stream
2013-08-07 18:13:21 [SEVERE] Reached end of stream for /192.168.5.167
2013-08-07 18:13:21 [INFO] /198.162.5.167:42057 lost connection
INFO is flowing but lost
2013-08-07 18:13:39 [INFO] /127.0.0.1:42057 lost connection
2013-08-07 18:13:43 [INFO] /127.0.0.1:42000 messy connection
2013-08-07 18:13:45 [INFO] /127.0.0.1:22057 under consideration
SEVERE you forgot the doughnuts
root@debiangeek:~#
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Extract the same lines from the two files

I used to use this script to extract the same lines from two files: grep -f file1 file2 > outputfile now I have file1 AB029895 AF208401 AF309648 AF526378 AJ444445 AJ720950 AJ851546 AY568629 AY591907 AY994087 BU116401 BU116599 BU119689 BU121308 BU125622 BU231446 BU236750 BU237045 (4 Replies)
Discussion started by: yuejian
4 Replies

2. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

3. Shell Programming and Scripting

Extract lines from text files

I have some files containing the following data # RESIDUE AA STRUCTURE BP1 BP2 ACC N-H-->O O-->H-N N-H-->O O-->H-N TCO KAPPA ALPHA PHI PSI X-CA Y-CA Z-CA 1 196 A M 0 0 230 0, 0.0 2,-0.2 0, 0.0 0, 0.0 0.000 360.0 360.0 360.0 76.4 21.7 -6.8 11.3 2 197 A D + 0 0 175 1,-0.1 2,-0.1 0, 0.0 0, 0.0... (10 Replies)
Discussion started by: edweena
10 Replies

4. Shell Programming and Scripting

How to remove the last 3 lines from many files?

Hello, I need to run a command or shell script that will remove the last 3 lines from every .js file that is under the directory /var/ww/vhost/ Can you please help ? thank you. (22 Replies)
Discussion started by: MaRiOsGR
22 Replies

5. Shell Programming and Scripting

Extract Log lines with Thread-(regex)

Hi everyone, Fist of all I must confess that I am pretty new in the Unix environment and especially to shell scripting, however due to work related requirements I have started to analyze software specific logs. The logs are structured so that it records by sessionID AND/OR Thread number, the... (3 Replies)
Discussion started by: sushimatt
3 Replies

6. Shell Programming and Scripting

How to extract lines between tags into different files?

I have an xml file with the below data: unix>Cat address.xml <Address City=”Amsterdam” Street = “station straat” ZIPCODE="2516 CK " </Address> <Address City=”Amsterdam” Street = “Leeuwen straat” ZIPCODE="2517 AB " </Address> <Address City=”The Hauge” Street = “kirk straat” ... (1 Reply)
Discussion started by: LinuxLearner
1 Replies

7. Shell Programming and Scripting

Remove the files that have less than certain lines

Hi all, I'm a newbie and I'm sorry if my question is too simple. I'm having problem to delete the files that have less than certain lines, say 16. #!/bin/tcsh set filen = `sh -c 'ls *csv 2> /dev/null'` foreach fil (${filen}) if ]; then rm -f ${filen} fi end exit ... (2 Replies)
Discussion started by: GoldenFire
2 Replies

8. Shell Programming and Scripting

Extract lines from files

hi all, I have three files. The first file (FILE_INFO in my code) consists of four parameters for each line. 0.00765600 0.08450704 M3 E3 0.00441931 0.04878049 M4 E5 0.01904574 0.21022727 M5 E10 0.00510400 0.05633803 M6 E12 0.00905960 ... (11 Replies)
Discussion started by: my_Perl
11 Replies

9. Shell Programming and Scripting

compare two files and to remove the matching lines on both the files

I have two files and need to compare the two files and to remove the matching lines from both the files (4 Replies)
Discussion started by: shellscripter
4 Replies

10. Shell Programming and Scripting

is it hard to extract particular lines & strings from the files??

Hi Experts, I have lots of big size files. Below is the snapshot of a file. From the files i want extract informmation like belows. What could be command or script for that? DELETE RESP:940120105 CREATE RESP:0 GET RESP:0 File contains like below- ... ... <log... (8 Replies)
Discussion started by: thepurple
8 Replies
Login or Register to Ask a Question