Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to grep logs for errors and receive specific additional lines?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to grep logs for errors and receive specific additional lines?
# 1  
Old 09-20-2013
How to grep logs for errors and receive specific additional lines?
Hi there,

I have a script that I've used to find errors in my Minecraft Server logs. But I'd like to refine that script to be more useful.

Here is the script:
Code:
grep -n "SEVERE" /minecraft/server.log | awk -F":" '{print $1-2 "," $1+10 "p"}' | xargs -t -i sed -n {} /minecraft/server.log  >> /minecraft/severe_errors.log

Here is what the script is looking through:
Code:
2013-09-20 09:55:28 [INFO] [PermissionsEx] Enabling PermissionsEx v1.19.5
2013-09-20 09:55:28 [INFO] [PermissionsEx] Superperms support enabled.
2013-09-20 09:55:28 [INFO] [PermissionsEx] v1.19.5 enabled
2013-09-20 09:55:28 [INFO] WEPIF: PermissionsEx detected! Using PermissionsEx for permissions.
2013-09-20 09:55:28 [INFO] [Vault][Chat] PermissionsEx_Chat hooked.
2013-09-20 09:55:28 [INFO] [Wither] Enabling Wither v2.0
2013-09-20 09:55:28 [INFO] [Wither] Wither v2.0 enabled!
2013-09-20 09:55:28 [INFO] [MobCatcher] Enabling MobCatcher v3.7
2013-09-20 09:55:28 [INFO] [MobCatcher] Author: Malikk
2013-09-20 09:55:28 [INFO] You are not running the correct version of Craftbukkit for this version of MobCatcher.
2013-09-20 09:55:28 [INFO] MobCatcher v3.7 is intended to be run on CB 1.5.1-R0.1-SNAPSHOT
2013-09-20 09:55:28 [INFO] Expect major bugs or failure to enable. Please do not report bugs until you are using compatible versions of CB and MobCatcher.
2013-09-20 09:55:28 [INFO] [MobCatcher] Hooked Vault 1.2.24-b304
2013-09-20 09:55:28 [INFO] [MobCatcher] Spout was not found.
2013-09-20 09:55:28 [INFO] [MobCatcher] remove stackability: true
2013-09-20 09:55:28 [SEVERE] Error occurred while enabling MobCatcher v3.7 (Is it up to date?)
java.lang.NoClassDefFoundError: net/minecraft/server/v1_5_R2/Item
    at me.jordan.mobcatcher.MobCatcher.removeStackability(MobCatcher.java:1170)
    at me.jordan.mobcatcher.MobCatcher.onEnable(MobCatcher.java:120)
    at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:217)
    at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:457)
    at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:381)
    at org.bukkit.craftbukkit.v1_5_R3.CraftServer.loadPlugin(CraftServer.java:282)
    at org.bukkit.craftbukkit.v1_5_R3.CraftServer.enablePlugins(CraftServer.java:264)
    at net.minecraft.server.v1_5_R3.MinecraftServer.j(MinecraftServer.java:304)
    at net.minecraft.server.v1_5_R3.MinecraftServer.e(MinecraftServer.java:283)
    at net.minecraft.server.v1_5_R3.MinecraftServer.a(MinecraftServer.java:243)
    at net.minecraft.server.v1_5_R3.DedicatedServer.init(DedicatedServer.java:151)
    at net.minecraft.server.v1_5_R3.MinecraftServer.run(MinecraftServer.java:382)
    at net.minecraft.server.v1_5_R3.ThreadServerApplication.run(SourceFile:573)
Caused by: java.lang.ClassNotFoundException: net.minecraft.server.v1_5_R2.Item
    at org.bukkit.plugin.java.PluginClassLoader.findClass0(PluginClassLoader.java:70)
    at org.bukkit.plugin.java.PluginClassLoader.findClass(PluginClassLoader.java:53)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 13 more
2013-09-20 09:55:28 [INFO] [Multiverse-Inventories] Enabling Multiverse-Inventories v2.5-b346
2013-09-20 09:55:28 [INFO] [Multiverse-Inventories 2.5-b346] enabled.
2013-09-20 09:55:28 [INFO] [WormholeXTreme] Enabling WormholeXTreme v1.031
2013-09-20 09:55:28 [INFO] [WormholeXTreme][v1.031] Boot sequence initiated...
2013-09-20 09:55:28 [INFO] [WormholeXTreme][v1.031] Wormhole Worlds support disabled in settings.txt, loading stargates and worlds by our self.
2013-09-20 09:55:29 [INFO] [WormholeXTreme] 49 Wormholes loaded from WormholeDB.
2013-09-20 09:55:29 [INFO] [WormholeXTreme] PermissionAlias backend: 'pex' registered!
2013-09-20 09:55:29 [INFO] [WormholeXTreme] PermissionAlias backend: 'bukkit' registered!
2013-09-20 09:55:29 [INFO] [WormholeXTreme] Attempting to use supported permissions plugin 'PermissionsEx'
2013-09-20 09:55:29 [WARNING] [WormholeXTreme] This plugin is not ready yet.
2013-09-20 09:55:29 [INFO] [WormholeXTreme] Initializing bukkit backend
2013-09-20 09:55:29 [INFO] [WormholeXTreme] Config node PERMISSIONS_BACKEND changed to 'pex'
2013-09-20 09:55:29 [INFO] [WormholeXTreme] Initializing pex backend
2013-09-20 09:55:29 [WARNING] [WormholeXTreme] This plugin is not ready yet.
2013-09-20 09:55:29 [INFO] [WormholeXTreme] Attached to PermissionsEx version 1.19.5
2013-09-20 09:55:29 [INFO] [WormholeXTreme][v1.031] Boot sequence completed
2013-09-20 09:55:29 [INFO] [Multiverse-NetherPortals] Enabling Multiverse-NetherPortals v2.5-b660
2013-09-20 09:55:29 [INFO] [Multiverse-NetherPortals] - Version 2.5-b660 Enabled - By Rigby and fernferret
2013-09-20 09:55:29 [INFO] Server permissions file permissions.yml is empty, ignoring it
2013-09-20 09:55:29 [INFO] Done (4.261s)! For help, type "help" or "?"

Here is what the script grabs:
Code:
2013-09-20 09:55:28 [SEVERE] Error occurred while enabling MobCatcher v3.7 (Is it up to date?)
java.lang.NoClassDefFoundError: net/minecraft/server/v1_5_R2/Item
        at me.jordan.mobcatcher.MobCatcher.removeStackability(MobCatcher.java:1170)
        at me.jordan.mobcatcher.MobCatcher.onEnable(MobCatcher.java:120)
        at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:217)
        at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:457)
        at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:381)
        at org.bukkit.craftbukkit.v1_5_R3.CraftServer.loadPlugin(CraftServer.java:282)
        at org.bukkit.craftbukkit.v1_5_R3.CraftServer.enablePlugins(CraftServer.java:264)
        at net.minecraft.server.v1_5_R3.MinecraftServer.j(MinecraftServer.java:304)
        at net.minecraft.server.v1_5_R3.MinecraftServer.e(MinecraftServer.java:283)

Here is what I wish the script grabbed instead:
Code:
2013-09-20 09:55:28 [INFO] [MobCatcher] Enabling MobCatcher v3.7
2013-09-20 09:55:28 [INFO] [MobCatcher] Author: Malikk
2013-09-20 09:55:28 [INFO] You are not running the correct version of Craftbukkit for this version of MobCatcher.
2013-09-20 09:55:28 [INFO] MobCatcher v3.7 is intended to be run on CB 1.5.1-R0.1-SNAPSHOT
2013-09-20 09:55:28 [INFO] Expect major bugs or failure to enable. Please do not report bugs until you are using compatible versions of CB and MobCatcher.
2013-09-20 09:55:28 [INFO] [MobCatcher] Hooked Vault 1.2.24-b304
2013-09-20 09:55:28 [INFO] [MobCatcher] Spout was not found.
2013-09-20 09:55:28 [INFO] [MobCatcher] remove stackability: true
2013-09-20 09:55:28 [SEVERE] Error occurred while enabling MobCatcher v3.7 (Is it up to date?)
java.lang.NoClassDefFoundError: net/minecraft/server/v1_5_R2/Item
    at me.jordan.mobcatcher.MobCatcher.removeStackability(MobCatcher.java:1170)
    at me.jordan.mobcatcher.MobCatcher.onEnable(MobCatcher.java:120)
    at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:217)
    at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:457)
    at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:381)
    at org.bukkit.craftbukkit.v1_5_R3.CraftServer.loadPlugin(CraftServer.java:282)
    at org.bukkit.craftbukkit.v1_5_R3.CraftServer.enablePlugins(CraftServer.java:264)
    at net.minecraft.server.v1_5_R3.MinecraftServer.j(MinecraftServer.java:304)
    at net.minecraft.server.v1_5_R3.MinecraftServer.e(MinecraftServer.java:283)
    at net.minecraft.server.v1_5_R3.MinecraftServer.a(MinecraftServer.java:243)
    at net.minecraft.server.v1_5_R3.DedicatedServer.init(DedicatedServer.java:151)
    at net.minecraft.server.v1_5_R3.MinecraftServer.run(MinecraftServer.java:382)
    at net.minecraft.server.v1_5_R3.ThreadServerApplication.run(SourceFile:573)
Caused by: java.lang.ClassNotFoundException: net.minecraft.server.v1_5_R2.Item
    at org.bukkit.plugin.java.PluginClassLoader.findClass0(PluginClassLoader.java:70)
    at org.bukkit.plugin.java.PluginClassLoader.findClass(PluginClassLoader.java:53)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 13 more

In essence, what I wish it did was grab ALL lines after a "[SEVERE]" tag until it gets to another line that starts with "2013" or a "yyyy-mm-dd" pattern. In addition, I'd like it to grab the 10 lines the precede the [SEVERE] entry to help give me context.

Thanks for any help you can provide!
# 2  
Old 09-20-2013
You said you want 10 lines before the line containing SEVERE, but your sample output only shows 8 lines. Is there a reason why the lines:
Code:
2013-09-20 09:55:28 [INFO] [Wither] Enabling Wither v2.0
2013-09-20 09:55:28 [INFO] [Wither] Wither v2.0 enabled!

were not included?

Do you only want context lines if they contain the string "MobCatcher"?
This User Gave Thanks to Don Cragun For This Post:
nbsparks
# 3  
Old 09-20-2013
I threw out 10 lines as an arbitrary number because it could really vary. I would think that 10 lines would be MORE than I need most of the time. And I wouldn't want to hardcode on "Mobcatcher" as it could be any number of server plugins that could fail or error out.

The idea is for me to be able to troubleshoot and diagnose server issues with this subset of the logfile rather than looking at this logfile, noting the time, and then sifting through the main server.log file to get the context I need to troubleshoot.

---------- Post updated at 05:13 PM ---------- Previous update was at 05:11 PM ----------

I suppose that one could parse the [SEVERE] line and then look for any lines within the previous 10-20 lines that mention any of the words used in the same line as the [SEVERE] tag, but that seems overkill and unnecessary complication.
# 4  
Old 09-20-2013
I was just confused because you said you wanted 10 lines of context before the SEVERE line, but your sample output provided 8 lines instead of 10.

The following awk script seems to do what you requested (with 10 lines of context):
Code:
awk -F"[][]" '
# pc() -- print context
# print up to 11 lines of context and clear context.
function pc(    i) {
        for(i = (lc > 10) ? lc - 10 : 1; i <= lc; i++)
                print l[i]
        lc = 0
}
!/^[1-9][0-9]{3}-[01][0-9]-[0-3][0-9]/ {
        nts = 1 # we have found a line with No Time Stamp
        if(severe) {
                pc()    # print context
                print   # print this line (which follows the SEVERE tag).
        }
        next
}
/^[1-9][0-9]{3}-[01][0-9]-[0-3][0-9]/ {
        if(nts) 
                # 1st line after one or more lines with no time stamp...
                severe = nts = lc = 0
        l[++lc] = $0
        if($2 == "SEVERE") severe = 1
}' /minecraft/server.log

If you want to run this on a Solaris/SunOS system, use /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk instead of awk.
This User Gave Thanks to Don Cragun For This Post:
nbsparks
# 5  
Old 09-21-2013
My servers run Debian 7.1 (wheezy) 64-bit. I'll give this a try.

---------- Post updated at 10:01 PM ---------- Previous update was at 09:52 PM ----------

I couldn't get that script to produce anything. I added a >filename.log at the end and it produced a 0-byte file.

---------- Post updated at 10:03 PM ---------- Previous update was at 10:01 PM ----------

I added a "set -x" to the beginning of the script to see if that would produce anything helpful:

Code:
mc@minecraftserver ~/scripts$ ./test.sh
+ awk '-F[][]' '
# pc() -- print context
# print up to 11 lines of context and clear context.
function pc(    i) {
        for(i = (lc > 10) ? lc - 10 : 1; i <= lc; i++)
                print l[i]
        lc = 0
}
!/^[1-9][0-9]{3}-[01][0-9]-[0-3][0-9]/ {
        nts = 1 # we have found a line with No Time Stamp
        if(severe) {
                pc()    # print context
                print   # print this line (which follows the SEVERE tag).
        }
        next
}
/^[1-9][0-9]{3}-[01][0-9]-[0-3][0-9]/ {
        if(nts)
                # 1st line after one or more lines with no time stamp...
                severe = nts = lc = 0
        l[++lc] = $0
        if($2 == "SEVERE") severe = 1
}' /home/mc/servers/bukkitpvt/server.log

---------- Post updated at 10:07 PM ---------- Previous update was at 10:03 PM ----------

Here is the version information for "awk" for Debian 7.1 if that's helpful.

Debian -- Details of package original-awk in wheezy
# 6  
Old 09-21-2013
With the sample input shown in the 1st message in this thread the awk program I provided produces the output:
Code:
2013-09-20 09:55:28 [INFO] [Wither] Enabling Wither v2.0
2013-09-20 09:55:28 [INFO] [Wither] Wither v2.0 enabled!
2013-09-20 09:55:28 [INFO] [MobCatcher] Enabling MobCatcher v3.7
2013-09-20 09:55:28 [INFO] [MobCatcher] Author: Malikk
2013-09-20 09:55:28 [INFO] You are not running the correct version of Craftbukkit for this version of MobCatcher.
2013-09-20 09:55:28 [INFO] MobCatcher v3.7 is intended to be run on CB 1.5.1-R0.1-SNAPSHOT
2013-09-20 09:55:28 [INFO] Expect major bugs or failure to enable. Please do not report bugs until you are using compatible versions of CB and MobCatcher.
2013-09-20 09:55:28 [INFO] [MobCatcher] Hooked Vault 1.2.24-b304
2013-09-20 09:55:28 [INFO] [MobCatcher] Spout was not found.
2013-09-20 09:55:28 [INFO] [MobCatcher] remove stackability: true
2013-09-20 09:55:28 [SEVERE] Error occurred while enabling MobCatcher v3.7 (Is it up to date?)
java.lang.NoClassDefFoundError: net/minecraft/server/v1_5_R2/Item
    at me.jordan.mobcatcher.MobCatcher.removeStackability(MobCatcher.java:1170)
    at me.jordan.mobcatcher.MobCatcher.onEnable(MobCatcher.java:120)
    at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:217)
    at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:457)
    at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:381)
    at org.bukkit.craftbukkit.v1_5_R3.CraftServer.loadPlugin(CraftServer.java:282)
    at org.bukkit.craftbukkit.v1_5_R3.CraftServer.enablePlugins(CraftServer.java:264)
    at net.minecraft.server.v1_5_R3.MinecraftServer.j(MinecraftServer.java:304)
    at net.minecraft.server.v1_5_R3.MinecraftServer.e(MinecraftServer.java:283)
    at net.minecraft.server.v1_5_R3.MinecraftServer.a(MinecraftServer.java:243)
    at net.minecraft.server.v1_5_R3.DedicatedServer.init(DedicatedServer.java:151)
    at net.minecraft.server.v1_5_R3.MinecraftServer.run(MinecraftServer.java:382)
    at net.minecraft.server.v1_5_R3.ThreadServerApplication.run(SourceFile:573)
Caused by: java.lang.ClassNotFoundException: net.minecraft.server.v1_5_R2.Item
    at org.bukkit.plugin.java.PluginClassLoader.findClass0(PluginClassLoader.java:70)
    at org.bukkit.plugin.java.PluginClassLoader.findClass(PluginClassLoader.java:53)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 13 more

(although the copy and paste processing seems to have lost the terminal color changing escape sequences) when run on a MacBook Pro running OS X. The actual output (including escape sequences) is attached.

There is nothing in this awk script that depends on any awk feature that is not required by the standards.
out.txt (2.4 KB) 
This User Gave Thanks to Don Cragun For This Post:
nbsparks
# 7  
Old 09-21-2013
weird... so I just grabbed my original data sample (that you're using) and saved that as "server2.log" and redirected the last line of the script to server2.log. Still no results.

---------- Post updated at 10:30 PM ---------- Previous update was at 10:28 PM ----------

Just to check, I switched top line of script from "#!/bin/bash" to "#!/bin/sh". Still no difference.

---------- Post updated at 10:34 PM ---------- Previous update was at 10:30 PM ----------

I only modified the path line that I know and added the set-x debug line. I did have it > to a file for a bit, but here's how it is now.

Code:
mc@minecraftserver ~/scripts$ cat test.sh
#!/bin/sh

set -x

awk -F"[][]" '
# pc() -- print context
# print up to 11 lines of context and clear context.
function pc(    i) {
        for(i = (lc > 10) ? lc - 10 : 1; i <= lc; i++)
                print l[i]
        lc = 0
}
!/^[1-9][0-9]{3}-[01][0-9]-[0-3][0-9]/ {
        nts = 1 # we have found a line with No Time Stamp
        if(severe) {
                pc()    # print context
                print   # print this line (which follows the SEVERE tag).
        }
        next
}
/^[1-9][0-9]{3}-[01][0-9]-[0-3][0-9]/ {
        if(nts)
                # 1st line after one or more lines with no time stamp...
                severe = nts = lc = 0
        l[++lc] = $0
        if($2 == "SEVERE") severe = 1
}' /home/mc/servers/bukkitpvt/server.log
mc@minecraftserver ~/scripts$ ./test.sh
+ awk -F[][]
# pc() -- print context
# print up to 11 lines of context and clear context.
function pc(    i) {
        for(i = (lc > 10) ? lc - 10 : 1; i <= lc; i++)
                print l[i]
        lc = 0
}
!/^[1-9][0-9]{3}-[01][0-9]-[0-3][0-9]/ {
        nts = 1 # we have found a line with No Time Stamp
        if(severe) {
                pc()    # print context
                print   # print this line (which follows the SEVERE tag).
        }
        next
}
/^[1-9][0-9]{3}-[01][0-9]-[0-3][0-9]/ {
        if(nts)
                # 1st line after one or more lines with no time stamp...
                severe = nts = lc = 0
        l[++lc] = $0
        if($2 == "SEVERE") severe = 1
} /home/mc/servers/bukkitpvt/server.log
mc@minecraftserver ~/scripts$
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep content between specific lines

cat file1 *FileHeader* Partition 0 Total Data Bytes 1416 Avg Bytes/Record 1416 Others 1 PRDX22.AUDIT_DATA_INFO Partition 4 Total Data Bytes 4615 Avg... (8 Replies)
Discussion started by: Veera_V
8 Replies

2. UNIX for Dummies Questions & Answers

Grep specific lines

Hello I have a file with nearly 90000 lines in x,y,z format but have some lines that I do not need to show. Is there anyway to delete those 3 lines after every 288 lines. Eg I keep the first 288 lines delete (289, 290 291); keep the next 288 lines after those and so on... Thanks (6 Replies)
Discussion started by: Madiouma Ndiaye
6 Replies

3. Shell Programming and Scripting

Script to grep logs for Errors

Hi Guys, I want to write a script which can grep the logs (server.log) from a file for Error String and output to a other file. Problems: How to know about the errors only between the current restart and not in previous as server.log has earlier restarts also? thanks for the help! Much... (5 Replies)
Discussion started by: ankur328
5 Replies

4. Shell Programming and Scripting

grep the output between specific lines

Symmetrix ID : 00000001234 Host Name : myown Identifiers Found : 5000000000000000 5000000000000001 Device Cap(MB) Attr Dir:P ------ ------- ---- ---- 1234 25886 (M) 8D:1, 9D:1 0123 25886 (M) 8D:1, 9D:1 1345 25886 (M) ... (5 Replies)
Discussion started by: maddy.san
5 Replies

5. Shell Programming and Scripting

using awk to get specific section of lines in logs

i have a log file that has the date and time that looks like this: Wed Jun 28 15:46:21 2012 test failed tailed passed passed not error panic what we want to focus on is the first 5 columns because they contain the date and time. the date and time can be anywhere on the line. in this... (6 Replies)
Discussion started by: SkySmart
6 Replies

6. Shell Programming and Scripting

Problems to print specific lines with awk and grep...HELP!

Hi all I have data like this: model: 1, misfit value: 0.74987 1 1.182 1.735 2.056 1.867 2 0.503 1.843 2.018 1.888 3 2.706 2.952 2.979 1.882 4 8.015 3.414 3.675 1.874 ... (1 Reply)
Discussion started by: fedora2011
1 Replies

7. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

8. UNIX Desktop Questions & Answers

grep lines with two specific characters somewhere in the line

I'm having trouble with extracting certain lines from a file based on whether they have all the required fields. Original file: snt:594:Sam N This bpt:2342:Bob P That lr:123 wrp:23:Whoever Person cor:794 Desired output: snt:594:Sam N This bpt:2342:Bob P That wrp:23:Whoever Person ... (3 Replies)
Discussion started by: Chthonic
3 Replies

9. Shell Programming and Scripting

Grep for lines in web logs

I want to find the unique url in a apache logs which got 404 error . I can do something like "cat apache.log|grep 404|awk '{print $2,$3}'|grep 404 this will give me say /foo.html 404 /foo.html 404 /foo.html 404 /bar.html 404 /cat.html 404 However my output should only find... (3 Replies)
Discussion started by: gubbu
3 Replies

10. Shell Programming and Scripting

retrieving specific lines from a file - can I use grep ?

Hi there, if i had a file that looked like this my_server1 red green blue yellow blue my_server2 blue blue yellow green blue my_server3 yellow (9 Replies)
Discussion started by: hcclnoodles
9 Replies
Login or Register to Ask a Question