To remove date and duplicate rows from a log file using unix commands
Hi,
I have a log file having size of 48mb.
For such a large log file. I want to get the message in a particular format which includes only unique error and exception messages.
The following things to be done :
1) To remove all the date and time from the log file
2) To remove all the duplicate rows from the same.
3) To remove all the lines that shows the message like
For e.g if my log file is like :
I want the ouput message as :
Kindly let me know how to do the same.
Regards,
pank
Next time use CODE-tags when posting code, data or logs to enhance readability and to preserve formatting like indention etc., ty.
Last edited by zaxxon; 08-03-2009 at 08:30 AM..
Reason: code tags
hi all
can anyone please let me know if there is a way to find out duplicate rows in a file. i have a file that has hundreds of numbers(all in next row).
i want to find out the numbers that are repeted in the file.
eg.
123434
534
5575
4746767
347624
5575
i want 5575
please help (3 Replies)
Hi,
I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g.
COL1 COL2 COL3
A 1234 1234
B 3k32 2322
C Xk32 TTT
A NEW XX22
B 3k32 ... (7 Replies)
Hi all,
I have written one shell script. The output file of this script is having sql output.
In that file, I want to extract the rows which are having multiple entries(duplicate rows).
For example, the output file will be like the following way.
... (7 Replies)
i have the long file more than one ns and www and mx in the line like .
i need the first ns record and first www and first mx from line .
the records are seperated with tthe ; i am try ing in awk scripting not getiing the solution.
... (4 Replies)
Gurus,
From a file I need to remove duplicate rows based on the first column data but also we need to consider a date column where we need to keep the latest date (13th column).
Ex:
Input File:
Output File:
I know how to take out the duplicates but I couldn't figure out... (5 Replies)
Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column.
e.g.
a 1
a 2
a 3
b 1
c 1
gives
b 1
c 1
but requires 11 duplicates before it deletes.
Thanks for the help
Video tutorial on how to use code tags in The UNIX... (11 Replies)
Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed.
example of my file... (3 Replies)
Hello All !
I need your help on this case,
I have a csv file with this:
ITEM105;ARI FSR;2016-02-01 08:02;243
ITEM101;ARI FSR;2016-02-01 06:02;240
ITEM032;RNO TLE;2016-02-01 11:03;320
ITEM032;RNO TLE;2016-02-02 05:43;320
ITEM032;RNO TLE;2016-02-01 02:03;320
ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
LEARN ABOUT DEBIAN
java-wrappers
JAVA-WRAPPERS(7) Java wrappers JAVA-WRAPPERS(7)NAME
java-wrappers - capacities shared by java wrapper scripts
DESCRIPTION
Most Java programs are not run directly, but through small shell scripts that take care of various things, such as finding a suitable java
environment and looking for libraries.
To ease the task of the maintainers, they rely on a library providing runtime detection, library detection and other features. This manual
page describes the common features of all those scripts, and to which point you can configure them. This is mainly done via environment
variables.
ENVIRONMENT VARIABLES
java-wrappers understands some environment variables:
JAVA_CMD
The java command that will be run. If this variable is set, it disables all lookup for a java runtime.
JAVA_BINDIR
Specifies a directory that will be looked for a java or a jdb executable (depending on the setting of JAVA_DEBUGGER). It has prece-
dence over JAVA_HOME but not over JAVA_CMD.
JAVA_HOME
A path to a java runtime. If this variable is set, all lookup for a java runtime is disabled, except that if no java executable is
found in the path, the command java is used.
JAVA_FLAVOR
A probably more easy-to-use version of the JAVA_HOME variable: instead of specifying the full path of the java runtime, you name it.
List of available flavors can be found in the file /usr/lib/java-wrappers/jvm-list.sh. See examples below.
JAVA_DEBUGGER
If this is set, the wrapper will try to pick up a java debugger rather than a java interpreter. This will fail if the jbd of the
runtime found is a stub.
JAVA_CLASSPATH
Additional classpath, will have priority over the one found by the wrapper.
JAVA_ARGS
Additional arguments to the java command. They will come before all other arguments.
FORCE_CLASSPATH
If this variable is set, it will be the only classpath. You'd better know what you are doing.
DEBUG_WRAPPER
This is probably the most important variable; if it set, the wrapper will print out useful information as it goes by its business,
such as which runtime it did find, and which command is run eventually.
JAVA_JARPATH
The path where the wrappers will go looking for jar archives. If not set, the wrapper will look into the default directory,
/usr/share/java. Warning : the wrapper will not look anywhere else than in JAVA_JARPATH. Setting it incorrectly will most probably
result in early crashes.
EXAMPLES
The examples all rely on rasterizer(1), from the package libbatik-java, but they really apply to all scripts that use java-wrappers.
Print out debugging information:
DEBUG_WRAPPER=1 rasterizer
Limit rasterizer's memory to 80 MB:
JAVA_ARGS=-Xmx80m rasterizer
Force rasterizer to run with kaffe(1):
JAVA_HOME=/usr/lib/kaffe rasterizer
The same, but using JAVA_BINDIR:
JAVA_BINDIR=/usr/lib/kaffe/bin rasterizer
Force rasterizer to run with openjdk:
JAVA_FLAVOR=openjdk rasterizer
Debug rasterizer with Sun's debugger, while printing debugging information from the wrapper:
DEBUG_WRAPPER=1 JAVA_CMD=/usr/lib/jvm/java-6-sun/bin/jdb rasterizer
BUGS
Care has been taken to make the wrappers bug-free. If that was not the case, please file a bug report against the java-wrappers package.
If you wish to submit any problem with a java executable relying on java-wrappers, please also submit the output of the command run with
DEBUG_WRAPPER=1. It will save one mail exchange and therefore potentially reduce the time it takes to fix the bug.
DEVELOPERS
There is currently no documentation about writing a wrapper script save the comments in /usr/lib/java-wrappers/java-wrappers.sh. If you
have to write one, we suggest you base yourself upon, for instance, the rasterizer wrapper script, or any other one (just pick up any
direct reverse dependency of java-wrappers and look for scripts).
SEE ALSO java(1), jdb(1)
/usr/lib/java-wrappers/java-wrappers.sh
AUTHOR
java-wrappers and its documentation were written by Vincent Fourmond <fourmond@debian.org>
Version 0.1.16 2010-05-04 JAVA-WRAPPERS(7)