To remove date and duplicate rows from a log file using unix commands
Hi,
I have a log file having size of 48mb.
For such a large log file. I want to get the message in a particular format which includes only unique error and exception messages.
The following things to be done :
1) To remove all the date and time from the log file
2) To remove all the duplicate rows from the same.
3) To remove all the lines that shows the message like
For e.g if my log file is like :
I want the ouput message as :
Kindly let me know how to do the same.
Regards,
pank
Next time use CODE-tags when posting code, data or logs to enhance readability and to preserve formatting like indention etc., ty.
Last edited by zaxxon; 08-03-2009 at 08:30 AM..
Reason: code tags
hi all
can anyone please let me know if there is a way to find out duplicate rows in a file. i have a file that has hundreds of numbers(all in next row).
i want to find out the numbers that are repeted in the file.
eg.
123434
534
5575
4746767
347624
5575
i want 5575
please help (3 Replies)
Hi,
I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g.
COL1 COL2 COL3
A 1234 1234
B 3k32 2322
C Xk32 TTT
A NEW XX22
B 3k32 ... (7 Replies)
Hi all,
I have written one shell script. The output file of this script is having sql output.
In that file, I want to extract the rows which are having multiple entries(duplicate rows).
For example, the output file will be like the following way.
... (7 Replies)
i have the long file more than one ns and www and mx in the line like .
i need the first ns record and first www and first mx from line .
the records are seperated with tthe ; i am try ing in awk scripting not getiing the solution.
... (4 Replies)
Gurus,
From a file I need to remove duplicate rows based on the first column data but also we need to consider a date column where we need to keep the latest date (13th column).
Ex:
Input File:
Output File:
I know how to take out the duplicates but I couldn't figure out... (5 Replies)
Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column.
e.g.
a 1
a 2
a 3
b 1
c 1
gives
b 1
c 1
but requires 11 duplicates before it deletes.
Thanks for the help
Video tutorial on how to use code tags in The UNIX... (11 Replies)
Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed.
example of my file... (3 Replies)
Hello All !
I need your help on this case,
I have a csv file with this:
ITEM105;ARI FSR;2016-02-01 08:02;243
ITEM101;ARI FSR;2016-02-01 06:02;240
ITEM032;RNO TLE;2016-02-01 11:03;320
ITEM032;RNO TLE;2016-02-02 05:43;320
ITEM032;RNO TLE;2016-02-01 02:03;320
ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
LEARN ABOUT MOJAVE
git-check-attr
GIT-CHECK-ATTR(1) Git Manual GIT-CHECK-ATTR(1)NAME
git-check-attr - Display gitattributes information
SYNOPSIS
git check-attr [-a | --all | attr...] [--] pathname...
git check-attr --stdin [-z] [-a | --all | attr...]
DESCRIPTION
For every pathname, this command will list if each attribute is unspecified, set, or unset as a gitattribute on that pathname.
OPTIONS -a, --all
List all attributes that are associated with the specified paths. If this option is used, then unspecified attributes will not be
included in the output.
--cached
Consider .gitattributes in the index only, ignoring the working tree.
--stdin
Read pathnames from the standard input, one per line, instead of from the command-line.
-z
The output format is modified to be machine-parseable. If --stdin is also given, input paths are separated with a NUL character instead
of a linefeed character.
--
Interpret all preceding arguments as attributes and all following arguments as path names.
If none of --stdin, --all, or -- is used, the first argument will be treated as an attribute and the rest of the arguments as pathnames.
OUTPUT
The output is of the form: <path> COLON SP <attribute> COLON SP <info> LF
unless -z is in effect, in which case NUL is used as delimiter: <path> NUL <attribute> NUL <info> NUL
<path> is the path of a file being queried, <attribute> is an attribute being queried and <info> can be either:
unspecified
when the attribute is not defined for the path.
unset
when the attribute is defined as false.
set
when the attribute is defined as true.
<value>
when a value has been assigned to the attribute.
Buffering happens as documented under the GIT_FLUSH option in git(1). The caller is responsible for avoiding deadlocks caused by
overfilling an input buffer or reading from an empty output buffer.
EXAMPLES
In the examples, the following .gitattributes file is used:
*.java diff=java -crlf myAttr
NoMyAttr.java !myAttr
README caveat=unspecified
o Listing a single attribute:
$ git check-attr diff org/example/MyClass.java
org/example/MyClass.java: diff: java
o Listing multiple attributes for a file:
$ git check-attr crlf diff myAttr -- org/example/MyClass.java
org/example/MyClass.java: crlf: unset
org/example/MyClass.java: diff: java
org/example/MyClass.java: myAttr: set
o Listing all attributes for a file:
$ git check-attr --all -- org/example/MyClass.java
org/example/MyClass.java: diff: java
org/example/MyClass.java: myAttr: set
o Listing an attribute for multiple files:
$ git check-attr myAttr -- org/example/MyClass.java org/example/NoMyAttr.java
org/example/MyClass.java: myAttr: set
org/example/NoMyAttr.java: myAttr: unspecified
o Not all values are equally unambiguous:
$ git check-attr caveat README
README: caveat: unspecified
SEE ALSO gitattributes(5).
GIT
Part of the git(1) suite
Git 2.17.1 10/05/2018 GIT-CHECK-ATTR(1)