Sponsored Content
Top Forums Shell Programming and Scripting To remove date and duplicate rows from a log file using unix commands Post 302340258 by Pank10 on Monday 3rd of August 2009 05:59:20 AM
Old 08-03-2009
To remove date and duplicate rows from a log file using unix commands

Hi,

I have a log file having size of 48mb.
For such a large log file. I want to get the message in a particular format which includes only unique error and exception messages.

The following things to be done :

1) To remove all the date and time from the log file
2) To remove all the duplicate rows from the same.
3) To remove all the lines that shows the message like

Code:
"at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)"

For e.g if my log file is like :
Code:
29 Jul 2009 04:36:53,915 [ajp-0.0.0.0-8310-415] ERROR impl.ProfileSearchManagerImpl - /JProduction/JHomeJuly17/search/user/Ixo0PRS9/_2ox.cfs (No such file or directory)
java.io.FileNotFoundException: /JProduction/JHomeJuly17/search/user/Ixo0PRS9/_2ox.cfs (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:506)
29 Jul 2009 04:36:53,956 [ajp-0.0.0.0-8310-140] ERROR com.Jsoftware - com.Jsoftware.community.ForumThreadNotFoundException: Thread 5174 could not be loaded from the database.
29 Jul 2009 04:36:58,335 [ajp-0.0.0.0-8310-239] ERROR interceptor.ExceptionMappingInterceptor - 
java.lang.NullPointerException
at com.Jsoftware.community.action.RSSPolls.execute(RSSPolls.java:95)
at sun.reflect.GeneratedMethodAccessor3418.invoke(Unknown Source)29 Jul 2009 04:36:53,956 [ajp-0.0.0.0-8310-140] ERROR com.Jsoftware - com.Jsoftware.community.ForumThreadNotFoundException: Thread 5174 could not be loaded from the database.
29 Jul 2009 04:36:58,335 [ajp-0.0.0.0-8310-239] ERROR interceptor.ExceptionMappingInterceptor - 
java.lang.NullPointerException
at com.Jsoftware.community.action.RSSPolls.execute(RSSPolls.java:95)
at sun.reflect.GeneratedMethodAccessor3418.invoke(Unknown Source)


I want the ouput message as :
Code:
ERROR impl.ProfileSearchManagerImpl - /JProduction/JHomeJuly17/search/user/Ixo0PRS9/_2ox.cfs (No such file or directory)
java.io.FileNotFoundException: /JProduction/JHomeJuly17/search/user/Ixo0PRS9/_2ox.cfs (No such file or directory)
ERROR interceptor.ExceptionMappingInterceptor - 
java.lang.NullPointerException

Kindly let me know how to do the same.


Regards,
pank

Next time use CODE-tags when posting code, data or logs to enhance readability and to preserve formatting like indention etc., ty.

Last edited by zaxxon; 08-03-2009 at 08:30 AM.. Reason: code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

duplicate rows in a file

hi all can anyone please let me know if there is a way to find out duplicate rows in a file. i have a file that has hundreds of numbers(all in next row). i want to find out the numbers that are repeted in the file. eg. 123434 534 5575 4746767 347624 5575 i want 5575 please help (3 Replies)
Discussion started by: infyanurag
3 Replies

2. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

3. HP-UX

How to get Duplicate rows in a file

Hi all, I have written one shell script. The output file of this script is having sql output. In that file, I want to extract the rows which are having multiple entries(duplicate rows). For example, the output file will be like the following way. ... (7 Replies)
Discussion started by: raghu.iv85
7 Replies

4. Shell Programming and Scripting

awk script to remove duplicate rows in line

i have the long file more than one ns and www and mx in the line like . i need the first ns record and first www and first mx from line . the records are seperated with tthe ; i am try ing in awk scripting not getiing the solution. ... (4 Replies)
Discussion started by: kiranmosarla
4 Replies

5. UNIX for Dummies Questions & Answers

Removing duplicate rows & selecting only latest date

Gurus, From a file I need to remove duplicate rows based on the first column data but also we need to consider a date column where we need to keep the latest date (13th column). Ex: Input File: Output File: I know how to take out the duplicates but I couldn't figure out... (5 Replies)
Discussion started by: shash
5 Replies

6. Shell Programming and Scripting

remove consecutive duplicate rows

I have some data that looks like, 1 3300665.mol 3300665 5177008 102.093 2 3300665.mol 3300665 5177008 102.093 3 3294015.mol 3294015 5131552 102.114 4 3294015.mol 3294015 5131552 102.114 5 3293734.mol 3293734 5129625 104.152 6 3293734.mol ... (13 Replies)
Discussion started by: LMHmedchem
13 Replies

7. UNIX for Dummies Questions & Answers

Remove duplicate rows when >10 based on single column value

Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column. e.g. a 1 a 2 a 3 b 1 c 1 gives b 1 c 1 but requires 11 duplicates before it deletes. Thanks for the help Video tutorial on how to use code tags in The UNIX... (11 Replies)
Discussion started by: informaticist
11 Replies

8. Shell Programming and Scripting

Remove duplicate rows based on one column

Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed. example of my file... (3 Replies)
Discussion started by: clarissab
3 Replies

9. Shell Programming and Scripting

Removing Duplicate Rows in a file

Hello I have a file with contents like this... Part1 Field2 Field3 Field4 (line1) Part2 Field2 Field3 Field4 (line2) Part3 Field2 Field3 Field4 (line3) Part1 Field2 Field3 Field4 (line4) Part4 Field2 Field3 Field4 (line5) Part5 Field2 Field3 Field4 (line6) Part2 Field2 Field3 Field4... (7 Replies)
Discussion started by: ekbaazigar
7 Replies

10. UNIX for Dummies Questions & Answers

Log file - Delete duplicate line & keep last date

Hello All ! I need your help on this case, I have a csv file with this: ITEM105;ARI FSR;2016-02-01 08:02;243 ITEM101;ARI FSR;2016-02-01 06:02;240 ITEM032;RNO TLE;2016-02-01 11:03;320 ITEM032;RNO TLE;2016-02-02 05:43;320 ITEM032;RNO TLE;2016-02-01 02:03;320 ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
DOVEADM-LOG(1)							      Dovecot							    DOVEADM-LOG(1)

NAME
doveadm-log - Locate, test or reopen Dovecot's log files SYNOPSIS
doveadm [-Dv] log errors doveadm [-Dv] log find [directory] doveadm [-Dv] log reopen doveadm [-Dv] log test DESCRIPTION
The doveadm log commands are used to locate and reopen the log files of dovecot(1). It's also possible to test the configured targets of the *log_path settings. OPTIONS
Global doveadm(1) options: -D Enables verbosity and debug messages. -v Enables verbosity, including progress counter. COMMANDS
log errors doveadm log errors The log errors command is used to show the last - up to 1,000 - errors and warnings. If no output is generated, no errors have occurred since the last start. log find doveadm log find [directory] The log find command is used to show the location of the log files, to which dovecot(1) sends its log messages. If dovecot(1) logs its messages through syslogd(8) and doveadm(1) could not find any log files, you can specify the directory where your syslogd writes its log files. log reopen doveadm log reopen This command causes doveadm to reopen all log files, configured in the log_path, info_log_path and debug_log_path settings. These settings are configured in /etc/dovecot/conf.d/10-logging.conf. This is for example useful after manually rotating the log files. log test doveadm log test This command causes doveadm to write the message "This is Dovecot's priority log (timestamp)" to the configured log files. The used prior- ities are: debug, info, warning, error and fatal. EXAMPLE
This example shows how to locate the log files used by dovecot(1). doveadm log find Looking for log files from /var/log Debug: /var/log/dovecot.debug Info: /var/log/mail.log Warning: /var/log/mail.log Error: /var/log/mail.log Fatal: /var/log/mail.log REPORTING BUGS
Report bugs, including doveconf -n output, to the Dovecot Mailing List <dovecot@dovecot.org>. Information about reporting bugs is avail- able at: http://dovecot.org/bugreport.html SEE ALSO
doveadm(1) Dovecot v2.1 2012-02-22 DOVEADM-LOG(1)
All times are GMT -4. The time now is 03:27 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy