Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Read files incrementally and search for particular string.
Top Forums Shell Programming and Scripting Read files incrementally and search for particular string. Post 302980280 by nareshnani211 on Thursday 25th of August 2016 04:44:03 PM
Old 08-25-2016
Read files incrementally and search for particular string.
Example

I have following requirements where i need to search for particular string from the log files.Files will be archived with number attached end to it and creates a new log file.

First Day i will ran at 8:00 AM


Filename:a.log1

[
Code:
Wed Aug 24 04:46:34 2016]Local/ESSBASE0///140276513093952/Debug(1311001)
{
All the entries with in this date range
}
[ Aug 25 04:44:53 2016]Local/ESSBASE0///140276532042048/Debug(1311001)




Filename:a.log

Code:
[ Aug 25 04:44:53 2016]Local/ESSBASE0///140276532042048/Debug(1311001)
{
All the entries with in this date  range
}
[ Aug 25 08:00:00 2016]Local/ESSBASE0///140276502567232/Debug(1056250)

At 8 i will be search for particular string and extract that string into the file.



some more enries added to a.log after 8:00 AM and after reaching particular size , file is archived and created a.log3 . when i am running on second day, i need to search entries from the log files where it read from the last time.

Instead of scanning total files, i need to scan from
Code:
[ Aug 25 08:01:00 2016]Local/ESSBASE0///140276502567232/Debug(1056250) to [ Aug 26 08:00:00 2016]Local/ESSBASE0///140276502567232/Debug(1056250)


Filename:a.log2

Code:
[ Aug 25 04:44:53 2016]Local/ESSBASE0///140276532042048/Debug(1311001)
{
All the entries with in this range
}
[ Aug 25 15:00:00 2016]Local/ESSBASE0///140276502567232/Debug(1056250)

Filename:a.log


Code:
[ Aug 25 15:00:00 2016]Local/ESSBASE0///140276502567232/Debug(1056250)
{
All the entries with in this range
}
[ Aug 26 08:00:00 2016]Local/ESSBASE0///140276502567232/Debug(1056250)




I need to search file incrementally because i have lot of files generated which takes time to read all files. instead i want to read only the entries which are added after last time read.
Last edited by jim mcnamara; 08-25-2016 at 08:14 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

What is the Best way to search files for a string??

I'm looking to seach all the files in a directory and sub-directories looking for a string. When the string is found, I want to display the filename and the entire line of that file that the string was found on. what is the best way to do this ?? I've been playing around with awk, find, and... (15 Replies)
Discussion started by: 35Soinc
15 Replies

2. Shell Programming and Scripting

Search and replace string in files

I'm trying to remove the following string from several files. <img heigth="1" width="1" border="0" src="http://myteenmovies.net/t.php?id=5540372">I'm using the following script #!/bin/bash # This script will search and replace all regular files for a string # supplied by the user and... (1 Reply)
Discussion started by: d13g0sv
1 Replies

3. Shell Programming and Scripting

to read two files, search for patterns and store the output in third file

hello i have two files temp.txt and temp_unique.text the second file consists the unique fields from the temp.txt file the strings stored are in the following form 4,4 17,12 15,65 4,4 14,41 15,65 65,89 1254,1298i'm able to run the following script to get the total count of a... (3 Replies)
Discussion started by: vaibhavkorde
3 Replies

4. Shell Programming and Scripting

Help with search string between two files

Hi, Basically i want to search for a string in file two based on the input file one and if it matches get the nextline and print the value of the field name. cat one abc xyz defcat two <src> <name="path/to/abc" test="value_version"> <new name="Y2" > </src> <src> <name="path/to/xyz"... (5 Replies)
Discussion started by: greet_sed
5 Replies

5. Shell Programming and Scripting

perl- read search and replace string from the file

Dear all, I have a number of files and each file has two sections separated by a blank line. At the top section, I have lines which describes the values of the alphabetical characters, # s #; 0.123 # p #; 12.3 # d #; -2.33 # f #; 5.68 <blank line> sssssss spfdffff sdfffffff Now I... (4 Replies)
Discussion started by: sasharma
4 Replies

6. Shell Programming and Scripting

Search several string and convert into a single line for each search string using awk command AIX?.

I need to search the file using strings "Request Type" , " Request Method" , "Response Type" and by using result set find the xml tags and convert into a single line?. below are the scenarios. Cat test Nov 10, 2012 5:17:53 AM INFO: Request Type Line 1.... (5 Replies)
Discussion started by: laknar
5 Replies

7. Shell Programming and Scripting

Search string within a file and list common words from the line having the search string

Hi, Need your help for this scripting issue I have. I am not really good at this, so seeking your help. I have a file looking similar to this: Hello, i am human and name=ABCD. How are you? Hello, i am human and name=PQRS. I am good. Hello, i am human and name=ABCD. Good bye. Hello, i... (12 Replies)
Discussion started by: royzlife
12 Replies

8. Programming

C program to read a binary file and search for a string?

Hi, I am not a C programmer. The only C exposure I have is reading and completing the exercises from the C (ANSI C ) Programming Language book:o At the moment, I am using the UNIX strings command to extract information for a binary file and grepping for a particular string and the value... (3 Replies)
Discussion started by: newbie_01
3 Replies

9. Shell Programming and Scripting

Read in search strings from text file, search for string in second text file and output to CSV

Hi guys, I have a text file named file1.txt that is formatted like this: 001 , ID , 20000 002 , Name , Brandon 003 , Phone_Number , 616-234-1999 004 , SSNumber , 234-23-234 005 , Model , Toyota 007 , Engine ,V8 008 , GPS , OFF and I have file2.txt formatted like this: ... (2 Replies)
Discussion started by: An0mander
2 Replies

10. Shell Programming and Scripting

Linux command rsync sending files incrementally

Please, i have a question about rsync command: Here is the command that i have used in my script: rsync -ratlz --rsh="/usr/bin/sshpass ssh -o StrictHostKeyChecking=no" -aAXHltzh --progress --numeric-ids --devices --rsync-path="rsync" $1 $hostname:$1 using this command, i can... (0 Replies)
Discussion started by: chercheur111
0 Replies

LEARN ABOUT CENTOS

doveadm-log

DOVEADM-LOG(1)							      Dovecot							    DOVEADM-LOG(1)

NAME

       doveadm-log - Locate, test or reopen Dovecot's log files

SYNOPSIS

       doveadm [-Dv] log errors [-s min_timestamp]
       doveadm [-Dv] log find [directory]
       doveadm [-Dv] log reopen
       doveadm [-Dv] log test

DESCRIPTION

       The  doveadm  log commands are used to locate and reopen the log files of dovecot(1).  It's also possible to test the configured targets of
       the *log_path settings.

OPTIONS

       Global doveadm(1) options:

       -D     Enables verbosity and debug messages.

       -v     Enables verbosity, including progress counter.

COMMANDS

   log errors
       doveadm log errors [-s min_timestamp]

       The log errors command is used to show the last	- up to 1,000 - errors and warnings.  If no output is generated, no errors  have  occurred
       since the last start.

       -s min_timestamp
	      An  integer  value, representing seconds since the epoch - also known as Unix timestamp.	When a min_timestamp was given, doveadm(1)
	      will only show errors occurred since that point in time.

   log find
       doveadm log find [directory]

       The log find command is used to show the location of the log files, to which dovecot(1) sends its log messages.	 If  dovecot(1)  logs  its
       messages  through  syslogd(8)  and doveadm(1) could not find any log files, you can specify the directory where your syslogd writes its log
       files.

   log reopen
       doveadm log reopen

       This command causes doveadm to reopen all log files, configured in the log_path, info_log_path and debug_log_path settings.  These settings
       are configured in /etc/dovecot/conf.d/10-logging.conf.
       This is for example useful after manually rotating the log files.

   log test
       doveadm log test

       This command causes doveadm to write the message "This is Dovecot's priority log (timestamp)" to the configured log files.  The used prior-
       ities are: debug, info, warning, error and fatal.

EXAMPLE

       This example shows how to locate the log files used by dovecot(1).

       doveadm log find
       Looking for log files from /var/log
       Debug: /var/log/dovecot.debug
       Info: /var/log/mail.log
       Warning: /var/log/mail.log
       Error: /var/log/mail.log
       Fatal: /var/log/mail.log

REPORTING BUGS

       Report bugs, including doveconf -n output, to the Dovecot Mailing List <dovecot@dovecot.org>.  Information about reporting bugs	is  avail-
       able at: http://dovecot.org/bugreport.html

SEE ALSO

       doveadm(1)

Dovecot v2.2							    2013-11-24							    DOVEADM-LOG(1)
All times are GMT -4. The time now is 04:25 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy