Advise to print lines before and after patterh match and checking and removing duplicate files
Hi,
I have a script that search log files for the string CORRUPT and I then print 10 lines before and after the pattern match. Let's call this pattern_match.ksh
First I do a
which gives me the list of files that has the string "CORRUPTION DETECTED" in them
Then using a while loop, I do something like below. Ignore the ..., am just showing the part where I print the matching pattern and lines before and after the pattern match.
So, at the moment, it is doing what I am after, that is, so now I have extracts of files that contain the "CORRUPTION DETECTED" string with +/- 10 lines from the pattern match.
There is also the sed one liner example
but unfortunately I can't get the proper syntax to get it to print more lines before the pattern match. I know how to print more lines after the pattern match but using several
. Is there a short version for sed if you want to do
which is to print 10 lines after the match
I don't have the grep version also that will allow me to grep and print lines before and after match, i.e.the
thingy.
Hence, I end up doing a grep -in and then doing +/- and sed -n is a long and crude way of doing what I am after but I don't know of any other way of doing it the way I can understand it I am having a hard time understanding the sed and awk one-liners. Also, my method makes it simpler for if I want to print more than +/- 10 lines, I simply change the lines that do the +/- section.
However, there are flaws to my script as always
If for example the log file is small that it only has 10 lines for example, the sed -n "${str_before},${str_after}p" will then give error. I can't find a way of getting sed to check for valid line numbers to do a sed on, is there?
Because the files that I am doing grep on doesn't get deleted until after a month or so, and I run this corruption check script daily, I do end up with several duplicate files named differently.
How do I check and remove duplicate files that are named differently? I used the following script and running md5sum. Script is name x.ksh at the moment, will change it later :-)
Sample run of the x.ksh script with some example log files is as below:
Is there any other way of checking for duplicate files? At the moment, I run pattern_match.ksh and then call x.ksh from there. My question is, is there a way to check for duplicate files 'immediately' instead of how am doing it at the moment running x.ksh?
For example, if I already have files log.1 to log.50 and they have different checksum meaning they are all different files, non-duplicated. Then the sed/pattern_match.ksh generates file log.51, I want to be able to check log.51 against log.1 to log.50 that it isn't a duplicate of any of them. Or is this already exactly what my x.ksh script is doing and am just over-complicating stuff I hope I am explaining this correctly.
Hi,
I am trying to remove duplicate lines from a file. For example the contents of example.txt is:
this is a test
2342
this is a test
34343
this is a test
43434
and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with:
2342... (4 Replies)
Hi,
I need to concatenate three files in to one destination file.In this if some duplicate data occurs it should be deleted.
eg:
file1:
-----
data1 value1
data2 value2
data3 value3
file2:
-----
data1 value1
data4 value4
data5 value5
file3:
-----
data1 value1
data4 value4 (3 Replies)
Hello,
I am in need of removing duplicate lines from within a file per section.
File:
ABC1 012345 header
ABC2 7890-000
ABC3 012345 Header Table
ABC4
ABC5 593.0000 587.4800
ABC5 593.5000 587.6580 <= dup need to remove
ABC5 593.5000 ... (5 Replies)
So I have two files. The first file, file1.txt, has lines of numbers separated by commas.
file1.txt
10,2,30,50
22,6,3,15,16,100
73,55
78,40,33,30,11
73,55
99,82,85
22,6,3,15,16,100
The second file, file2.txt, has sentences.
file2.txt
"the cat is fat"
"I like eggs"
"fish live in... (6 Replies)
Hi,
I have attached an output file which is some kind of database file mapping. It is basically like an allocation mapping of a tablespace and its datafile/s.
The output is generated by the SQL script that I found from 401 Authorization Required
Excerpts of the file are as below:
... (2 Replies)