UNIX for Dummies Questions & Answers

Hi unix masters,

Im needing some guidance or a small code to enlight my problem.

Problem Example:
I have 3 different text ascii files. At each file, inside the text
have repeater marks.

--text 1 start--
123 -> mark
anytextanytext
anythinganything

123 ->mark
blahblah
blah

...
123->mark
...

--text 1 end--

Each file is different BUT some blocks of text, after the marker, are repeated
in the other file.

What im planning to do is merge all 3 files in 1. Find the repeated blocks
between the marks and delete the repeated parts leaving only 1 of
them. And dont delete the others non repeated blocks. I dont need sort the
blocks. Save the result in a new file.

I couldnt find a sourcecode (any language) or an program to do
what i need or to guide me. In the example is with only 3 files but
i need run the rotine at 350 files with thousand of blocks inside each file *dies*.

Should i use bash, pearl, python, emacs, some text processor? Some one
know a done code close to what i need to download? My skills are enough
to modify a file but i unable for now to code something from zero.

Thanks in advance guys!

Give more clue : please provide a representative sample of input and expected output

Ok!

--File 1

123
asdf
qwert
zxcv

123
yuio
hjkl
bnm

--File 2
123
rtyu
fghj
vbnm

123
asdf
qwert
zxcv

--Result
123
yuio
hjkl
bnm

123
rtyu
fghj
vbnm

123
asdf
qwert
zxcv

*2 unique blocks + 1 block of repeated

Do your blocks have a fix length (always 4 lines + 1 empty line) ?
Or does it happend that some block can have more or less than 4 lines ?

Code:

awk 'BEGIN{RS="";FS="\n"}
! a[$0]++ {b[$1]=b[$1]?b[$1] FS FS $0:$0} 
END {for ( i in b) print b[i] FS}' file1 file2

Hi Ctsgnb,
The blocks dont have a fixed size. The range is 3~15 lines. And some blocks repeat 2 or more times in same file and in other file too. I have many files to process. So i guess i need join all in one and then process the repeated blocks. The pattern to define the start of block is 123 and a empty line at end of block.

Rdcwayx, i will test the command submited. But i have thousand of files to process. So execute by pairs isnt fast enough.

I was doing some more search at this forum and started to think about use awk to solve the problem. I never used this tool. Looks very versatile.

Thanks guys for the patience and help for a novice.

---------- Post updated at 08:31 AM ---------- Previous update was at 05:45 AM ----------

Rdcwayx!

Sorry, im slowmind as hell. I tested the command line. Worked.
I will check the consistency of output file but looking quickly worked.

Next will run in large scale, all files, and report the result.

---------- Post updated at 08:59 AM ---------- Previous update was at 08:31 AM ----------

God. Worked Perfectly!
Thank you very much. I downloaded and will start to read the awk manual right now.
Is a nice tool. Looks light and clean.

Thanks again guys!

If you have more files, just add the file names one by one.

Code:

awk 'BEGIN{RS="";FS="\n"}
! a[$0]++ {b[$1]=b[$1]?b[$1] FS FS $0:$0} 
END {for ( i in b) print b[i] FS}' file1 file2 file3 file4 ...

or the efficient way

Code:

awk 'BEGIN{RS="";FS="\n"}
! a[$0]++ {b[$1]=b[$1]?b[$1] FS FS $0:$0} 
END {for ( i in b) print b[i] FS}' file*