Shell Programming and Scripting

There wasn't any duplicate text in your sample input file. And, it isn't clear if you want to remove text that is duplicated or if you want to remove text along with the opening and closing XML <Text> tags where the text and tags are duplicated.

Furthermore, although gawk on Linux systems uses the entire string assigned to the RS variable as the input record separator, the standards say that the behavior is unspecified if RS is more than one character. The version of awk I'm using only uses the first character of RS, so the structure of my code is slightly different from that suggested by cgkmal.

Your sample input included the line:

Code:

<Text Text_ID="10154713369385165_10154714426085165" From="415855878601070" Created="2015-10-30T23:27:48+0000" use_count="1">This is the fourth text........</Text>

but that line does not appear in the output that you said should be produced. Why shouldn't this line be included in the output?

Assuming that you just want to consider the text between the tags (and not the tags themselves) when looking for duplicates, you could try something like:

Code:

awk '
BEGIN {	RS = ">"
}
$1 == "<Text" {
	open_tag = substr($0 RS, index($0, "<"))
	next
}
NF >= limit && !($0 in seen) {
	print open_tag $0 RS
	seen[$0]
}' limit=4 file

If the sample input you showed us in post #1 is contained in a file named file, it produces the output:

Code:

<Text Text_ID="10155645315850165_10155645333075165" From="460350337463650" Created="2014-10-16T17:05:37+0000" use_count="536">This is the first text</Text>
<Text Text_ID="10155645315850165_10155645317025165" From="1626711840908498" Created="2014-10-16T17:01:02+0000" use_count="408">This is the second text</Text>
<Text Text_ID="10155645315850165_10155645320000165" From="1481727095388591" Created="2014-10-16T17:02:04+0000" use_count="1064">This is the third text
If counted 
GOT IT... ����</Text>
<Text Text_ID="10154713369385165_10154714450825165" From="464236763734179" Created="2015-10-30T23:34:47+0000" use_count="1">This is is just a sample text......</Text>
<Text Text_ID="10154713369385165_10154714444345165" From="642181809247720" Created="2015-10-30T23:31:48+0000" use_count="1">This is just another sample text.......</Text>
<Text Text_ID="10154713369385165_10154714426085165" From="415855878601070" Created="2015-10-30T23:27:48+0000" use_count="1">This is the fourth text........</Text>
<Text Text_ID="10154713369385165_10154714406055165" From="10202898434142187" Created="2015-10-30T23:23:34+0000" use_count="1">Jor se Bharat Mata ki jai</Text>

(including the line shown in red that was not included in your desired output).

Note that this code assumes that there is no whitespace between the last word in your text and the closing </Text> tag, that there are no > characters in the text in your file, and that the only tags in your XML file are opening and closing text tags (<Text ...> and </Text>, respectively). The code cgkmal provided makes these same assumptions and additionally assumes that there is no whitespace after the end of the opening text tag before the first word of the text, that there is nothing other than a <newline> character after a closing text tag, and that there are always five words in an opening text tag.

Yes, the following text was missed out in my output. Sorry for the confusion.

Code:

<Text Text_ID="10154713369385165_10154714426085165" From="415855878601070" Created="2015-10-30T23:27:48+0000" use_count="1">This is the fourth text........</Text>

should appear in the output. Thanks a lot for the correction. . I made the correction in the posting accordingly.

Is there some way I can remove the non-Roman characters from the texts and remove the blanks in between line if any and followed by trimming of tabs and spaces at the end of each line if any?

I'm glad to hear that I understood part of your original requirements.

Does the code I suggested work for you?

What do you consider to be duplicated text? If I didn't guess correctly, please show us some sample input that contains duplicated text and explain clearly what your requirements for removing duplicates.

UPDATE: I see that you added text to your last post after I started answering it...

PLEASE stop making us guess at your requirements. State them clearly with examples! If you refuse to make any attempts to write your own code and expect volunteers to write all of your code for you, you could at least clearly define your terms and show us examples of what you're trying to do. You can start with:

Thanks a lot it worked.