Sponsored Content
Top Forums Shell Programming and Scripting How to remove duplicate text blocks from a file? Post 302943253 by mahasona on Wednesday 6th of May 2015 08:08:43 PM
Old 05-06-2015
Thanks RudiC

This is what I was after, simple solution
If possible can you explain what is done using this code sample,

Code:
'!T[$0]++'

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete blocks of lines from text file

Hello, Hello Firends, I have file like below. I want to remove selected blocks say abc,pqr,lst. how can i remove those blocks from file. zone abc { blah blah blah } zone xyz { blah blah blah } zone pqr { blah blah blah } (4 Replies)
Discussion started by: nrbhole
4 Replies

2. Shell Programming and Scripting

Remove duplicate text

Hello, I have a log file which is generated by a script which looks like this: userid: 7 starttime: Sat May 24 23:24:13 CEST 2008 endtime: Sat May 24 23:26:57 CEST 2008 total time spent: 2.73072 minutes / 163.843 seconds date: Sat Jun 7 16:09:03 CEST 2008 userid: 8 starttime: Sun May... (7 Replies)
Discussion started by: dejavu88
7 Replies

3. Shell Programming and Scripting

extract blocks of text from a file

Hi, This is part of a large text file I need to separate out. I'd like some help to build a shell script that will extract the text between sets of dashed lines, write that to a new file using the whole or part of the first text string as the new file name, then move on to the next one and... (7 Replies)
Discussion started by: cajunfries
7 Replies

4. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

5. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

6. UNIX for Dummies Questions & Answers

Duplicate blocks in an inode

I have 2 duplicate blocks in an inode and I want to get rid of one of them so that I can get into my pc. The message I get is Multiply-claimed block(s) in inode 5997500: 12690101 12690101. All help is appreciated. Thanks (7 Replies)
Discussion started by: Nighttrain
7 Replies

7. Shell Programming and Scripting

Adding and removing blocks of text from file

Hello all, short story: I'm writing a script to add and remove dns records in dns files. Its on a RHEL 5.5 So far i've locked up the basic operations in a couple of functions: - validate the parameters - search for existant ip in file when adding - search for existant name records in... (6 Replies)
Discussion started by: maverick72
6 Replies

8. Shell Programming and Scripting

Blocks of text in a file - extract when matches...

I sat down yesterday to write this script and have just realised that my methodology is broken........ In essense I have..... ----------------------------------------------------------------- (This line really is in the file) Service ID: 12345 ... (7 Replies)
Discussion started by: Bashingaway
7 Replies

9. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

10. Shell Programming and Scripting

Remove duplicate occurrences of text pattern

Hi folks! I have a file which contains a 1000 lines. On each line i have multiple occurrences ( 26 to be exact ) of pattern folder#/folder#. # is depicting the line number in the file some text here folder1/folder1 some text here folder1/folder1 some text here folder1/folder1 some text... (7 Replies)
Discussion started by: martinsmith
7 Replies
RDSWAP(1)						      General Commands Manual							 RDSWAP(1)

["NAME"]
       rdswap - a multi-language RD documents support tool

["SYNOPSIS"]
       rdswap [ -h | -v ] filename ...

["DESCRIPTION"]
       This tool is written to support you to write multi-language documents using the Ruby-Document-Format (RD).

       The idea for such a tool was originated by Minero Aoki, how has thought about, how to make life easier for developers who have to write and
       maintain scripts in more than one language.

       You have to specify at least two filenames on the command line. One containing the Ruby script, the second containing a translated  RD.	If
       the  script  does  not  end  with `.rb', it has to be the first filename mentioned on the command line! In opposition, all files containing
       translations must not ending with `.rb'! They should use a extension that describes the language. So that would give us the following  pic-
       ture:

	      o sample.rb : Script contains the original documentation.

	      o sample.jp : Documentation written in Japanese.

	      o sample.de : Translation to German.

       The  tool doesn't care about the language extensions. You can name them as you like! So the file containing the Japanese translation above,
       could also be names e.g. `sample.japan' or even `japantranslation.japan'.

       For every translation file, a new file will be created. The name is build from the script filename plus the language extension. So  regard-
       ing the example above, following files would be created:

	      o sample.rb.jp

	      o sample.rb.de

       or, given the alternative translation filename as mentioned above...

	      o sample.rb.japan

   ["How does it work?"]
       The  contents  of all files will be split into source and RD blocks. The source of the translation files, will be discarded! Every RD block
       may be of a certain type. The type will be taken from the contents directly following the `=begin' on the same line. If	there  is  only  a
       lonely `=begin' on a line by itself, the type of the block is `nil'. That means in
	   # File sample.rd
	   :
	   =begin
	    bla bla
	   =end
	   :
	   =begin whatever or not
	    blub blub
	   =end
	   :

       the first block would be of type `nil' and the second one of type `whatever or not'.

       Block  types  are  important for the translation. If a source will be generated from a script and a translation file, only these blocks are
       taken from the translation files, that comes in the right sequence and contains the same type as the block in the script! For example:
	   # File sample.rb
	   :
	   =begin gnark
	    Some comment
	   =end
	   :
	   =begin
	    block 2
	   =end
	   :
	   =begin
	    block 3
	   =end
	   :

	   # File sample.de
	   :
	   =begin
	    Block zwei
	   =end
	   :
	   =begin
	    Block drei
	   =end
	   :

       Here, the first block of `sample.rb' will *not* be translated, as there is no translation block with that type in sample.de! So	the  first
       block  would  be  inserted as-it-is into the translated script. The blocks afterwards, however, are translated as the block type does match
       (it is `nil' there).

       Attention: In a translation file, a second block will only be used, if a first one was already used (matched). A third block will  only	be
       used, if a second one was used already!

       That  means,  if the first block of `sample.de' would be of type e.g. `Never match', then no block would ever be taken to replace anyone of
       `sample.rb'.

   ["OPTIONS"]
       ["-h"]
	      shows this help text.

       ["-v"]
	      shows some more text during processing.

       ["filename"]
	      means a file, that contains RD and/or Ruby code.

   ["EXAMPLES"]
	   rdswap -v sample.rb sample.ja sample.de
	   rdswap -v sample.ja sample.rb sample.de
	   rdswap -v sample.ja sample.de sample.rb
	   rdswap -v sample.??

   ["AUTHORS"]
       Clemens Hintze <c.hintze@gmx.net>.

								     June 2012								 RDSWAP(1)
All times are GMT -4. The time now is 10:45 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy