Sponsored Content
Top Forums Shell Programming and Scripting How to remove duplicate text blocks from a file? Post 302943253 by mahasona on Wednesday 6th of May 2015 08:08:43 PM
Old 05-06-2015
Thanks RudiC

This is what I was after, simple solution
If possible can you explain what is done using this code sample,

Code:
'!T[$0]++'

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete blocks of lines from text file

Hello, Hello Firends, I have file like below. I want to remove selected blocks say abc,pqr,lst. how can i remove those blocks from file. zone abc { blah blah blah } zone xyz { blah blah blah } zone pqr { blah blah blah } (4 Replies)
Discussion started by: nrbhole
4 Replies

2. Shell Programming and Scripting

Remove duplicate text

Hello, I have a log file which is generated by a script which looks like this: userid: 7 starttime: Sat May 24 23:24:13 CEST 2008 endtime: Sat May 24 23:26:57 CEST 2008 total time spent: 2.73072 minutes / 163.843 seconds date: Sat Jun 7 16:09:03 CEST 2008 userid: 8 starttime: Sun May... (7 Replies)
Discussion started by: dejavu88
7 Replies

3. Shell Programming and Scripting

extract blocks of text from a file

Hi, This is part of a large text file I need to separate out. I'd like some help to build a shell script that will extract the text between sets of dashed lines, write that to a new file using the whole or part of the first text string as the new file name, then move on to the next one and... (7 Replies)
Discussion started by: cajunfries
7 Replies

4. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

5. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

6. UNIX for Dummies Questions & Answers

Duplicate blocks in an inode

I have 2 duplicate blocks in an inode and I want to get rid of one of them so that I can get into my pc. The message I get is Multiply-claimed block(s) in inode 5997500: 12690101 12690101. All help is appreciated. Thanks (7 Replies)
Discussion started by: Nighttrain
7 Replies

7. Shell Programming and Scripting

Adding and removing blocks of text from file

Hello all, short story: I'm writing a script to add and remove dns records in dns files. Its on a RHEL 5.5 So far i've locked up the basic operations in a couple of functions: - validate the parameters - search for existant ip in file when adding - search for existant name records in... (6 Replies)
Discussion started by: maverick72
6 Replies

8. Shell Programming and Scripting

Blocks of text in a file - extract when matches...

I sat down yesterday to write this script and have just realised that my methodology is broken........ In essense I have..... ----------------------------------------------------------------- (This line really is in the file) Service ID: 12345 ... (7 Replies)
Discussion started by: Bashingaway
7 Replies

9. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

10. Shell Programming and Scripting

Remove duplicate occurrences of text pattern

Hi folks! I have a file which contains a 1000 lines. On each line i have multiple occurrences ( 26 to be exact ) of pattern folder#/folder#. # is depicting the line number in the file some text here folder1/folder1 some text here folder1/folder1 some text here folder1/folder1 some text... (7 Replies)
Discussion started by: martinsmith
7 Replies
ICHECK(8)						      System Manager's Manual							 ICHECK(8)

NAME
icheck - file system storage consistency check SYNOPSIS
icheck [ -s ] [ -b numbers ] [ filesystem ] DESCRIPTION
Icheck examines a file system, builds a bit map of used blocks, and compares this bit map against the free list maintained on the file sys- tem. If the file system is not specified, a set of default file systems is checked. The normal output of icheck includes a report of The total number of files and the numbers of regular, directory, block special and character special files, quota nodes, and sym- bolic links. The total number of blocks in use and the numbers of single-, double-, and triple-indirect blocks and directory blocks. The number of free blocks. The number of blocks missing; i.e. not in any file nor in the free list. The -s option causes icheck to ignore the actual free list and reconstruct a new one by rewriting the super-block of the file system. The file system should be dismounted while this is done; if this is not possible (for example if the root file system has to be salvaged) care should be taken that the system is quiescent and that it is rebooted immediately afterwards so that the old, bad in-core copy of the super- block will not continue to be used. Notice also that the words in the super-block which indicate the size of the free list and of the i- list are believed. If the super-block has been curdled these words will have to be patched. The -s option causes the normal output reports to be suppressed. Following the -b option is a list of block numbers; whenever any of the named blocks turns up in a file, a diagnostic is produced. Icheck is faster if the raw version of the special file is used, since it reads the i-list many blocks at a time. SEE ALSO
filsys(5), clri(8), dcheck(8), fsck(8), ncheck(8) DIAGNOSTICS
For duplicate blocks and bad blocks (which lie outside the file system) icheck announces the difficulty, the i-number, and the kind of block involved. If a read error is encountered, the block number of the bad block is printed and icheck considers it to contain 0. `Bad freeblock' means that a block number outside the available space was encountered in the free list. `n dups in free' means that n blocks were found in the free list which duplicate blocks either in some file or in the earlier part of the free list. BUGS
Since icheck is inherently two-pass in nature, extraneous diagnostics may be produced if applied to active file systems. Since default file systems vary with installations, icheck should use fstab(5). It believes even preposterous super-blocks and consequently can get core images. 3rd Berkeley Distribution ICHECK(8)
All times are GMT -4. The time now is 02:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy