Sponsored Content
Top Forums Shell Programming and Scripting [uniq + awk?] How to remove duplicate blocks of lines in files? Post 302557107 by raidzero on Tuesday 20th of September 2011 11:53:13 AM
Old 09-20-2011
[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example:

input:
Code:
<string-array name="threeItems">
<item>item1</item>
<item>item2</item>
<item>item3</item>
</string-array>
<string-array name="twoItems">
<item>item1</item>
<item>item2</item>
</string-array>
<string-array name="threeItems">
<item>item1</item>
<item>item2</item>
<item>item3</item>
</string-array>

I cannot have two arrays by the name of "threeItems". This is the desired output:

Code:
<string-array name="threeItems">
<item>item1</item>
<item>item2</item>
<item>item3</item>
</string-array>
<string-array name="twoItems">
<item>item1</item>
<item>item2</item>
</string-array>

The arrays can be taken out of order, but the array contents need to stay intact. I have been using awk to pull arrays and all their elements from several files into one like this

Code:
awk '/string-array name/,/string-array>/' $file

but it seems that the same array appears in more than one file ><

Thanks for any tips!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Hi, is it possible to remove all duplicate lines from all txt files in a specific folder? This is too hard for me maybe someone could help. lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50 each textfile has lines with text. I want all lines of all textfiles... (8 Replies)
Discussion started by: lowmaster
8 Replies

2. UNIX for Dummies Questions & Answers

deleteing duplicate lines sing uniq while ignoring a column

I have a data set that has 4 columns, I want to know if I can delete duplicate lines while ignoring one of the columns, for example 10 chr1 ASF 30 15 chr1 ASF 20 5 chr1 ASF 30 6 chr2 EBC 15 4 chr2 EBC 30 ... I want to know if I can delete duplicate lines while ignoring column 1, so the... (5 Replies)
Discussion started by: japaneseguitars
5 Replies

3. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Input: hello hello hello hello monkey donkey hello hello drink dance drink Output should be: hello hello monkey donkey drink dance (9 Replies)
Discussion started by: cola
9 Replies

4. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

5. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

6. Shell Programming and Scripting

Cant get awk 1liner to remove duplicate lines from Delimited file, get "event not found" error..help

Hi, I am on a Solaris8 machine If someone can help me with adjusting this awk 1 liner (turning it into a real awkscript) to get by this "event not found error" ...or Present Perl solution code that works for Perl5.8 in the csh shell ...that would be great. ****************** ... (3 Replies)
Discussion started by: andy b
3 Replies

7. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

8. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22... (3 Replies)
Discussion started by: mahasona
3 Replies

9. Shell Programming and Scripting

Remove lines from output in files using awk

I have two large files (~250GB) that I am trying to remove the where GT: 0/0 or 1/1 or 2/2 for both files. I was going to use a bash with the below awk, which I think will find each line but how do I remove that line is that condition is found? Thank you :). Input 20 60055 . A ... (4 Replies)
Discussion started by: cmccabe
4 Replies

10. Shell Programming and Scripting

How to put the command to remove duplicate lines in my awk script?

I create a CGI in bash/html. My awk script looks like : echo "<table>" for fn in /var/www/cgi-bin/LPAR_MAP/*; do echo "<td>" echo "<PRE>" awk -F',|;' -v test="$test" ' NR==1 { split(FILENAME ,a,""); } $0 ~ test { if(!header++){ ... (12 Replies)
Discussion started by: Tim2424
12 Replies
UI::Dialog::Backend::Zenity(3pm)			User Contributed Perl Documentation			  UI::Dialog::Backend::Zenity(3pm)

NAME
UI::Dialog::Backend::Zenity SYNOPSIS
use UI::Dialog::Backend::Zenity; my $d = new UI::Dialog::Backend::Zenity ( backtitle => 'Demo', title => 'Default' ); $d->msgbox( title => 'Welcome!', text => 'Welcome one and all!' ); ABSTRACT
Zenity is the speedy, slick, and ultimately cool GTK2 dialog variant that is destined to obselete the current GNOME dialog variant; GDia- log. This is an OOPerl wrapper of the Zenity application. DESCRIPTION
UI::Dialog::Backend::Zenity is the UI::Dialog backend for the new GNOME dialog variant. While this module is used through UI::Dialog or any other loader module only the compatible methods are ever accessible. However, when using this module directly in your application (as in the SYNOPSIS example) you are given access to all the options and features of the real zenity(1) application. EXPORT
None INHERITS
UI::Dialog::Backend CONSTRUCTOR
new( @options ) EXAMPLE my $d = new( title => 'Default Title', backtitle => 'Backtitle', width => 65, height => 20, listheight => 5 ); DESCRIPTION This is the Class Constructor method. It accepts a list of key => value pairs and uses them as the defaults when interacting with the various widgets. RETURNS A blessed object reference of the UI::Dialog::Backend::Zenity class. OPTIONS The (...)'s after each option indicate the default for the option. An * denotes support by all the widget methods on a per-use policy defaulting to the values decided during object creation. debug = 0,1,2(0) literal = 0,1(0) backtitle = "backtitle" ('') * title = "title" ('') * height = d+(0) * width = d+(0) * display = ":0" ('') * name = "wmname" ('') * class = "wmclass" ('') * beepbin = "/usr/bin/beep" ('') beepbefore = 0,1(0) * beepafter = 0,1(0) * WIDGET METHODS
yesno( ) question( ) EXAMPLE if ($d->yesno( text => 'A binary type question?') ) { # user pressed yes } else { # user pressed no or cancel } DESCRIPTION Present the end user with a message box that has two buttons, OK and CANCEL (aka: Yes and No). yesno() is a wrapper for ques- tion(). RETURNS TRUE(1) for a response of YES or FALSE(0) for anything else. msgbox( ) EXAMPLE $d->msgbox( text => 'A simple message' ); DESCRIPTION Pesent the end user with a message box that has an OK button. RETURNS TRUE(1) for a response of OK or FALSE(0) for anything else. password( ) EXAMPLE my $string = $d->password( text => 'Enter some (hidden) text.' ); DESCRIPTION Present the end user with a text input field that doesn't reveal the input (except to the script) and a message. RETURNS a SCALAR if the response is OK and FALSE(0) for anything else. inputbox( ) EXAMPLE my $string = $d->inputbox( text => 'Please enter some text...', entry => 'this is the input field' ); DESCRIPTION Present the end user with a text input field and a message. RETURNS a SCALAR if the response is OK and FALSE(0) for anything else. textbox( ) EXAMPLE $d->textbox( path => '/path/to/a/text/file' ); DESCRIPTION Present the end user with a simple scrolling box containing the contents of the given text file. RETURNS TRUE(1) if the response is OK and FALSE(0) for anything else. editbox( ) EXAMPLE $d->editbox( path => '/path/to/a/text/file' ); DESCRIPTION Present the end user with an editable textbox containing the contents of the given text file. RETURNS A SCALAR containing the edited text if the response is OK and FALSE(0) for anything else. menu( ) EXAMPLE my $selection1 = $d->menu( text => 'Select one:', list => [ 'tag1', 'item1', 'tag2', 'item2', 'tag3', 'item3' ] ); DESCRIPTION Present the user with a selectable list. RETURNS a SCALAR of the chosen tag if the response is OK and FALSE(0) for anything else. checklist( ) EXAMPLE my @selection = $d->checklist( text => 'Select one:', list => [ 'tag1', [ 'item1', 0 ], 'tag2', [ 'item2', 1 ], 'tag3', [ 'item3', 1 ] ] ); DESCRIPTION Present the user with a selectable checklist. RETURNS an ARRAY of the chosen tags if the response is OK and FALSE(0) for anything else. radiolist( ) EXAMPLE my $selection = $d->radiolist( text => 'Select one:', list => [ 'tag1', [ 'item1', 0 ], 'tag2', [ 'item2', 1 ], 'tag3', [ 'item3', 0 ] ] ); DESCRIPTION Present the user with a selectable radiolist. RETURNS a SCALAR of the chosen tag if the response is OK and FALSE(0) for anything else. fselect( ) EXAMPLE my $text = $d->fselect( path => '/path/to/a/file/or/directory' ); DESCRIPTION Present the user with a file selection widget preset with the given path. RETURNS a SCALAR if the response is OK and FALSE(0) for anything else. dselect( ) EXAMPLE my $text = $d->dselect( path => '/path/to/a/directory' ); DESCRIPTION Present the user with a file selection widget preset with the given path. Unlike fselect() this widget will only return a direc- tory selection. RETURNS a SCALAR if the response is OK and FALSE(0) for anything else. calendar( ) EXAMPLE my $date = $d->calendar( day => 10, month => 10, year => 1977, 'date-format' => '%d/%m/%y' ); DESCRIPTION Present the user with a calendar so that they may select a date. The 'date-format' option follows the same format definition as the date(1) command line program. If the day, month and year options are not provided, the widget defaults to the current date. The default format for the date string is '%d/%m/%y' which breaks down to: "dd/mm/yy". RETURNS a SCALAR if the response is OK and FALSE(0) for anything else. gauge_start( ) EXAMPLE $d->gauge_start( text => 'gauge...', percentage => 1 ); DESCRIPTION Display a meter bar to the user. This get's the widget realized but requires the use of the other gauge_*() methods for function- ality. RETURNS TRUE(1) if the widget loaded fine and FALSE(0) for anything else. gauge_inc( ) EXAMPLE $d->gauge_inc( 1 ); DESCRIPTION Increment the meter by the given amount. RETURNS TRUE(1) if the widget incremented fine and FALSE(0) for anything else. gauge_dec( ) EXAMPLE $d->gauge_dec( 1 ); DESCRIPTION Decrement the meter by the given amount. RETURNS TRUE(1) if the widget incremented fine and FALSE(0) for anything else. gauge_set( ) EXAMPLE $d->gauge_set( 99 ); DESCRIPTION Set the meter bar to the given amount. RETURNS TRUE(1) if the widget set fine and FALSE(0) for anything else. gauge_stop( ) EXAMPLE $d->gauge_stop(); DESCRIPTION End the meter bar widget process. RETURNS TRUE(1) if the widget closed fine and FALSE(0) for anything else. SEE ALSO
PERLDOC UI::Dialog UI::Dialog::GNOME UI::Dialog::Backend UI::Dialog::Backend::Nautilus UI::Dialog::Backend::XOSD MAN FILES zenity(1) BUGS
Please email the author with any bug reports. Include the name of the module in the subject line. AUTHOR
Kevin C. Krinke, <kckrinke@opendoorsoftware.com> COPYRIGHT AND LICENSE
Copyright (C) 2003 Kevin C. Krinke <kckrinke@opendoorsoftware.com> This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version. This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA perl v5.8.8 2004-08-01 UI::Dialog::Backend::Zenity(3pm)
All times are GMT -4. The time now is 01:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy