[uniq + awk?] How to remove duplicate blocks of lines in files? Post: 302557574

Sponsored Content

Top Forums Shell Programming and Scripting [uniq + awk?] How to remove duplicate blocks of lines in files? Post 302557574 by raidzero on Wednesday 21st of September 2011 02:59:46 PM

09-21-2011

Registered User

Quote:

Originally Posted by vgersh99

Based on the OP's previous explanation, one cannot hard-wire the array "names" as they differ.

I have it wrapped in a function to take the array name as an argument, and the function is run as many times as the number of names. it takes several input files, one for string-arrays, one for styles, plurals, dimens, strings, colors, drawables, etc (all the android resources) and produces two final xml files: one for strings and colors with each item being one line, and one called arrays.xml which is what I am working with now. I hope that clears it up.

---------- Post updated at 02:59 PM ---------- Previous update was at 02:47 PM ----------

I figured it out...

here is my final function

Code:

dupArrayDelete() {
	echo "removing duplicate"
	arrayName=$1
	echo $arrayName
	#get arrays and their contents on their own line
	#first awk prints the file ignoring new lines, putting the whole file in one line
	#sed inserts newlines after each closing </style> or </string-array>, etc
	#second awk removes all lines that have the same column 2
	awk '{printf$0}' $2 | sed 's#</'$arrayName'>#&\n#g' | awk '!A[$2]++' >> $3
}

$1 is the array name, $2 is the source file and $3 is the destination Smilie

thanks everyone!

raidzero

View Public Profile for raidzero

Find all posts by raidzero

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Hi, is it possible to remove all duplicate lines from all txt files in a specific folder? This is too hard for me maybe someone could help. lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50 each textfile has lines with text. I want all lines of all textfiles...

2. UNIX for Dummies Questions & Answers

deleteing duplicate lines sing uniq while ignoring a column

I have a data set that has 4 columns, I want to know if I can delete duplicate lines while ignoring one of the columns, for example 10 chr1 ASF 30 15 chr1 ASF 20 5 chr1 ASF 30 6 chr2 EBC 15 4 chr2 EBC 30 ... I want to know if I can delete duplicate lines while ignoring column 1, so the...

3. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Input: hello hello hello hello monkey donkey hello hello drink dance drink Output should be: hello hello monkey donkey drink dance

4. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to...

5. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused:

6. Shell Programming and Scripting

Cant get awk 1liner to remove duplicate lines from Delimited file, get "event not found" error..help

Hi, I am on a Solaris8 machine If someone can help me with adjusting this awk 1 liner (turning it into a real awkscript) to get by this "event not found error" ...or Present Perl solution code that works for Perl5.8 in the csh shell ...that would be great. ****************** ...

7. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines.

8. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22...

9. Shell Programming and Scripting

Remove lines from output in files using awk

I have two large files (~250GB) that I am trying to remove the where GT: 0/0 or 1/1 or 2/2 for both files. I was going to use a bash with the below awk, which I think will find each line but how do I remove that line is that condition is found? Thank you :). Input 20 60055 . A ...

10. Shell Programming and Scripting

How to put the command to remove duplicate lines in my awk script?

I create a CGI in bash/html. My awk script looks like : echo "<table>" for fn in /var/www/cgi-bin/LPAR_MAP/*; do echo "<td>" echo "<PRE>" awk -F',|;' -v test="$test" ' NR==1 { split(FILENAME ,a,""); } $0 ~ test { if(!header++){ ...

LEARN ABOUT REDHAT

uniq

UNIQ(1) 								FSF								   UNIQ(1)

NAME

       uniq - remove duplicate lines from a sorted file

SYNOPSIS

       uniq [OPTION]... [INPUT [OUTPUT]]

DESCRIPTION

       Discard all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output).

       Mandatory arguments to long options are mandatory for short options too.

       -c, --count
	      prefix lines by the number of occurrences

       -d, --repeated
	      only print duplicate lines

       -D, --all-repeated[=delimit-method] print all duplicate lines
	      delimit-method={none(default),prepend,separate} Delimiting is done with blank lines.

       -f, --skip-fields=N
	      avoid comparing the first N fields

       -i, --ignore-case
	      ignore differences in case when comparing

       -s, --skip-chars=N
	      avoid comparing the first N characters

       -u, --unique
	      only print unique lines

       -w, --check-chars=N
	      compare no more than N characters in lines

       --help display this help and exit

       --version
	      output version information and exit

       A field is a run of whitespace, then non-whitespace characters.	Fields are skipped before chars.

AUTHOR

       Written by Richard Stallman and David MacKenzie.

REPORTING BUGS

       Report bugs to <bug-coreutils@gnu.org>.

COPYRIGHT

       Copyright (C) 2002 Free Software Foundation, Inc.
       This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICU-
       LAR PURPOSE.

SEE ALSO

       The full documentation for uniq is maintained as a Texinfo manual.  If the info and uniq programs are properly installed at your site,  the
       command

	      info uniq

       should give you access to the complete manual.

uniq (coreutils) 4.5.3						   February 2003							   UNIQ(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Discussion started by: lowmaster

2. UNIX for Dummies Questions & Answers

deleteing duplicate lines sing uniq while ignoring a column

Discussion started by: japaneseguitars

3. Shell Programming and Scripting

Command to remove duplicate lines with perl,sed,awk

Discussion started by: cola

4. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Discussion started by: arvindng