Sponsored Content
Full Discussion: Removing all the duplicates
Homework and Emergencies Emergency UNIX and Linux Support Removing all the duplicates Post 302547556 by bartus11 on Tuesday 16th of August 2011 03:57:20 AM
Old 08-16-2011
Try:
Code:
awk -F"|" '{a[$1]++;b[$1]=b[$1]?b[$1]"\n"$0:$0}END{for(i in a){if(a[i]==1){print b[i]>"file1"}else{print b[i]>"file2"}}}' input

It will create two files: file1 and file2.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates

Hi, I've been trying to removed duplicates lines with similar columns in a fixed width file and it's not working. I've search the forum but nothing comes close. I have a sample file: 27147140631203RA CCD * 27147140631203RA PPN * 37147140631207RD AAA 47147140631203RD JNA... (12 Replies)
Discussion started by: giannicello
12 Replies

2. UNIX for Dummies Questions & Answers

removing duplicates and sort -k

Hello experts, I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter sort -u -k 2,2 File.csv > Output.csv File.csv File Name|Document Name|Document Title|Organization Word Doc 1.doc|Word Document|Sample... (3 Replies)
Discussion started by: orahi001
3 Replies

3. Shell Programming and Scripting

removing duplicates

Hi I have a file that are a list of people & their credentials i recieve frequently The issue is that whne I catnet this list that duplicat entries exists & are NOT CONSECUTIVE (i.e. uniq -1 may not weork here ) I'm trying to write a scrip that will remove duplicate entries the script can... (5 Replies)
Discussion started by: stevie_velvet
5 Replies

4. Shell Programming and Scripting

Removing duplicates

Hi, I have a file in the below format., test test (10) to to (25) see see (45) and i need the output in the format of test 10 to 25 see 45 Some one help me? (6 Replies)
Discussion started by: imdadulla
6 Replies

5. UNIX for Advanced & Expert Users

removing duplicates.

Hi All In unix ,we have a file ,there we have to remove the duplicates by using one specific column. Can any body tell me the command. ex: file1 id,name 1,ww 2,qwq 2,asas 3,asa 4,asas 4,asas o/p: 1,ww 2,qwq 3,asa (7 Replies)
Discussion started by: raju4u
7 Replies

6. Shell Programming and Scripting

Removing duplicates

I have a test file with the following 2 columns: Col 1 | Col 2 T1 | 1 <= remove T5 | 1 T4 | 2 T1 | 3 T3 | 3 T4 | 1 <= remove T1 | 2 <= remove T3 ... (7 Replies)
Discussion started by: gctex
7 Replies

7. Shell Programming and Scripting

Help in removing duplicates

I have an input file abc.txt with info like: abcd rateuse inklite robet rateuse abcd I need to remove duplicates from the file (eg: abcd,rateuse) from the file and need to place the contents in same file abc.txt if needed can be placed in another file. can anyone help me in this :( (4 Replies)
Discussion started by: rkrish
4 Replies

8. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

9. Shell Programming and Scripting

Removing duplicates except the last occurrence

Hi All, i have a file like below, @DB_FCTS\src\Data\Scripts\Delete_CU_OM_BIL_PRT_STMT_TYP.sql @DB_FCTS\src\Data\Scripts\Delete_CDP_BILL_LBL_MSG.sql @DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql @DB_FCTS\src\Data\Scripts\Insert_CU_OM_LBL_MSG.sql... (11 Replies)
Discussion started by: mechvijays
11 Replies

10. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies
h5diff(1)						      General Commands Manual							 h5diff(1)

NAME
h5diff - Compares two HDF5 files and reports the differences. SYNOPSIS
h5diff file1 file2 [OPTIONS] [object1 [object2 ] ] DESCRIPTION
h5diff is a command line tool that compares two HDF5 files, file1 and file2, and reports the differences between them. Optionally, h5diff will compare two objects within these files. If only one object, object1, is specified, h5diff will compare object1 in file1 with object1 in file2. In two objects, object1 and object2, are specified, h5diff will compare object1 in file1 with object2 in file2. These objects must be HDF5 datasets. object1 and object2 must be expressed as absolute paths from the respective file's root group. Additional information, with several sample cases, can be found in the document H5diff Examples. OPTIONS
file1 file2 The HDF5 files to be compared. -h Print all differences. -r Print only the names of objects that differ; do not print the differences. These objects may be HDF5 datasets, groups, or named datatypes. -n count Print difference up to count differences, then stop. count must be a positive integer. -d delta Print only differences that are greater than the limit delta. delta must be a positive number. The comparison criterion is whether the absolute value of the difference of two corresponding values is greater than delta (e.g., |a-b| > delta, where a is a value in file1 and b is a value in file2). -p relative Print only differences that are greater than a relative error. relative must be a positive number. The comparison criterion is whether the absolute value of the difference 1 and the ratio of two corresponding values is greater than relative (e.g., |1-(b/a)| > relative where a is a value in file1 and b is a value in file2). object1 object2 Specific object(s) within the files to be compared. EXAMPLES
The following h5diff call compares the object /a/b in file1 with the object /a/c in file2: h5diff file1 file2 /a/b /a/c This h5diff call compares the object /a/b in file1 with the same object in file2: h5diff file1 file2 /a/b And this h5diff call compares all objects in both files: h5diff file1 file2 SEE ALSO
h5dump(1), h5ls(1), h5repart(1), h5import(1), gif2h5(1), h52gif(1), h5perf(1) h5diff(1)
All times are GMT -4. The time now is 01:06 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy