Sponsored Content
Full Discussion: awk remove first duplicates
Top Forums Shell Programming and Scripting awk remove first duplicates Post 302885760 by sybadm on Tuesday 28th of January 2014 09:28:34 AM
Old 01-28-2014
Quote:
Originally Posted by pamu
Code:
awk 'NR==FNR{A[$3]++;next}{if(A[$3] > 1 && !B[$3]){B[$3]++;next} }1' file file
 
12  NIL ABD LON
12  NIL ABC AMR
13  NIL ABC AMR
11  NIL ABK AMR


Works really well. bit slow.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to remove duplicates without sorting

Hello, I can remove duplicate entries in a file by: sort File1 | uniq > File2 but how can I remove duplicates without sorting the file? I tried cat File1 | uniq > File2 but it doesn't work thanks (4 Replies)
Discussion started by: orahi001
4 Replies

2. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

3. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

4. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

5. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies

6. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

7. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

8. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

awk - Remove duplicates during array build

Greetings Experts, Issue: Within awk script, remove the duplicate occurrences that are space (1 single space character) separated Description: I am processing 2 files using awk and during processing, I am building an array and there are duplicates on this; how can I delete the duplicates... (3 Replies)
Discussion started by: chill3chee
3 Replies
AMR(4)							   BSD Kernel Interfaces Manual 						    AMR(4)

NAME
amr -- MegaRAID SCSI/ATA/SATA RAID driver SYNOPSIS
To compile this driver into the kernel, place the following lines in your kernel configuration file: device pci device scbus device amr Alternatively, to load the driver as a module at boot time, place the following line in loader.conf(5): amr_load="YES" DESCRIPTION
The amr driver provides support for LSI Logic MegaRAID SCSI, ATA and SATA RAID controllers and legacy American Megatrends MegaRAID SCSI RAID controllers, including models relabeled and sold by Dell and Hewlett-Packard. LSI MegaRAID SAS controllers are supported by mfi(4) and will not work with this driver. HARDWARE
Controllers supported by the amr driver include: o MegaRAID SATA 150-4 o MegaRAID SATA 150-6 o MegaRAID SATA 300-4X o MegaRAID SATA 300-8X o MegaRAID SCSI 320-1E o MegaRAID SCSI 320-2E o MegaRAID SCSI 320-4E o MegaRAID SCSI 320-0X o MegaRAID SCSI 320-2X o MegaRAID SCSI 320-4X o MegaRAID SCSI 320-0 o MegaRAID SCSI 320-1 o MegaRAID SCSI 320-2 o MegaRAID SCSI 320-4 o MegaRAID Series 418 o MegaRAID i4 133 RAID o MegaRAID Elite 1500 (Series 467) o MegaRAID Elite 1600 (Series 493) o MegaRAID Elite 1650 (Series 4xx) o MegaRAID Enterprise 1200 (Series 428) o MegaRAID Enterprise 1300 (Series 434) o MegaRAID Enterprise 1400 (Series 438) o MegaRAID Enterprise 1500 (Series 467) o MegaRAID Enterprise 1600 (Series 471) o MegaRAID Express 100 (Series 466WS) o MegaRAID Express 200 (Series 466) o MegaRAID Express 300 (Series 490) o MegaRAID Express 500 (Series 475) o Dell PERC o Dell PERC 2/SC o Dell PERC 2/DC o Dell PERC 3/DCL o Dell PERC 3/QC o Dell PERC 4/DC o Dell PERC 4/IM o Dell PERC 4/SC o Dell PERC 4/Di o Dell PERC 4e/DC o Dell PERC 4e/Di o Dell PERC 4e/Si o Dell PERC 4ei o HP NetRAID-1/Si o HP NetRAID-3/Si (D4943A) o HP Embedded NetRAID o Intel RAID Controller SRCS16 o Intel RAID Controller SRCU42X DIAGNOSTICS
Driver initialisation/shutdown phase amr%d: memory window not available amr%d: I/O window not available The PCI BIOS did not allocate resources necessary for the correct operation of the controller. The driver cannot attach to this controller. amr%d: busmaster bit not set, enabling The PCI BIOS did not enable busmaster DMA, which is required for the correct operation of the controller. The driver has enabled this bit and initialisation will proceed. amr%d: can't allocate register window amr%d: can't allocate interrupt amr%d: can't set up interrupt amr%d: can't allocate parent DMA tag amr%d: can't allocate buffer DMA tag amr%d: can't allocate scatter/gather DMA tag amr%d: can't allocate s/g table amr%d: can't allocate mailbox tag amr%d: can't allocate mailbox memory A resource allocation error occurred while initialising the driver; initialisation has failed and the driver will not attach to this con- troller. amr%d: can't obtain configuration data from controller amr%d: can't obtain product data from controller The driver was unable to obtain vital configuration data from the controller. Initialisation has failed and the driver will not attach to this controller. amr%d: can't establish configuration hook amr%d: can't scan controller for drives The scan for logical drives managed by the controller failed. No drives will be attached. amr%d: device_add_child failed amr%d: bus_generic_attach returned %d Creation of the logical drive instances failed; attachment of one or more logical drives may have been aborted. amr%d: flushing cache... The controller cache is being flushed prior to shutdown or detach. Operational diagnostics amr%d: I/O beyond end of unit (%u,%d > %u) A partitioning error or disk corruption has caused an I/O request beyond the end of the logical drive. This may also occur if FlexRAID Vir- tual Sizing is enabled and an I/O operation is attempted on a portion of the virtual drive beyond the actual capacity available. amr%d: polled command timeout An initialisation command timed out. The initialisation process may fail as a result. amr%d: bad slot %d completed The controller reported completion of a command that the driver did not issue. This may result in data corruption, and suggests a hardware or firmware problem with the system or controller. amr%d: I/O error - %x An I/O error has occurred. SEE ALSO
cd(4), da(4), mfi(4), sa(4), scsi(4) AUTHORS
The amr driver was written by Mike Smith <msmith@FreeBSD.org>. This manual page was written by Mike Smith <msmith@FreeBSD.org> and Jeroen Ruigrok van der Werven <asmodai@FreeBSD.org>. BSD
March 29, 2006 BSD
All times are GMT -4. The time now is 12:46 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy