Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Filter/remove duplicate .dat file with certain criteria
Top Forums Shell Programming and Scripting Filter/remove duplicate .dat file with certain criteria Post 302504549 by mukeshguliao on Tuesday 15th of March 2011 12:22:45 AM
Old 03-15-2011
hi i tried these two options in csh and bash

Code:
 
cat UnixEg.dat | awk -F',' '{key=$1$2$3;if(key in a) next; else a[$1$2$3]=$0; print a[$1$2$3]}'
 
awk -F',' '{key=$1$2$3;if(key in a) next; else a[$1$2$3]=$0; print a[$1$2$3]}' UnixEg.dat

in both I am getting an

Code:
awk: syntax error near line 1
awk: illegal statement near line 1
awk: illegal statement near line 1

:banghead

---------- Post updated at 11:22 PM ---------- Previous update was at 09:47 PM ----------

found the solution

https://www.unix.com/shell-programmin...laination.html
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

2. Shell Programming and Scripting

How to remove duplicates from the .dat file

All, I have a file 1181CUSTOMER-L061411_003500.dat.Z having duplicate records in it. bash-2.05$ zcat 1181CUSTOMER-L061411_003500.dat.Z|grep "90876251S" 90876251S|ABG, AN ADAYANA COMPANY|3550 DEPAUW BLVD|||US|IN|INDIANAPOLIS||DAL|46268||||||GEN|||||||USD|||ABG, AN ADAYANA... (3 Replies)
Discussion started by: Oracle_User
3 Replies

3. Shell Programming and Scripting

Remove interspersed headers in .dat file with AWK

Heya there, A small selection of my data is shown below. DATE TIME FRAC_DAYS_SINCE_JAN1 2011-06-25 08:03:20.000 175.33564815 2011-06-25 08:03:25.000 175.33570602 2011-06-25 ... (4 Replies)
Discussion started by: gd9629
4 Replies

4. Shell Programming and Scripting

Filter or remove duplicate block of text without distinguishing marks or fields

Hello, Although I have found similar questions, I could not find advice that could help with our problem. The issue: We have several hundreds text files containing repeated blocks of text (I guess back at the time they were prepared like that to optmize printing). The block of texts... (13 Replies)
Discussion started by: samask
13 Replies

5. Shell Programming and Scripting

Remove <CR><LF> from the dat file in unix

Hi, The source system has created the file in the dat format and put into the linux directory as mentioned below. I want to do foloowing things. a) Delete the Line started with <CR><LF> in the record b)Also line ...........................................................<CR><LF> ... (1 Reply)
Discussion started by: mr_harish80
1 Replies

6. Shell Programming and Scripting

Help with filter result that fulfill criteria

Input file: ##fileformat=tab ##reference=file:input.txt #Line Position Score Input_185827_2127 1071 67 Input_18213_21 1021 100 Input_9012_214 200 150 Input_935_217 124 70 Output file: ##fileformat=tab ##reference=file:input.txt #Line Position Score Input_18213_21 1021... (2 Replies)
Discussion started by: perl_beginner
2 Replies

7. Shell Programming and Scripting

Filter datablocks meeting criteria

Hello, I am trying to extract valid data blocks from invalid ones. In the input the data blocks are separated by one or more blank rows. The criteria are 1) second column value must be 30 or more for the row to be valid and considered for calculation and output. 2) the sum of all valid... (2 Replies)
Discussion started by: sheetalk
2 Replies

8. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

9. Shell Programming and Scripting

Awk/sed/cut to filter out records from a file based on criteria

I have two files and would need to filter out records based on certain criteria, these column are of variable lengths, but the lengths are uniform throughout all the records of the file. I have shown a sample of three records below. Line 1-9 is the item number "0227546_1" in the case of the first... (15 Replies)
Discussion started by: MIA651
15 Replies

10. Shell Programming and Scripting

Filter duplicate records from csv file with condition on one column

I have csv file with 30, 40 columns Pasting just three column for problem description I want to filter record if column 1 matches CN or DN then, check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345 and if column 2 contains 6789, 6789... (5 Replies)
Discussion started by: as7951
5 Replies

LEARN ABOUT DEBIAN

arp2ethers

ARP2ETHERS(8)						      System Manager's Manual						     ARP2ETHERS(8)

NAME

       arp2ethers - convert arpwatch address database to ethers file format

SYNOPSIS

       arp2ethers [ arp.dat file ]

DESCRIPTION

       arp2ethers  converts  the file /var/lib/arpwatch/arp.dat (or the file specified on the command line) into ethers(5) format on stdout.  Usu-
       ally arp.dat is an ethernet/ip database file generated by arpwatch(8).  The arpwatch daemon in Debian will create different arp.dat depend-
       ing on its configuration. All of them will be available at /var/lib/arpwatch/.

FILES

       /var/lib/arpwatch - default directory for arp.dat
       arp.dat - ethernet/ip address database

SEE ALSO

       arpwatch(8), ethers(5), rarp(8), arp(8),

BUGS

       Please send bug reports to arpwatch@ee.lbl.gov.

AUTHORS

       Original  version  by  Craig Leres of the Lawrence Berkeley National Laboratory Network Research Group, University of California, Berkeley,
       CA.

       Modified for the Debian Project by Peter Kelemen, with additions from Erik Warmelink.

       The current version is available via anonymous ftp:

	      ftp://ftp.ee.lbl.gov/arpwatch.tar.gz

       This manual page was contributed by Hugo Graumann.

																     ARP2ETHERS(8)
All times are GMT -4. The time now is 03:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy