Sponsored Content
Top Forums Shell Programming and Scripting Filter/remove duplicate .dat file with certain criteria Post 302504549 by mukeshguliao on Tuesday 15th of March 2011 12:22:45 AM
Old 03-15-2011
hi i tried these two options in csh and bash

Code:
 
cat UnixEg.dat | awk -F',' '{key=$1$2$3;if(key in a) next; else a[$1$2$3]=$0; print a[$1$2$3]}'
 
awk -F',' '{key=$1$2$3;if(key in a) next; else a[$1$2$3]=$0; print a[$1$2$3]}' UnixEg.dat

in both I am getting an

Code:
awk: syntax error near line 1
awk: illegal statement near line 1
awk: illegal statement near line 1

:banghead

---------- Post updated at 11:22 PM ---------- Previous update was at 09:47 PM ----------

found the solution

https://www.unix.com/shell-programmin...laination.html
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

2. Shell Programming and Scripting

How to remove duplicates from the .dat file

All, I have a file 1181CUSTOMER-L061411_003500.dat.Z having duplicate records in it. bash-2.05$ zcat 1181CUSTOMER-L061411_003500.dat.Z|grep "90876251S" 90876251S|ABG, AN ADAYANA COMPANY|3550 DEPAUW BLVD|||US|IN|INDIANAPOLIS||DAL|46268||||||GEN|||||||USD|||ABG, AN ADAYANA... (3 Replies)
Discussion started by: Oracle_User
3 Replies

3. Shell Programming and Scripting

Remove interspersed headers in .dat file with AWK

Heya there, A small selection of my data is shown below. DATE TIME FRAC_DAYS_SINCE_JAN1 2011-06-25 08:03:20.000 175.33564815 2011-06-25 08:03:25.000 175.33570602 2011-06-25 ... (4 Replies)
Discussion started by: gd9629
4 Replies

4. Shell Programming and Scripting

Filter or remove duplicate block of text without distinguishing marks or fields

Hello, Although I have found similar questions, I could not find advice that could help with our problem. The issue: We have several hundreds text files containing repeated blocks of text (I guess back at the time they were prepared like that to optmize printing). The block of texts... (13 Replies)
Discussion started by: samask
13 Replies

5. Shell Programming and Scripting

Remove <CR><LF> from the dat file in unix

Hi, The source system has created the file in the dat format and put into the linux directory as mentioned below. I want to do foloowing things. a) Delete the Line started with <CR><LF> in the record b)Also line ...........................................................<CR><LF> ... (1 Reply)
Discussion started by: mr_harish80
1 Replies

6. Shell Programming and Scripting

Help with filter result that fulfill criteria

Input file: ##fileformat=tab ##reference=file:input.txt #Line Position Score Input_185827_2127 1071 67 Input_18213_21 1021 100 Input_9012_214 200 150 Input_935_217 124 70 Output file: ##fileformat=tab ##reference=file:input.txt #Line Position Score Input_18213_21 1021... (2 Replies)
Discussion started by: perl_beginner
2 Replies

7. Shell Programming and Scripting

Filter datablocks meeting criteria

Hello, I am trying to extract valid data blocks from invalid ones. In the input the data blocks are separated by one or more blank rows. The criteria are 1) second column value must be 30 or more for the row to be valid and considered for calculation and output. 2) the sum of all valid... (2 Replies)
Discussion started by: sheetalk
2 Replies

8. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

9. Shell Programming and Scripting

Awk/sed/cut to filter out records from a file based on criteria

I have two files and would need to filter out records based on certain criteria, these column are of variable lengths, but the lengths are uniform throughout all the records of the file. I have shown a sample of three records below. Line 1-9 is the item number "0227546_1" in the case of the first... (15 Replies)
Discussion started by: MIA651
15 Replies

10. Shell Programming and Scripting

Filter duplicate records from csv file with condition on one column

I have csv file with 30, 40 columns Pasting just three column for problem description I want to filter record if column 1 matches CN or DN then, check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345 and if column 2 contains 6789, 6789... (5 Replies)
Discussion started by: as7951
5 Replies
DPKG-WWW(1)						      General Commands Manual						       DPKG-WWW(1)

NAME
dpkg-www - program to remotely open a WWW Debian package browser SYNOPSIS
dpkg-www [-s|--stdout] [-h|--host host] [query] DESCRIPTION
dpkg-www is used to remotely control a WEB browser and open a dpkg URL on the local host from an interactive shell or script. If the command is invoked while running under X-window the script will try to find an installed X browser to open the specified URL. If not running under X it will try to use a text browser instead. If Netscape is found and a Netscape instance is already running it will be asked to open the dpkg URL on localhost with the optional query supplied on the command line. If there is no browser running, it will start automatically a new one. The functionality provided by this program is identical to running a web browser with the -remote openURL(http://localhost/cgi-bin/dpkg) argument or opening the same URL from within the browser. dpkg-www-installer is an helper application which can configured in the WWW browser for web installation. It should never be invoked directly by the user. OPTIONS
-s, --stdout Redirect output to stdout. Requires one of the text browsers (lynx, lynx-ssl or links) installed. -h, --host host Send the query to a remote host, where dpkg-www must be installed. query Specifies an optional package name, an absolute pathname or a query argument which is passed to the dpkg cgi-bin. See dpkg-www(8) for more information about the use of the cgi. FILES
/etc/dpkg-www.conf ~/.dpkg-www Configuration files for dpkg-www. It is not necessary for these files to exist, there are sensible defaults for everything, but you can specify your preferred www browser with the DPKG_WWW_BROWSER variable, for example: DPKG_WWW_BROWSER=mozilla EXAMPLES
dpkg-www This would open a dpkg URL on localhost listing all the installed packages. dpkg-www bash This would open a dpkg URL asking info on the bash package. dpkg-www -h pisolo bash This would open a dpkg URL asking info on the bash package on host pisolo. dpkg-www 'dpkg*' This would open a dpkg URL listing all packages matching dpkg*. dpkg-www /bin/bash This would open a dpkg URL asking info on the package(s) owning the file /bin/bash . dpkg-www depends=svgalib This would open a dpkg URL listing all packages depending on svgalib. dpkg-www --stdout depends=awk | grep ^ii This would list on stdout all packages depending on awk and grep all lines of installed packages. SEE ALSO
dpkg(8), dpkg-www(8) AUTHOR
Massimo Dal Zotto <dz@debian.org>. Bugs should be reported via the normal Debian bug reporting system. LICENCE
dpkg-www is licensed under the GNU General Public License version 2. September 1, 2004 DPKG-WWW(1)
All times are GMT -4. The time now is 05:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy