Sponsored Content
Top Forums UNIX for Dummies Questions & Answers remove duplicates based on a field and criteria Post 302611043 by Scrutinizer on Thursday 22nd of March 2012 06:31:25 AM
Old 03-22-2012
Hi, for every line it creates an (associative) array element with field 4 as the index (if it does not exist yet) without a value (or 0 is you will). The exclamation mark negates that value so the outcome is 1 (true). The value of 1 in awk means perform the default action which is {print $0} so the entire line gets printed.

Afterwards the ++ comes into action and 1 is added to the array value, which now becomes 1. So that next time a line with the same value in $4 is encountered the value returned by the array is 1 which is then negated to 0 by the exclamation mark, so nothing will get printed.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove lines based on score criteria

Hi guys, Please guide for Solution. PART-I INPUT FILE (has 2 columns ID and score) TC5584_1 93.9 DV161411_2 79.5 BP132435_5 46.8 EB682112_1 34.7 BP132435_4 29.5 TC13860_2 10.1 OUTPUT FILE (It shudn't contain the line ' BP132435_4 29.5 ' as BP132435 is repeated... (2 Replies)
Discussion started by: smriti_shridhar
2 Replies

2. Shell Programming and Scripting

split large file based on field criteria

I have a file containing date/time sorted data of the form ... 2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1 2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1 2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0 2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1... (6 Replies)
Discussion started by: asriva
6 Replies

3. Shell Programming and Scripting

need Shell script for Sort BASED ON FIRST FIELD and PRINT THE WHOLE FILE WITHOUT DUPLICATES

Can some one provide me a shell script. I have file with many columns and many rows. need to sort the first column and then remove the duplicates records if exists.. finally print the full data with first coulm as unique. Sort BASED ON FIRST FIELD and remove the duplicates if exists... (2 Replies)
Discussion started by: tuffEnuff
2 Replies

4. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

5. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

6. Shell Programming and Scripting

remove duplicates based on single column

Hello, I am new to shell scripting. I have a huge file with multiple columns for example: I have 5 columns below. HWUSI-EAS000_29:1:105 + chr5 76654650 AATTGGAA HHHHG HWUSI-EAS000_29:1:106 + chr5 76654650 AATTGGAA B@HYL HWUSI-EAS000_29:1:108 + ... (4 Replies)
Discussion started by: Diya123
4 Replies

7. Shell Programming and Scripting

Remove the partial duplicates by checking the length of a field

Hi Folks - I'm quite new to awk and didn't come across such issues before. The problem statement is that, I've a file with duplicate records in 3rd and 4th fields. The sample is as below: aaaaaa|a12|45|56 abbbbaaa|a12|45|56 bbaabb|b1|51|45 bbbbbabbb|b2|51|45 aaabbbaaaa|a11|45|56 ... (3 Replies)
Discussion started by: asyed
3 Replies

8. Shell Programming and Scripting

CSV with commas in field values, remove duplicates, cut columns

Hi Description of input file I have: ------------------------- 1) CSV with double quotes for string fields. 2) Some string fields have Comma as part of field value. 3) Have Duplicate lines 4) Have 200 columns/fields 5) File size is more than 10GB Description of output file I need:... (4 Replies)
Discussion started by: krishnix
4 Replies

9. Shell Programming and Scripting

Remove duplicates based on a field's value

Hi All, I have a text file with three columns. I would like a simple script that removes lines in which column 1 has duplicate entries, but use the largest value in column 3 to decide which one to keep. For example: Input file: 12345a rerere.rerere len=23 11111c fsdfdf.dfsdfdsf len=33 ... (3 Replies)
Discussion started by: anniecarv
3 Replies

10. Shell Programming and Scripting

Trying to remove duplicates based on field and row

I am trying to see if I can use awk to remove duplicates from a file. This is the file: -==> Listvol <== deleting /vol/eng_rmd_0941 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_1006 deleting /vol/eng_rmd_1012 rearrange /vol/eng_rmd_0943 ... (6 Replies)
Discussion started by: newbie2010
6 Replies
ltrace(1)						      General Commands Manual							 ltrace(1)

NAME
ltrace - A library call tracer SYNOPSIS
ltrace [-CfhiLrStttV] [-a column] [-A maxelts] [-d level] [-e expr] [-l filename] [-n nr] [-o filename] [-p pid] ... [-s strsize] [-u user- name] [-X extern] [-x extern] ... [--align=column] [--debug=level] [--demangle] [--help] [--indent=nr] [--library=filename] [--output=file- name] [--version] [command [arg ...]] DESCRIPTION
ltrace is a program that simply runs the specified command until it exits. It intercepts and records the dynamic library calls which are called by the executed process and the signals which are received by that process. It can also intercept and print the system calls exe- cuted by the program. Its use is very similar to strace(1). OPTIONS
-a, --align column Align return values in a specific column (default column is 5/8 of screen width). -A maxelts Maximum number of array elements to print before suppressing the rest with an ellipsis ("...") -c Count time and calls for each library call and report a summary on program exit. -C, --demangle Decode (demangle) low-level symbol names into user-level names. Besides removing any initial underscore prefix used by the system, this makes C++ function names readable. -d, --debug level Show debugging output of ltrace itself. level must be a sum of some of the following numbers: 0x10 DEBUG_EVENT. Shows every event received by a traced program 0x20 DEBUG_PROCESS. Shows every action ltrace carries upon a traced program -e expr A qualifying expression which modifies which events to trace. The format of the expression is: [!]value1[,value2]... where the values are the functions to trace. Using an exclamation mark negates the set of values. For example -e printf means to trace only the printf library call. By contrast, -e !printf means to trace every library call except printf. Note that some shells use the exclamation point for history expansion; even inside quoted arguments. If so, you must escape the exclamation point with a backslash. -f Trace child processes as they are created by currently traced processes as a result of the fork(2) or clone(2) system calls. The new process is attached immediately. -F Load an alternate config file. Normally, /etc/ltrace.conf and ~/.ltrace.conf will be read (the latter only if it exists). Use this option to load the given file or files instead of those two default files. -h, --help Show a summary of the options to ltrace and exit. -i Print the instruction pointer at the time of the library call. -l, --library filename Display only the symbols included in the library filename. Up to 30 library names can be specified with several instances of this option. -L DON'T display library calls (use it with the -S option). -n, --indent nr Indent trace output by nr number of spaces for each new nested call. Using this option makes the program flow visualization easy to follow. -o, --output filename Write the trace output to the file filename rather than to stderr. -p pid Attach to the process with the process ID pid and begin tracing. -r Print a relative timestamp with each line of the trace. This records the time difference between the beginning of successive lines. -s strsize Specify the maximum string size to print (the default is 32). -S Display system calls as well as library calls -t Prefix each line of the trace with the time of day. -tt If given twice, the time printed will include the microseconds. -ttt If given thrice, the time printed will include the microseconds and the leading portion will be printed as the number of seconds since the epoch. -T Show the time spent inside each call. This records the time difference between the beginning and the end of each call. -u username Run command with the userid, groupid and supplementary groups of username. This option is only useful when running as root and enables the correct execution of setuid and/or setgid binaries. -X extern Some architectures need to know where to set a breakpoint that will be hit after the dynamic linker has run. If this flag is used, then the breakpoint is set at extern, which must be an external function. By default, '_start' is used. NOTE: this flag is only available on the architectures that need it. -x extern Trace the external function extern. This option may be repeated. -V, --version Show the version number of ltrace and exit. BUGS
It has most of the bugs stated in strace(1). Manual page and documentation are not very up-to-date. Option -f sometimes fails to trace some children. It only works on Linux and in a small subset of architectures. Only ELF32 binaries are supported. Calls to dlopen()ed libraries will not be traced. If you would like to report a bug, send a message to the mailing list (ltrace-devel@lists.alioth.debian.org), or use the reportbug(1) pro- gram if you are under the Debian GNU/Linux distribution. FILES
/etc/ltrace.conf System configuration file ~/.ltrace.conf Personal config file, overrides /etc/ltrace.conf AUTHOR
Juan Cespedes <cespedes@debian.org> SEE ALSO
strace(1), ptrace(2) ltrace(1)
All times are GMT -4. The time now is 04:38 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy