Sponsored Content
Top Forums Shell Programming and Scripting How to Remove duplicate value from file? Post 302770737 by Jotne on Monday 18th of February 2013 07:09:15 AM
Old 02-18-2013
I think this is more correct
Code:
awk '{if ($8=="XXX") {arr[$7]=$0} else if (!arr[$7]) {arr[$7]=$0}} END {for (i in arr) print arr[i]}' infile
BI      M       IT056868        UNICREDI        VIA     LIBHIA  UNCRIT2B        XXX     UNCR
HH      M       MNOOOO          98989089        IIC     UMNKSS  MOHAN844        XXX     KLKL
BI      M       US001165        NEUBERGE        LLC     NEWYRK  NEUBUS33        253     NEUB
HH      M       IND90909        SBILIFES        HNI     NANANN  GGGGGGGG        142     UICC

get every single instance of $7, but keep the one line that have XXX in $8

---------- Post updated at 13:09 ---------- Previous update was at 12:49 ----------

Some updated.
Code:
awk '$8=="XXX" {arr[$7]=$0} !arr[$7] {arr[$7]=$0} END {for (i in arr) print arr[i]}' infile

 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

Sort and Remove Duplicate on file

How do we sort and remove duplicate on column 1,2 retaining the record with maximum date (in feild 3) for the file with following format. aaa|1234|2010-12-31 aaa|1234|2010-11-10 bbb|345|2011-01-01 ccc|346|2011-02-01 bbb|345|2011-03-10 aaa|1234|2010-01-01 Required Output ... (5 Replies)
Discussion started by: mabarif16
5 Replies

4. Shell Programming and Scripting

Formatting a file - Remove Duplicate

Hi I have a file in the following format. Basically the file contains tablename and their aliases: TABLE1 TABLE1 A TABLE2 TABLE2 B TABLE3 TABLE4 TABLE4 C TABLE4 Upon formatting an sql statement I am getting such output. Problem: Whenever a tablename appears with alias, it has... (5 Replies)
Discussion started by: freakygs
5 Replies

5. Shell Programming and Scripting

remove of duplicate line from a file

I have a file a.txt having content like deepak ram sham deepram sita kumar I Want to delete the first line containing "deep" ... I tried using... grep -i 'deep' a.txt It gives me 2 rows...I want to delete the first one.. + need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies

6. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

7. Shell Programming and Scripting

Remove the duplicate content in a file

Here is the contents of test.txt Dependencies Resolved Changes in packages about to be updated: ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64, - Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code) Dependencies Resolved Changes in packages about to be updated: ... (5 Replies)
Discussion started by: ashokvpp
5 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

9. Shell Programming and Scripting

Remove duplicate values in a column(not in the file)

Hi Gurus, I have a file(weblog) as below abc|xyz|123|agentcode=sample code abcdeeess,agentcode=sample code abcdeeess,agentcode=sample code abcdeeess|agentadd=abcd stereet 23343,agentadd=abcd stereet 23343 sss|wwq|999|agentcode=sample1 code wqwdeeess,gentcode=sample1 code... (4 Replies)
Discussion started by: ratheeshjulk
4 Replies
ARRAY_MAP(3)								 1							      ARRAY_MAP(3)

array_map - Applies the callback to the elements of the given arrays

SYNOPSIS
array array_map (callable $callback, array $array1, [array $...]) DESCRIPTION
array_map(3) returns an array containing all the elements of $array1 after applying the $callback function to each one. The number of parameters that the $callback function accepts should match the number of arrays passed to the array_map(3) PARAMETERS
o $callback - Callback function to run for each element in each array. o $array1 - An array to run through the $callback function. o $... - Variable list of array arguments to run through the $callback function. RETURN VALUES
Returns an array containing all the elements of $array1 after applying the $callback function to each one. EXAMPLES
Example #1 array_map(3) example <?php function cube($n) { return($n * $n * $n); } $a = array(1, 2, 3, 4, 5); $b = array_map("cube", $a); print_r($b); ?> This makes $b have: Array ( [0] => 1 [1] => 8 [2] => 27 [3] => 64 [4] => 125 ) Example #2 array_map(3) using a lambda function (as of PHP 5.3.0) <?php $func = function($value) { return $value * 2; }; print_r(array_map($func, range(1, 5))); ?> Array ( [0] => 2 [1] => 4 [2] => 6 [3] => 8 [4] => 10 ) Example #3 array_map(3) - using more arrays <?php function show_Spanish($n, $m) { return("The number $n is called $m in Spanish"); } function map_Spanish($n, $m) { return(array($n => $m)); } $a = array(1, 2, 3, 4, 5); $b = array("uno", "dos", "tres", "cuatro", "cinco"); $c = array_map("show_Spanish", $a, $b); print_r($c); $d = array_map("map_Spanish", $a , $b); print_r($d); ?> The above example will output: // printout of $c Array ( [0] => The number 1 is called uno in Spanish [1] => The number 2 is called dos in Spanish [2] => The number 3 is called tres in Spanish [3] => The number 4 is called cuatro in Spanish [4] => The number 5 is called cinco in Spanish ) // printout of $d Array ( [0] => Array ( [1] => uno ) [1] => Array ( [2] => dos ) [2] => Array ( [3] => tres ) [3] => Array ( [4] => cuatro ) [4] => Array ( [5] => cinco ) ) Usually when using two or more arrays, they should be of equal length because the callback function is applied in parallel to the corre- sponding elements. If the arrays are of unequal length, shorter ones will be extended with empty elements to match the length of the long- est. An interesting use of this function is to construct an array of arrays, which can be easily performed by using NULL as the name of the callback function Example #4 Creating an array of arrays <?php $a = array(1, 2, 3, 4, 5); $b = array("one", "two", "three", "four", "five"); $c = array("uno", "dos", "tres", "cuatro", "cinco"); $d = array_map(null, $a, $b, $c); print_r($d); ?> The above example will output: Array ( [0] => Array ( [0] => 1 [1] => one [2] => uno ) [1] => Array ( [0] => 2 [1] => two [2] => dos ) [2] => Array ( [0] => 3 [1] => three [2] => tres ) [3] => Array ( [0] => 4 [1] => four [2] => cuatro ) [4] => Array ( [0] => 5 [1] => five [2] => cinco ) ) If the array argument contains string keys then the returned array will contain string keys if and only if exactly one array is passed. If more than one argument is passed then the returned array always has integer keys. Example #5 array_map(3) - with string keys <?php $arr = array("stringkey" => "value"); function cb1($a) { return array ($a); } function cb2($a, $b) { return array ($a, $b); } var_dump(array_map("cb1", $arr)); var_dump(array_map("cb2", $arr, $arr)); var_dump(array_map(null, $arr)); var_dump(array_map(null, $arr, $arr)); ?> The above example will output: array(1) { ["stringkey"]=> array(1) { [0]=> string(5) "value" } } array(1) { [0]=> array(2) { [0]=> string(5) "value" [1]=> string(5) "value" } } array(1) { ["stringkey"]=> string(5) "value" } array(1) { [0]=> array(2) { [0]=> string(5) "value" [1]=> string(5) "value" } } SEE ALSO
array_filter(3), array_reduce(3), array_walk(3), information about the callback type. PHP Documentation Group ARRAY_MAP(3)
All times are GMT -4. The time now is 03:29 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy