Sponsored Content
Top Forums Shell Programming and Scripting Finding duplicates from positioned substring across lines Post 302271066 by jim mcnamara on Tuesday 23rd of December 2008 06:36:44 PM
Old 12-23-2008
Code:
awk '{ arr[substr($0,50,4))]++ } 
      END { for (i in arr) { if (arr[i]>1) {print arr[i], i}}}' inputfile

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

finding duplicates with perl

I have a huge file (over 30mb) that I am processing through with perl. I am pulling out a list of filenames and placing it in an array called @reports. I am fine up till here. What I then want to do is go through the array and find any duplicates. If there is a duplicate, output it to the screen.... (3 Replies)
Discussion started by: dangral
3 Replies

2. Shell Programming and Scripting

finding the last substring...

hii, i want to know the shell command for finding the last occurance of a substring in string.. i can use grep command or sed to find out the occurance of a substring in a string but how do i find out the last occurance.shud i use grep amd and cut the string everytime and store it in a new... (7 Replies)
Discussion started by: cutelucks
7 Replies

3. Shell Programming and Scripting

finding duplicates in columns and removing lines

I am trying to figure out how to scan a file like so: 1 ralphs office","555-555-5555","ralph@mail.com","www.ralph.com 2 margies office","555-555-5555","ralph@mail.com","www.ralph.com 3 kims office","555-555-5555","kims@mail.com","www.ralph.com 4 tims... (17 Replies)
Discussion started by: totus
17 Replies

4. Shell Programming and Scripting

Finding longest common substring among filenames

I will be performing a task on several directories, each containing a large number of files (2500+) that follow a regular naming convention: YYYY_MM_DD_XX.foo_bar.A.B.some_different_stuff.EXT What I would like to do is automatically discover the part of the filenames that are common to all... (1 Reply)
Discussion started by: cmcnorgan
1 Replies

5. Shell Programming and Scripting

How to delete lines in a file that have duplicates or derive the lines that aper once

Input: a b b c d d I need: a c I know how to get this (the lines that have duplicates) : b d sort file | uniq -d But i need opossite of this. I have searched the forum and other places as well, but have found solution for everything except this variant of the problem. (3 Replies)
Discussion started by: necroman08
3 Replies

6. Shell Programming and Scripting

Help finding non duplicates

I am currently creating a script to find filenames that are listed once in an input file (find non duplicates). I then want to report those single files in another file. Here is the function that I have so far: function dups_filenames { file2="" file1="" file="" dn="" ch="" pn="" ... (6 Replies)
Discussion started by: chipblah84
6 Replies

7. Shell Programming and Scripting

finding duplicates in csv based on key columns

Hi team, I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record. can one help me on finding the duplicates, Thanks in advance. ... (2 Replies)
Discussion started by: baskivs
2 Replies

8. UNIX for Dummies Questions & Answers

Finding duplicates then copying, almost there, maybe?

Hi everyone. I'm trying to help my wife with a project, she has exported 200 images from many different folders, unfortunately there was a problem with the export and I need to find the master versions so that she doesn't have to go through and select them again. I need to: For each image in... (2 Replies)
Discussion started by: Rhinoskin
2 Replies

9. Shell Programming and Scripting

Finding duplicates in a file excluding specific pattern

I have unix file like below >newuser newuser <hello hello newone I want to find the unique values in the file(excluding <,>),so that the out put should be >newuser <hello newone can any body tell me what is command to get this new file. (7 Replies)
Discussion started by: shiva2985
7 Replies

10. UNIX for Beginners Questions & Answers

Finding a word through substring in a file

I have a text file that has some data like: PADHOGOA1 IOP055_VINREG5_1 ( .IO(VINREG5_1), .MONI(), .MON_D(px_IOP055_VINREG5_1_MON_D), .R0T(px_IOP054_VINREG5_0_R0T), .IO1() ); PADV30MA0 IOP056_VOUT3_IN ( .IO(VOUT3_IN), .V30M(px_IOP056_VOUT3_IN_V30M)); PADV30MA0 IOP057_VOUT3_OUT (... (2 Replies)
Discussion started by: utkarshkhanna44
2 Replies
ARRAY_MAP(3)								 1							      ARRAY_MAP(3)

array_map - Applies the callback to the elements of the given arrays

SYNOPSIS
array array_map (callable $callback, array $array1, [array $...]) DESCRIPTION
array_map(3) returns an array containing all the elements of $array1 after applying the $callback function to each one. The number of parameters that the $callback function accepts should match the number of arrays passed to the array_map(3) PARAMETERS
o $callback - Callback function to run for each element in each array. o $array1 - An array to run through the $callback function. o $... - Variable list of array arguments to run through the $callback function. RETURN VALUES
Returns an array containing all the elements of $array1 after applying the $callback function to each one. EXAMPLES
Example #1 array_map(3) example <?php function cube($n) { return($n * $n * $n); } $a = array(1, 2, 3, 4, 5); $b = array_map("cube", $a); print_r($b); ?> This makes $b have: Array ( [0] => 1 [1] => 8 [2] => 27 [3] => 64 [4] => 125 ) Example #2 array_map(3) using a lambda function (as of PHP 5.3.0) <?php $func = function($value) { return $value * 2; }; print_r(array_map($func, range(1, 5))); ?> Array ( [0] => 2 [1] => 4 [2] => 6 [3] => 8 [4] => 10 ) Example #3 array_map(3) - using more arrays <?php function show_Spanish($n, $m) { return("The number $n is called $m in Spanish"); } function map_Spanish($n, $m) { return(array($n => $m)); } $a = array(1, 2, 3, 4, 5); $b = array("uno", "dos", "tres", "cuatro", "cinco"); $c = array_map("show_Spanish", $a, $b); print_r($c); $d = array_map("map_Spanish", $a , $b); print_r($d); ?> The above example will output: // printout of $c Array ( [0] => The number 1 is called uno in Spanish [1] => The number 2 is called dos in Spanish [2] => The number 3 is called tres in Spanish [3] => The number 4 is called cuatro in Spanish [4] => The number 5 is called cinco in Spanish ) // printout of $d Array ( [0] => Array ( [1] => uno ) [1] => Array ( [2] => dos ) [2] => Array ( [3] => tres ) [3] => Array ( [4] => cuatro ) [4] => Array ( [5] => cinco ) ) Usually when using two or more arrays, they should be of equal length because the callback function is applied in parallel to the corre- sponding elements. If the arrays are of unequal length, shorter ones will be extended with empty elements to match the length of the long- est. An interesting use of this function is to construct an array of arrays, which can be easily performed by using NULL as the name of the callback function Example #4 Creating an array of arrays <?php $a = array(1, 2, 3, 4, 5); $b = array("one", "two", "three", "four", "five"); $c = array("uno", "dos", "tres", "cuatro", "cinco"); $d = array_map(null, $a, $b, $c); print_r($d); ?> The above example will output: Array ( [0] => Array ( [0] => 1 [1] => one [2] => uno ) [1] => Array ( [0] => 2 [1] => two [2] => dos ) [2] => Array ( [0] => 3 [1] => three [2] => tres ) [3] => Array ( [0] => 4 [1] => four [2] => cuatro ) [4] => Array ( [0] => 5 [1] => five [2] => cinco ) ) If the array argument contains string keys then the returned array will contain string keys if and only if exactly one array is passed. If more than one argument is passed then the returned array always has integer keys. Example #5 array_map(3) - with string keys <?php $arr = array("stringkey" => "value"); function cb1($a) { return array ($a); } function cb2($a, $b) { return array ($a, $b); } var_dump(array_map("cb1", $arr)); var_dump(array_map("cb2", $arr, $arr)); var_dump(array_map(null, $arr)); var_dump(array_map(null, $arr, $arr)); ?> The above example will output: array(1) { ["stringkey"]=> array(1) { [0]=> string(5) "value" } } array(1) { [0]=> array(2) { [0]=> string(5) "value" [1]=> string(5) "value" } } array(1) { ["stringkey"]=> string(5) "value" } array(1) { [0]=> array(2) { [0]=> string(5) "value" [1]=> string(5) "value" } } SEE ALSO
array_filter(3), array_reduce(3), array_walk(3), information about the callback type. PHP Documentation Group ARRAY_MAP(3)
All times are GMT -4. The time now is 10:30 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy