Sponsored Content
Full Discussion: Removing duplicates
Top Forums Shell Programming and Scripting Removing duplicates Post 302517294 by gctex on Tuesday 26th of April 2011 10:59:46 AM
Old 04-26-2011
Quote:
Originally Posted by pravin27
Try this,
Code:
awk -F"|" 'NR==FNR{if(a[$1]){ if(a[$1]<$2) {a[$1]=$2;b[$1]=NR}} else {a[$1]=$2;b[$1]=NR}}
NR>FNR{if(b[$1]==FNR){print}}' infile infile

Not sure, this is what I am getting:

Code:
 
!. srt1.sh
T1          |         1
T5          |         1
  T4        |         2
    T1      |         3
    T3      |         3
T4          |         1
  T1        |         2
  T3        |         2
T3          |         1
T2          |         1
T1          |         1
T5          |         1
  T4        |         2
    T1      |         3
    T3      |         3
T4          |         1
  T1        |         2
  T3        |         2
T3          |         1
T2          |         1
 
!cat srt1.sh
awk -F"|" 'NR==FNR{if(a[$1]){ if(a[$1]<$2) {a[$1]=$2;b[$1]=NR}} else {a[$1]=$2;b[$1]=NR}}
NR>FNR{if(b[$1]==FNR){print}}' fp1.txt fp1.txt

---------- Post updated at 09:59 AM ---------- Previous update was at 09:56 AM ----------

Quote:
Originally Posted by palanisvr
Code:
##--get unique tags
for i in ` cat testfile.txt | awk  '{print $1}'|sort -u`
do
grep $i testfile.txt >temp.txt
cat temp.txt | sort -n |tail -1  >>finaldata.txt
done

This is what I am getting:

Code:
!. srt.sh
T1          |         1
T2          |         1
T3          |         1
T4          |         1
T5          |         1
 
!cat srt.sh
for i in `cat fp1.txt | awk  '{print $1}'|sort -u`
do
grep $i fp1.txt >temp.txt
cat temp.txt | sort -n |tail -1  >>finaldata.txt
done
cat finaldata.txt

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates

Hi, I've been trying to removed duplicates lines with similar columns in a fixed width file and it's not working. I've search the forum but nothing comes close. I have a sample file: 27147140631203RA CCD * 27147140631203RA PPN * 37147140631207RD AAA 47147140631203RD JNA... (12 Replies)
Discussion started by: giannicello
12 Replies

2. UNIX for Dummies Questions & Answers

removing duplicates and sort -k

Hello experts, I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter sort -u -k 2,2 File.csv > Output.csv File.csv File Name|Document Name|Document Title|Organization Word Doc 1.doc|Word Document|Sample... (3 Replies)
Discussion started by: orahi001
3 Replies

3. Shell Programming and Scripting

removing duplicates

Hi I have a file that are a list of people & their credentials i recieve frequently The issue is that whne I catnet this list that duplicat entries exists & are NOT CONSECUTIVE (i.e. uniq -1 may not weork here ) I'm trying to write a scrip that will remove duplicate entries the script can... (5 Replies)
Discussion started by: stevie_velvet
5 Replies

4. Shell Programming and Scripting

Removing duplicates

Hi, I have a file in the below format., test test (10) to to (25) see see (45) and i need the output in the format of test 10 to 25 see 45 Some one help me? (6 Replies)
Discussion started by: imdadulla
6 Replies

5. UNIX for Advanced & Expert Users

removing duplicates.

Hi All In unix ,we have a file ,there we have to remove the duplicates by using one specific column. Can any body tell me the command. ex: file1 id,name 1,ww 2,qwq 2,asas 3,asa 4,asas 4,asas o/p: 1,ww 2,qwq 3,asa (7 Replies)
Discussion started by: raju4u
7 Replies

6. Emergency UNIX and Linux Support

Removing all the duplicates

i want to remove all the duplictaes in a file.I dont want even a single entry. For the input data: 12345|12|34 12345|13|23 3456|12|90 15670|12|13 12345|10|14 3456|12|13 i need the below data in one file 15670|12|13 and the below data in another file (9 Replies)
Discussion started by: pandeesh
9 Replies

7. Shell Programming and Scripting

Help in removing duplicates

I have an input file abc.txt with info like: abcd rateuse inklite robet rateuse abcd I need to remove duplicates from the file (eg: abcd,rateuse) from the file and need to place the contents in same file abc.txt if needed can be placed in another file. can anyone help me in this :( (4 Replies)
Discussion started by: rkrish
4 Replies

8. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

9. Shell Programming and Scripting

Removing duplicates except the last occurrence

Hi All, i have a file like below, @DB_FCTS\src\Data\Scripts\Delete_CU_OM_BIL_PRT_STMT_TYP.sql @DB_FCTS\src\Data\Scripts\Delete_CDP_BILL_LBL_MSG.sql @DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql @DB_FCTS\src\Data\Scripts\Insert_CU_OM_LBL_MSG.sql... (11 Replies)
Discussion started by: mechvijays
11 Replies

10. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies
TEXT2PS(L)																TEXT2PS(L)

NAME
text2ps - convert text files to PostScript SYNOPSIS
text2ps [ options ] [ files ] DESCRIPTION
Text2ps reads the input files (standard input if none are specified) and produces PostScript code which, when fed to a PostScript printer, will print the files. With text2ps it is possible to select any font, point size and number of columns. Options and files can be inter- mixed on the command line. Options are effective for all following files until they are overridden. Options Here follows a list of options that text2ps recognizes. Most numeric arguments are significant to one decimal place. Options are evalu- ated from left to right. Later options override earlier ones. -# n Print n copies of each page. (Default 1.) -c n Print in n columns. (Default 1.) -f font Print using font font. (Default Courier.) -p n Print with point size n. (Default 9.) -v n Use a vertical spacing of n points. If the vertical spacing is set to 0, the spacing will be 1.2 times the point size. (Default 0.) -l n Print n lines per column. When the line count is 0, print as many lines as will fit. (Default 0.) -r [p|l] Set the orientation to either portrait mode (p) or landscape mode (l). (Default p.) -b [+|-] Set page break mode. An argument + will force new files to be always printed on a new page (this is the default). After - new files will be put on the same page if there are still empty columns and the number of columns, the orientation or the number of copies didn't change. New files always start new columns. (Default -.) -mt n The top margin is n points. (Default 63.) -mb n The bottom margin is n points. (Default 63.) -ml n The left margin is n points. (Default 59.) -mr n The right margin is n points. (Default 59.) -mg n The inter-column gap is n points. (Default 25.) -t [+|-] If the argument is + the name of the file being printed will be printed on each page. If the argument is - the file name will not be printed. -t + implies -b +. -T text Print text as title on each page. This implies -t - and -b +. This option can be switched off by specifying -t - or -t +. (Default no title.) -F font Set the title font to font. (Default Helvetica.) -P n Set the title point size to n. (Default 12.) -B n Draw borders around each page. The number n specifies how to draw borders. N can have any of the following values or-ed in: 1 Draw a line along the left of the page. 2 Draw a line along the bottom of the page. 4 Draw a line along the right of the page. 8 Draw a line along the top of the page. 16 Draw a line between columns. This line does not connect to the lines along the top or bottom. 32 Draw a connecting line between the line between columns and the line along the top. 64 Draw a connecting line between the line between columns and the line along the bottom. When n is 0, no border lines are drawn. (Default no bordering lines.) -w n Tab stops are set every n spaces. Set the width of the TAB character. (Default 8.) -1 Sets up options to print in one column in portrait mode with the Courier font, so that you get 66 lines on a page. Equivalent to specifying the options -c 1 -f Courier -p 9 -v 0 -r p -l 0 -mt 63 -mb 63 -ml 59 -mr 59. This is the default. -2 Sets up options to print in two columns in landscape mode with the Courier font, so that you get two 66-line columns on a page. Equivalent to specifying the options -c 2 -f Courier -p 6 -v 0 -r l -l 0 -mt 63 -mb 63 -ml 59 -mr 59 -mg 25. Together with the -1 option, this is probably the most useful option. The name - means standard input. BUGS
Too many options. There is no way to specify where the title will be placed. If the font being used is not a constant width font and there are other characters than just tabs and spaces in front of a tab, the next character may not align properly. TEXT2PS(L)
All times are GMT -4. The time now is 01:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy