Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate entries from the same line Post 302851003 by pamu on Friday 6th of September 2013 08:20:52 AM
Old 09-06-2013
Code:
 awk '{for(i=1;i<=NF;i++){if(!A[$i,NR]++){S=S?S OFS $i:$i}}print S;S=""}' OFS="\t" file

ID      source
1       GM      GF
2       GM      GF
3       GM      GF      SF
4       FF
5       FF      GM
6       FF

This User Gave Thanks to pamu For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove Duplicate line

Hi, I have a scenario here where I have created a flatfile with the below mentioned information. File as you can see is dispalyed in three columns 1st column is FileNameString 2nd column is Report_Name (this has spaces) 3rd column is Flag Result file needed is, removal of duplicate... (1 Reply)
Discussion started by: Student37
1 Replies

2. UNIX for Dummies Questions & Answers

Remove duplicate entry in one line

Can anyone help me how can i print only the unique entry in a line? MI_AP MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN MBS_AP NT_CAR so that it will on output the one unique entry per line. MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN NT_CAR I can't find the same situation on the knowledge... (5 Replies)
Discussion started by: kharen11
5 Replies

3. Shell Programming and Scripting

remove duplicate words in a line

Hi, Please help! I have a file having duplicate words in some line and I want to remove the duplicate words. The order of the words in the output file doesn't matter. INPUT_FILE pink_kite red_pen ball pink_kite ball yellow_flower white no white no cloud nine_pen pink cloud pink nine_pen... (6 Replies)
Discussion started by: sam_2921
6 Replies

4. Shell Programming and Scripting

awk script to remove duplicate rows in line

i have the long file more than one ns and www and mx in the line like . i need the first ns record and first www and first mx from line . the records are seperated with tthe ; i am try ing in awk scripting not getiing the solution. ... (4 Replies)
Discussion started by: kiranmosarla
4 Replies

5. Shell Programming and Scripting

remove of duplicate line from a file

I have a file a.txt having content like deepak ram sham deepram sita kumar I Want to delete the first line containing "deep" ... I tried using... grep -i 'deep' a.txt It gives me 2 rows...I want to delete the first one.. + need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies

6. Shell Programming and Scripting

Remove duplicate line on condition

Hi Ive been scratching over this for some time with no solution. I have a file like this 1 bla bla 1 2 bla bla 2 4 bla bla 3 5 bla bla 1 6 bla bla 1 I want to remove consecutive occurrences of lines like bla bla 1, but the first column may be different. Any ideasss?? (23 Replies)
Discussion started by: jamie_123
23 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Two Line Pairs?

So I have a bunch of files that look like this >gi|33332323 MMKCRGVIMVVEKVMKRDGRIVPFDESRIRWAVQ--- >gi|45235353 MMKCR----VEKMRDVFFDESIRWAVQ They go on...sequences are much longer but all in two line (fasta) format. I want to remove duplicate pairs of ID(GI) number and sequence. I tried... (12 Replies)
Discussion started by: bakere19
12 Replies

8. Shell Programming and Scripting

Remove duplicate line starting with a pattern

HI, I have the below input file /* ----------------- cmdsDlyStartFWJ -----------------*/ UNIX_JOB CMDS065J RUN ANY CMDNAME sleep 5 AGENT CMDSHP USER proddata RUN MON,TUE,WED,THU,FRI DELAYSUB 02:00 /* "Triggers daily file watcher jobs" */ ENVAR... (5 Replies)
Discussion started by: varun22486
5 Replies

9. Shell Programming and Scripting

Remove sections based on duplicate first line

Hi, I have a file with many sections in it. Each section is separated by a blank line. The first line of each section would determine if the section is duplicate or not. if the section is duplicate then remove the entire section from the file. below is the example of input and output.... (5 Replies)
Discussion started by: ahmedwaseem2000
5 Replies

10. UNIX for Dummies Questions & Answers

Using awk to remove duplicate line if field is empty

Hi all, I've got a file that has 12 fields. I've merged 2 files and there will be some duplicates in the following: FILE: 1. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, 100 2. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, (EMPTY) 3. CDC, 54321, TEST3,... (4 Replies)
Discussion started by: tugar
4 Replies
PAGESIZES(5)							File Formats Manual						      PAGESIZES(5)

NAME
pagesizes - HylaFAX page size definitions DESCRIPTION
The pagesizes database defines the page dimensions and guaranteed reproducible areas (GRA) for well-known page sizes. The GRA is the por- tion of the page that is guaranteed to be imaged during facsimile transmission. This region is typically less than the full page dimen- sions because of paper roller contacts and other mechanical aspects of the printing process in a facsimile machine. All HylaFAX programs that require page size information read the information from this database using a page size name. Documents should be prepared such that the full page dimensions are employed with the imaged area contained within the GRA. The system-wide default page size to use in preparing documents for transmission is given by the ``default'' entry in the database. (NB: the default entry should be placed last so that inverse matches find the real page size name and not the default entry.) The page size database is an ASCII file with the following format. Each entry consists of whitespace-separated fields: name abbrev width height gra-width gra-height top-margin left-margin Fields have the following interpretation: name the full name for the page size; e.g. ISO A4; abbrev an abbreviated version of the full name for use in compact listings such as the receive queue listing printed by faxstat(1); width the full width of the page; height the full height of the page; gra-width the width of the GRA; gra-height the height of the GRA; top-margin the margin between the top of the full page and the top of the GRA; left-margin the margin between the left side of the full page the left side of the GRA. The first two fields must be separated from the subsequent fields by a tab character (possibly followed by more whitespace); this is done to easily permit blank characters to be included in names. Otherwise fields can be separated by any amount of any kind of whitespace. Numbers are all base 10 and in basic measurement units (BMU); defined as 1/1200 x 25.4 millimeters for paper output with a scale factor of one. All fields must be present on a single line; otherwise the entry is ignored. Comments are introduced by the ``#'' character and con- tinue to the end the line. Page size database lookups are either by name or by page dimensions. Lookups by name are done by sequentially scanning the entries in the database for the first entry that has a matching abbreviation or substring of the page size name field. The string comparisons ignore case so, for example, ``a4'' would match a full name of ``ISO A4''. Lookups by dimension scan the entire database and return the page with the closest dimensions using a straightforward distance metric. If the difference in dimensions of the closest match is greater than 1/2 inch on each side, then no page entry is returned for a lookup by dimension. SEE ALSO
hylafax-client(1), faxmail(1), faxstat(1), sendfax(1), sgi2fax(1), textfmt(1). December 5, 1994 PAGESIZES(5)
All times are GMT -4. The time now is 04:35 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy