Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Remove Duplicate Two Line Pairs?

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Remove Duplicate Two Line Pairs?
# 8  
Old 02-26-2013
Can you post few lines from your file for which you are seeing issues? I guess there may be some differences which you might have missed.
# 9  
Old 02-26-2013
Code:
>gi|332981533
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------MKLTDNAIK-------------------------------VLEKRY-----LAKDE-----------------------------Q---GNII-----ET-PEQMFR-------------------------------------------------------------------------------------------RVAHHVAQAD--SIY--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------DPKVD--------VSKT--------EQE--FYD-----------I-MTELE----------------------F-------LP----NS-PTL-------------MNAG--------------------------R-----PLG--------------------------Q------LSA-------------CF------------VLPI-----------EDSM-----E------------GIFDSVKNAA------------------------------LIHKSG----------------------------------------------------------------------------GG----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TGFS----------------------------------------------------------------------------------------------------------------------FSRL--RP----KGATVRSTGGVA------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------SGPVSF-MKVFNAATE------------------------AVKQGG------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TRRGANMGILRVDHPD-----------IL--EFIQCKKDN-------------------------S-----------------D----ITNFN-I-SVGIT---EKFMEAV---EKDEEYD--LID--PH---------------------------------------------------------------TG-----KIT----------------------RRLRAR-----QVF--DLIVDMAWHNGE----PGIVFL-DRINK--DNV-VPALG-E------------IESTNPC-ITGDTWVLTENGPEQVVNILGKQISLALNGDFYSSSEIGF---------------------------------------------FKTGSKSTISIRTDKGYKIEVTPDHKIRVAVSIT---------RDNIQEEWKPAGELKPGDHIV-----------------------------------------------------------------------------------------LSDNRGLVWEG-----------RGSFEDGYLLGLLLGDGTLKNDGGIISVWGDDY---GADSIIKAAEEAA---------------FTLPHRADFNGFKSMIDVRKEHRMQMSALRDLAAVYKIFSGDKRITEELEKTSYDFHRGFLRGIFDADGTV--TGNQEKGVSVG---LWQNDIEGLRIIQRMLLRLGIVSTL-HIDRKAEGIKQMPDG------KRGTKEYHIRS-GHELVITSSNLA-IFFEKVGF-SDAKKHDLLKQRLAE---YKRAINRERFIATVDEIVQAGEKGVYDVQIPGVNAFDANGICVHNCGEQPLLPYESCNLGSINLLSVV--Q----------PVDGD-------------KWE--------INYAKLARIVDTAVHFLDNVID--VNLYPLPEIEKMT-KRMRKIGLGVMGWADMLFRLGIPYNSDEAIELGTKLMKFICDRARQQSAELA--EQRGAFPAYEQSI-W-------------------------------------------------------------------------------KDKGLK-VRNATVTTIAPTG--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------SISI-IA-GVSS-GIEPLFAISFVR-NVMD--NT-QL--PEVHPIFEEVAKQR----GF------------YS--------A---ELMRQIAH-------QG----SIQHM------DGIPDD--VKRVFVTAHD-------ISPEYHVRMQAAFQR------Y-TDNAVSK-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TVNFPNSATQEDVRQVYILAYKLGCKGVTIYRDGSRETQVLNIGDK-DK--------------------------------------------------------------------------------------------VGADVMIQPEPT-------------------------------IIPRPRP---EITRGITEKVR------IGCGNL------------------------YITVNYD-DQGIC---------EVFTNLGK-------AGG-----CPSQSEATSRLVSLALR------SGIDVKALVDQLKGIRCHT-----------TIRQRGLKVLSCPDAIART------IEKVMKIQSDEQQNHFGIADDIDEQNDKEGRA--------------------------------------------------------------------------------------------VCPECGGE------LEHESGCV--MCPS------CG--Y-SKCG---
>gi|332981533
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------MKLTDNAIK-------------------------------VLEKRY-----LAKDE-----------------------------Q---GNII-----ET-PEQMFR-------------------------------------------------------------------------------------------RVAHHVAQAD--SIY--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------DPKVD--------VSKT--------EQE--FYD-----------I-MTELE----------------------F-------LP----NS-PTL-------------MNAG--------------------------R-----PLG--------------------------Q------LSA-------------CF------------VLPI-----------EDSM-----E------------GIFDSVKNAA------------------------------LIHKSG----------------------------------------------------------------------------GG----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TGFS----------------------------------------------------------------------------------------------------------------------FSRL--RP----KGATVRSTGGVA------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------SGPVSF-MKVFNAATE------------------------AVKQGG------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TRRGANMGILRVDHPD-----------IL--EFIQCKKDN----------------------------------------------SDITNFN-I-SVGIT---EKFMEAV---EKDEEYD--LID--PH---------------------------------------------------------------TG-----KIT----------------------RRLRAR-----QVF--DLIVDMAWHNGE----PGIVFL-DRINK--DNV-VPALG-E------------IESTNPC-ITGDTWVLTENGPEQVVNILGKQISLALNGDFYSSSEIGF---------------------------------------------FKTGSKSTISIRTDKGYKIEVTPDHKIRVAVSIT---------RDNIQEEWKPAGELKPGDHIV-----------------------------------------------------------------------------------------LSDNRGLVWEG-----------RGSFEDGYLLGLLLGDGTLKNDGGIISVWGDDY---GADSIIKAAEEAA---------------FTLPHRADFNGFKSMIDVRKEHRMQMSALRDLAAVYKIFSGDKRITEELEKTSYDFHRGFLRGIFDADGTVTGNQEKGVSVG-----LWQNDIEGLRIIQRMLLRLGIVSTL-HIDRKAEGIKQMPDG------KRGTKEYHIRS-GHELVITSSNLA-IFFEKVGFSDAKKHDLLKQRLAE----YKRAINRERFIATVDEIVQAGEKGVYDVQIPGVNAFDANGICVHNCGEQPLLPYESCNLGSINLLSVV--Q----------PVDGD-------------KWE--------INYAKLARIVDTAVHFLDNVID--VNLYPLPEIEKMT-KRMRKIGLGVMGWADMLFRLGIPYNSDEAIELGTKLMKFICDRARQQSAELA--EQRGAFPAYEQSI-W-------------------------------------------------------------------------------KDKGLK-VRNATVTTIAPTG--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------SISI-IA-GVSS-GIEPLFAISFVR-NVMD--NT-QL--PEVHPIFEEVAKQR----GF------------YS--------A---ELMRQIAH-------QG----SIQHM------DGIPDD--VKRVFVTAHD-------ISPEYHVRMQAAFQR------Y-TDNAVSK-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TVNFPNSATQEDVRQVYILAYKLGCKGVTIYRDGSRETQVLNIGDK-DK--------------------------------------------------------------------------------------------VGADVMIQPEPT-------------------------------IIPRPRP---EITRGITEKVR------IGCGNL------------------------YITVNYD-DQGIC---------EVFTNLGK-------AGG-----CPSQSEATSRLVSLALR------SGIDVKALVDQLKGIRCHT-----------TIRQRGLKVLSCPDAIARTIEKVMKIQSDEQQNHFGIADDIDEQNDKEGRA--------------------------------------------------------------------------------------------------VCPECGGE------LEHESGCV--MCPS------CG--Y-SKCG---
>gi|240102479
---------------------------------------------------------MAVEKVMKRDGRIVPFDRERIRWAIK-----------RAMLEVGVHDDKLLNRVVR-----RVVRRINELY----DGQVPHIENIQDIVELELMRAGLFEV----------AKAYILYRKKK-----------AEIREEKKKI--------------LNKDRLDEIDKRFSLNALR-------------------------------VLASRY-----LIRNE-----------------------------K---GEII-----ES-PRELFE-------------------------------------------------------------------------------------------RVATLAVIPD--LLY-------------------------------------------------------------------DERVYDKNGKHEQDLSRVKYYLEHFEEFDGRYSIG------------------------------------------------------------RFKLNKYHFERLVNLYRELAEKGRMKVSIDEFLGMLENGAFDD--------YESE--------VEE--YFR-----------L-MTGQV----------------------F-------MP----NT-PAL-------------INSG--------------------------R-----PLG--------------------------M------LSA-------------CF------------VVPI-----------EDDM-----E------------SIMKAAHDVA------------------------------MIQKSG----------------------------------------------------------------------------GG----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TGLN----------------------------------------------------------------------------------------------------------------------FSKL--RP----EGDFVGSTAGAA------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------SGPVSF-MHLIDAVSD------------------------VIKQGG------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------VRRGANMGILEVWHPD-----------IE--KFIHAKEKN-------------------------------------------IGTNVLSNFN-I-SVGIW---EDFWEAL---RDGKRYP--LVN--PR---------------------------------------------------------------TG-----EKV----------------------KEIDPK-----SLF--EELAFMAWSKAD----PGVIFF-DVINR--RNVLEPAKG-G-----------PIRATNP-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------CGEEPLYEYESCNLASINLAKFV--K---------YDDEG--------------KPY--------FDWDEYAYVIQKVAKYLDNAID--VNRFPLPEIDYNT-KLTRRIGVGMMGLADALFKLGIPYNSEEGFAFMRKATEYLTFYAYKYSVEAA--KKRGTFPLYEKSR-Y--------------------------------------KDGELPVEGFY----------------HREIWNLPWDELVEEIKKHG-VRNGMVTTCPPTG--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------SVSM-IA-DTSS-GIEPIFALVYKK-SVTV--G--EF--YYVDPVFEAELKKR----GL------------WS--------D---EILKKISD-----N-YG----SVQGL------EEIPED--MQRVFVTSMD-------VHWLDHILAQANIQL------W-LTDSASK-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TINMPNDATVEDVKAAYLLAYKLGCKGITVYRDGSLSVQVYSVEGE------------------------------------------------------------------------------------------------------------------------------------------KRKRVPA---KPSRYAVEKLK-----AVVEAEP------------------------WLAKFINVE-------------AILNGTNGKGKAALPSGLTFSVAHITPAKPPVREHPHHAE------KPEIPEEKIKELLGVA-------------------------------------------------------------------------------------------------------------------------------------------------------------------YCPVCYERDGELVELRMESGCA--TCPR------CG--W-SKCVIS-
>gi|304315898
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------MNLTENSKK-------------------------------VLERRY-----LAKDE-----------------------------N---GRVV-----ET-VEELFE-------------------------------------------------------------------------------------------RVAKSISEID--KKY--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------DSNAN--------IEEL--------KNK--FYD-----------M-MTNLD----------------------F-------LP----NS-PTL-------------MNAG--------------------------R-----PLG--------------------------Q------LSA-------------CF------------VLPV-----------GDSM-----E------------EIFDAVKYAA------------------------------IIHKSG----------------------------------------------------------------------------GG----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TGFS----------------------------------------------------------------------------------------------------------------------FSRL--RP----KGATVKSTGGVA------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------SGPVSF-MKVFNSATE------------------------AVKQGG------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TRRGANMGILRIDHPD-----------IL--EFIQCKQDN--------------------------------------------NE--ITNFN-I-SVGIT---EDFMKAV---ESGEDYD--LVD--PH---------------------------------------------------------------TK-----KVV----------------------NKLNAR-----EVF--ELIVEMAWKNGE----PGIVFL-DRINE--KNP-TPAIG-E------------IESTNPC-VTGDTWVMTTEGPKQVNDLIGKPFEAVINGRFYRTTNEGF---------------------------------------------FKTGHKHIVLVETIEGYSIRLTDDHKILKVVDSS---------LNEMKTEWVSAIELKPGDKII----------------------------------------------LNNNRNLIGWSGELDEGDGYLLGLLV-------------GDGVLKRDTAILSVW-----------KEGKAVG---------DVNNCGVDNVMQYALDC---AMRLPH---------RRD----------FTGWMEI-----KGRNEYRLKLASLRDLALKMGM----HNGFKTVTPELEKMSSSAYIGFIRGLFDCDASV--QGSPEKGASIR---LAQSDLDLLKAVQRMLLRLGIVSKI-YVNRRKASMKLMPDG------KGSLKEYKIKP-QHELCISGDNIE-IYAKRIGF-QDLKKMHRLNTLLSS---YKKGSHQERFVARVLDIKESGFEDVYDVQVPGINSFDANGIIIHNCGEQPLLPYESCNLGSINLKNML--K----------EENG--------------KYE--------VDYDKLRDTVHNAVHFLDNVID--ANKYPLPQIDEMT-KGTRKIGLGVMGFADMLLMLNIPYNSEEAVEFADKLMKFIDEESKKASMELA--KKRGVFKYFDKSI-Y-------------------------------------------------------------------------------KDKNIK-LRNATTTTIAPTG--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TISI-IA-GTSS-GIEPLFAIAMTR-NVMD--NT-QL--VEVNPIFKEVALKR----GF------------YS--------D---ELMKKIAE-------QG----TLKGI------DSIPDD--VKKVFVTAHD-------IDPVWHIRMQAAFQK------H-VDNAVSK-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TVNFRHDATVDDVREVYELAYRLGLKGVTIYRDGSRDSQVLNLGIK-KD---------------------------------------------------------------------------------------KKEETKSDKKSDIKKNQ-------------------------------IVPRPRP---PVTKGITEKVR------IGCGNL------------------------YITVNYD-DNGIC---------EVFTNLGR-------AGG-----CPSQSEATSRLISIALR------SGLDAKSIVEQLKGIRCHS----TLRQMANNKEIKVL---SCPDAIAKVIEKVMKLKVEENENFAPIDVPINGSSDKYDDEEELYAAFTDDSHEDH-----------------------------------------------------------------------------------FCPECGSE------IEHEGGCV--VCKN------CG--Y-SKCG---
>gi|304315898
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------MNLTENSKK-------------------------------VLERRY-----LAKDE-----------------------------N---GRVV-----ET-VEELFE-------------------------------------------------------------------------------------------RVAKSISEID--KKY--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------DSNAN--------IEEL--------KNK--FYD-----------M-MTNLD----------------------F-------LP----NS-PTL-------------MNAG--------------------------R-----PLG--------------------------Q------LSA-------------CF------------VLPV-----------GDSM-----E------------EIFDAVKYAA------------------------------IIHKSG----------------------------------------------------------------------------GG----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TGFS----------------------------------------------------------------------------------------------------------------------FSRL--RP----KGATVKSTGGVA------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------SGPVSF-MKVFNSATE------------------------AVKQGG------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TRRGANMGILRIDHPD-----------IL--EFIQCKQDN----------------------------------------------NEITNFN-I-SVGIT---EDFMKAV---ESGEDYD--LVD--PH---------------------------------------------------------------TK-----KVV----------------------NKLNAR-----EVF--ELIVEMAWKNGE----PGIVFL-DRINE--KNP-TPAIG-E------------IESTNPC-VTGDTWVMTTEGPKQVNDLIGKPFEAVINGRFYRTTNEGF---------------------------------------------FKTGHKHIVLVETIEGYSIRLTDDHKILKVVDSS---------LNEMKTEWVSAIELKPGDKII----------------------------------------------------------------------------------------LNNNRNLIGWSG-----------ELDEGDGYLLGLLVGDGVLKRDTAILSVWKEGK---AVGDVNNCGVDNVMQYALDC--------AMRLPHRRDFTGWMEIKGRNEYRLKLASLRDLALKMGMHNGFKTVTPELEKMSSSAYIGFIRGLFDCDASVQGSPEKGASIR-----LAQSDLDLLKAVQRMLLRLGIVSKI-YVNRRKASMKLMPDG------KGSLKEYKIKP-QHELCISGDNIE-IYAKRIGFQDLKKMHRLNTLLSS----YKKGSHQERFVARVLDIKESGFEDVYDVQVPGINSFDANGIIIHNCGEQPLLPYESCNLGSINLKNML--K----------EENG--------------KYE--------VDYDKLRDTVHNAVHFLDNVID--ANKYPLPQIDEMT-KGTRKIGLGVMGFADMLLMLNIPYNSEEAVEFADKLMKFIDEESKKASMELA--KKRGVFKYFDKSI-Y-------------------------------------------------------------------------------KDKNIK-LRNATTTTIAPTG--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TISI-IA-GTSS-GIEPLFAIAMTR-NVMD--NT-QL--VEVNPIFKEVALKR----GF------------YS--------D---ELMKKIAE-------QG----TLKGI------DSIPDD--VKKVFVTAHD-------IDPVWHIRMQAAFQK------H-VDNAVSK-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------TVNFRHDATVDDVREVYELAYRLGLKGVTIYRDGSRDSQVLNLGIK-KD---------------------------------------------------------------------------------------KKEETKSDKKSDIKKNQ-------------------------------IVPRPRP---PVTKGITEKVR------IGCGNL------------------------YITVNYD-DNGIC---------EVFTNLGR-------AGG-----CPSQSEATSRLISIALR------SGLDAKSIVEQLKGIRCHS-------TLRQMANNKEIKVLSCPDAIAKVIEKVMKLKVEENENFAPIDVPINGSSDKYDDEEELYAAFTDDSHEDH-----------------------------------------------------------------------------------FCPECGSE------IEHEGGCV--VCKN------CG--Y-SKCG---

Unfortunately, they're really long...protein sequences.
# 10  
Old 02-26-2013
I checked these 5 number & sequence and they are all unique if you consider the dashes - in between the sequences.

But if you remove the dashes, then you have only 3 unique number & sequence combinations.

So do you want to remove the dashes and compare?
# 11  
Old 02-26-2013
Huh. That's really strange I'm not sure how that happened. I'm going to have to unalign anyways (removes dashes). So I'll just run another uniqueness command at that stage and it should be a done deal. Thanks.

---------- Post updated 02-26-13 at 07:26 AM ---------- Previous update was 02-25-13 at 11:48 PM ----------

Actually, how do I make it ignore the dashes?
# 12  
Old 02-26-2013
Quote:
Originally Posted by bakere19
Actually, how do I make it ignore the dashes?
Well if you want to remove dashes, you can try this:
Code:
paste - - < file | awk '{$1=$1;gsub(/-/,x)}1' | awk '!a[$0]++' | tr ' ' '\n'

But I have no idea how to ignore them. Someone else in this forum might have a solution for ignoring dashes.
# 13  
Old 02-26-2013
Quote:
Originally Posted by bipinajith
Code:
paste - - < uniquegilist | awk '!a[$0]++' | tr '\t' '\n'


Quote:
Originally Posted by bipinajith
Well if you want to remove dashes, you can try this:
Code:
paste - - < file | awk '{$1=$1;gsub(/-/,x)}1' | awk '!a[$0]++' | tr ' ' '\n'

But I have no idea how to ignore them. Someone else in this forum might have a solution for ignoring dashes.
Perhaps by modifying your first suggestion to use an array index in which consecutive dashes have been folded into a single dash.

Regards,
Alister
Last edited by alister; 02-26-2013 at 02:12 PM..
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Using awk to remove duplicate line if field is empty

Hi all, I've got a file that has 12 fields. I've merged 2 files and there will be some duplicates in the following: FILE: 1. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, 100 2. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, (EMPTY) 3. CDC, 54321, TEST3,... (4 Replies)
Discussion started by: tugar
4 Replies

2. Shell Programming and Scripting

Remove sections based on duplicate first line

Hi, I have a file with many sections in it. Each section is separated by a blank line. The first line of each section would determine if the section is duplicate or not. if the section is duplicate then remove the entire section from the file. below is the example of input and output.... (5 Replies)
Discussion started by: ahmedwaseem2000
5 Replies

3. Shell Programming and Scripting

Remove lines with duplicate pairs where AB is equal to BA

I have a file with four columns like dmn10003t1 PF00001 PF00022 dmn12390t1 dmn10008t1 PF00069 PF00027 dmn9781t1 dmn10008t1 PF00068 PF00027 dmn9781t1 dmn10008t1 PF00069 PF00069 dmn9781t1 dmn12390t1 PF00069 PF00076 dmn10003t1 I want to create a new file by comparing the repeated word pairs... (2 Replies)
Discussion started by: sammy777
2 Replies

4. Shell Programming and Scripting

Remove duplicate line starting with a pattern

HI, I have the below input file /* ----------------- cmdsDlyStartFWJ -----------------*/ UNIX_JOB CMDS065J RUN ANY CMDNAME sleep 5 AGENT CMDSHP USER proddata RUN MON,TUE,WED,THU,FRI DELAYSUB 02:00 /* "Triggers daily file watcher jobs" */ ENVAR... (5 Replies)
Discussion started by: varun22486
5 Replies

5. Shell Programming and Scripting

Remove duplicate entries from the same line

Hello, I have a file which have several duplicate entries on the same line: File ID source 1 GM GF GM 2 GM GF GM GF GM GF GM GF GM GF 3 GM GF GM SF GM GF GM SF 4 FF FF FF FF 5 FF GM FF ... (2 Replies)
Discussion started by: nans
2 Replies

6. Shell Programming and Scripting

Remove duplicate line on condition

Hi Ive been scratching over this for some time with no solution. I have a file like this 1 bla bla 1 2 bla bla 2 4 bla bla 3 5 bla bla 1 6 bla bla 1 I want to remove consecutive occurrences of lines like bla bla 1, but the first column may be different. Any ideasss?? (23 Replies)
Discussion started by: jamie_123
23 Replies

7. Shell Programming and Scripting

remove of duplicate line from a file

I have a file a.txt having content like deepak ram sham deepram sita kumar I Want to delete the first line containing "deep" ... I tried using... grep -i 'deep' a.txt It gives me 2 rows...I want to delete the first one.. + need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies

8. Shell Programming and Scripting

remove duplicate words in a line

Hi, Please help! I have a file having duplicate words in some line and I want to remove the duplicate words. The order of the words in the output file doesn't matter. INPUT_FILE pink_kite red_pen ball pink_kite ball yellow_flower white no white no cloud nine_pen pink cloud pink nine_pen... (6 Replies)
Discussion started by: sam_2921
6 Replies

9. UNIX for Dummies Questions & Answers

Remove duplicate entry in one line

Can anyone help me how can i print only the unique entry in a line? MI_AP MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN MBS_AP NT_CAR so that it will on output the one unique entry per line. MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN NT_CAR I can't find the same situation on the knowledge... (5 Replies)
Discussion started by: kharen11
5 Replies

10. UNIX for Dummies Questions & Answers

Remove Duplicate line

Hi, I have a scenario here where I have created a flatfile with the below mentioned information. File as you can see is dispalyed in three columns 1st column is FileNameString 2nd column is Report_Name (this has spaces) 3rd column is Flag Result file needed is, removal of duplicate... (1 Reply)
Discussion started by: Student37
1 Replies
Login or Register to Ask a Question