How to remove the lines from file using perl

dipakg · #1 (**permalink**) 06-03-2008

Can anyone tell me what could be the solution to following :

I have one .txt file which contains some seed information. This seed may appear multiple time in the file so what I want do is if this seed appears again in the file then that line should be removed.

here is the contents of .txt file :

187 TAL_SPI_CORRECT_REQ_OUT seed : 894328066;
187 TAL_SPI_NO_OF_BYTES_OUT seed : 894328066;
30445 TAL_SPI_MASTER_NO_MORE_DATA_OUT seed : 894328066;
352 TAL_SPI_MASTER_APPLY_COMMAND_OUT seed : 894328066;
944 TAL_SPI_MASTER_DATA_OUT seed : 894328066;
935 TAL_SPI_MASTER_ADDR_OUT seed : 894328066;
1 a_dsync100_no_wr_dsync_ucode_exec_nike seed : 1072076501;
1 a_dsync100_no_wr_dsync_ucode_exec_ath1 seed : 1805353229;
1 a_dsync100_no_wr_dsync_ucode_exec_ath0 seed : 1805353229;
1 a_dsync100_no_wr_dsync_ucode_exec_capt seed : 1805353229;
2 a_dsync_lcm_xm2_output seed : 894328066;
1 a_dsync_pulse_gen_to_rcsdi_clkgen seed : 894328066;
1 a_dsyncEnWrite_to_rcsdi_clkgen seed : 894328066;
3 a_extsynth_buf_en_ch1 seed : 1072076501;
3 a_extsynth_buf_en_ch0 seed : 1072076501;
3 a_extTrig2_width_ch1 seed : 1617187839;
2 a_extTrig1_width_ch1 seed : 1617187839;

So once seed : 894328066 appears in the file, the script should remove the line if it appears twice and so on.

Please help me ASAP.

Regards and Thanks,
Dipakg

radoulov · #2 (**permalink**) 06-03-2008

Use nawk or /usr/xpg4/bin/awk on Solaris.

Code:

awk '!_[$NF]++' input

Edit: Just read the subject.

Code:

perl -ane'print unless $_{$F[-1]}++' input

dipakg · #3 (**permalink**) 06-03-2008

Hi radoulov,

Thanks for your reply it worked but after some time it goes wrong or may be it worked wrong way...

What I did is
perl -ane'print unless $_{$F[-1]}++' YES.txt

As I manually does the analysis to find the seeds I get the result 71 seeds but by this command it gives only 21.

How is this possible I don't know.
Can you please figure it out?

Regards,
Dipak

radoulov · #4 (**permalink**) 06-03-2008

Quote:

Originally Posted by dipakg

Hi radoulov,

Thanks for your reply it worked but after some time it goes wrong or may be it worked wrong way...

What I did is
perl -ane'print unless $_{$F[-1]}++' YES.txt

As I manually does the analysis to find the seeds I get the result 71 seeds but by this command it gives only 21.
[...]

Please post more sampe data and the desired output using code tags

HTML Code:

[code]your text[/code]

What's the expected output given your first sample?
Is the below output the one you expect?

Code:

$ cat file
187 TAL_SPI_CORRECT_REQ_OUT seed : 894328066;
187 TAL_SPI_NO_OF_BYTES_OUT seed : 894328066;
30445 TAL_SPI_MASTER_NO_MORE_DATA_OUT seed : 894328066;
352 TAL_SPI_MASTER_APPLY_COMMAND_OUT seed : 894328066;
944 TAL_SPI_MASTER_DATA_OUT seed : 894328066;
935 TAL_SPI_MASTER_ADDR_OUT seed : 894328066;
1 a_dsync100_no_wr_dsync_ucode_exec_nike seed : 1072076501;
1 a_dsync100_no_wr_dsync_ucode_exec_ath1 seed : 1805353229;
1 a_dsync100_no_wr_dsync_ucode_exec_ath0 seed : 1805353229;
1 a_dsync100_no_wr_dsync_ucode_exec_capt seed : 1805353229;
2 a_dsync_lcm_xm2_output seed : 894328066;
1 a_dsync_pulse_gen_to_rcsdi_clkgen seed : 894328066;
1 a_dsyncEnWrite_to_rcsdi_clkgen seed : 894328066;
3 a_extsynth_buf_en_ch1 seed : 1072076501;
3 a_extsynth_buf_en_ch0 seed : 1072076501;
3 a_extTrig2_width_ch1 seed : 1617187839;
2 a_extTrig1_width_ch1 seed : 1617187839;
$ perl -ane'print unless $_{$F[-1]}++' file
187 TAL_SPI_CORRECT_REQ_OUT seed : 894328066;
1 a_dsync100_no_wr_dsync_ucode_exec_nike seed : 1072076501;
1 a_dsync100_no_wr_dsync_ucode_exec_ath1 seed : 1805353229;
3 a_extTrig2_width_ch1 seed : 1617187839;

dipakg · #5 (**permalink**) 06-03-2008

Here in I am attaching the text file.

For better view you can open it in wordpad.
Now this text file should give me total 71 seeds as output(I mean total 71 lines should be appear as output).

radoulov · #6 (**permalink**) 06-03-2008

Code:

perl -ne'/: (\d+?);/;print unless $_{$1}++' YES.txt

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
vi to remove lines in file	kjbaumann	UNIX for Dummies Questions & Answers	2	03-13-2009 03:40 AM
How to remove the specific lines from file using perl	dipakg	Shell Programming and Scripting	4	06-11-2008 02:45 AM
remove specific lines from flat file using perl	meghana	Shell Programming and Scripting	12	02-12-2008 09:50 PM
remove lines from file	bluemoon1	Shell Programming and Scripting	9	11-09-2007 01:40 AM
how to remove specific lines from a file	bluemoon1	Shell Programming and Scripting	17	10-07-2007 10:40 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by