Manipulating word based off of contents


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Manipulating word based off of contents
# 1  
Old 11-19-2009
Question Manipulating word based off of contents

Hello everyone, my first post here, please feel free to inform me if my question can be better formatted so my future posts may be more clear.

I have a large text file which I need parsed in one specific way, I have done the rest of the processing, I am simply lacking the last aspect of such.

I need to look for lines in a file that begin with 'Key:' and of those lines that begin with key, delete all words in that line that do not contain the character '@', ignoring the first word (KeySmilie

So for example:


EDIT: Fixed example


before:
Code:
...
Key: hsfkfs,  sdf@fsdfsdfm,  sfsfdsdfk...334, tester@3, joe
Joe:  hp-0098, fhnjf@jhgsg,  shg@hjgppp
...

after:
Code:
...
Key:  sdf@fsdfsdfm, tester@3,
Joe:  hp-0098, fhnjf@jhgsg,  shg@hjgppp
...

I would prefer my solution be stream processing based (simply using an awk or sed command) instead of for-looping through the file, however I do not know if that is possible. I have explored possibilities with many tools but I am running out of ideas.

Any help is appreciated.

Last edited by ryanfx; 11-19-2009 at 03:23 PM..
# 2  
Old 11-19-2009
Code:
sed "/^Key/s/\w[a-z0-9]*@[a-z0-9]*[ ,]*//g" file1

Key: hsfkfs,  sfsfdsdfk...334, joe
Joe:  hp-0098, fhnjf@jhgsg,  shg@hjgppp

# 3  
Old 11-19-2009
Quote:
Originally Posted by scottn
Code:
sed "/^Key/s/\w[a-z0-9]*@[a-z0-9]*[ ,]*//g" file1

Key: hsfkfs,  sfsfdsdfk...334, joe
Joe:  hp-0098, fhnjf@jhgsg,  shg@hjgppp

I'm terribly sorry my example was actually reversed - this needs to the other way around (my original text was correct). It needs to LEAVE all words with @ in it, and delete everything else. I'm sorry for the error.

Is there a way to reverse results (much like grep -v)?
# 4  
Old 11-19-2009
No probs.

Here's some horrible awk, until I fix your sed!

Code:
awk '
  /^Key/ {
    printf $1;
    for( I = 2; I <= NF; I++ )
      if ( $I ~ /@/ ) printf " " $I
    print ""
    next
  }
  1
' file1

Key: sdf@fsdfsdfm, tester@3,
Joe:  hp-0098, fhnjf@jhgsg,  shg@hjgppp


Last edited by Scott; 11-19-2009 at 03:33 PM..
# 5  
Old 11-19-2009
Purpose

What is the purpose of the filter? It looks like a good way to collect emails for spamming.
# 6  
Old 11-19-2009
Quote:
Originally Posted by m1xram
What is the purpose of the filter? It looks like a good way to collect emails for spamming.

I gave the @ symbol as an ambiguous symbol; its meaning was simply an example. If I have to come to unix forums to help me parse large ammounts of text to enhance my spamming capabilities one would conclude I am a terrible spammer!

I also see your sed statements work off a-z, 0-9; - how would one account for characters that are shift + numbers as well (possibly) being in the words such as '#$%-' while unsure of their exact order therein? Can you block special characters like [!-&] much like the alpha [a-z]? Does one have to do so in correct ASCII order?

I appreciate all of your help!
# 7  
Old 11-19-2009
filter

# Filter out non @ words on Key: records.

cat FILE | sed '/^Key:/s/ \+[^@ ]\+[@][^@ ]\+ \?//g'

# This is crude though, an AWK solution that splits and trims first by ':'
# and then by ',' would be much more exact. Will work on it.

---------- Post updated at 02:12 PM ---------- Previous update was at 12:44 PM ----------

# Ok, how about a PERL solution?
# Save below program as SOMETHING.pl
chmod +x SOMETHING.pl

# Example:
cat FILE | ./SOMETHING.pl -v -p '@'
# -v, Reverse match.
# -p PATTERN, Default is '@'
----------------------------------------------------------
Code:
#!/usr/bin/perl 

use Getopt::Std;

my (@in, @word, @out);
my $pat;
my $reverse = 0;

getopts("vp:");
if (defined($opt_v)) {
    $reverse = 1;
}
if (defined($opt_p)) {
    $pat = $opt_p;
} else {
    $pat = '@';
}

while(<STDIN>) {
    $line = $_;
    chomp $line;
    if ($line =~ /^Key:/) {
    @in = ();
    @in = (split(/: */, $line));
    @word = ();
    @word = (split(/, */, $in[1]));
    @out = ();
    foreach my $item (@word) {
        if ($reverse eq 0 && ($item =~ m/$pat/)) {
        push(@out, $item);
        } elsif ($reverse eq 1 && ($item !~ m/$pat/)) {
        push(@out, $item);
        }
    }
    print "$in[0]: ", join(", ", @out), "\n";
    }
    else {
    print "$line\n";
    }
}
exit 0;

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

File comparison based on contents

Hi I have 2 files 1.del ---- 1,2,3,4,5 1,2,3,4,4 1,1,1,1,2 2.del ---- 1,2,3,4,5 1, 1,2,3,4,4 1,1,1,1,2 I need to compare the above two files in unix, as in the output should only tell the difference in contents as I should get only the line 1 ( from 2.del) , rest all lines are... (4 Replies)
Discussion started by: Ethen561
4 Replies

2. UNIX Desktop Questions & Answers

Manipulating a list of numbers based on values

Hi, I have a single column of numbers from in tabulated text format ranging from 0 to 1. I want to manipulate the list of numbers so that if the number is greater than 0.5 (> 0.5), I get 1 - number. If the number is less than 0.5, the number is taken as it is and not altered. For example: ... (1 Reply)
Discussion started by: evelibertine
1 Replies

3. Shell Programming and Scripting

copy contents of unix file to Word document

Hello, I have a unix file about 3000lines which i want to copy from and paste it into a Word document. If i cat the file and try to scroll through it then not everything is captured so i am getting and incomplete paste. Any help is really appreciated. jak (2 Replies)
Discussion started by: jakSun8
2 Replies

4. Shell Programming and Scripting

Remove specific word from data contents problem asking

Hi, Below is my input file: >tempZ_1 SAFSDAFDSG GGERRTTZZZ ZASRARARET WETPOASDAZ ZZZASASFAS >temp_9 ASAEGPIOEO EIOPIZZZAS FDFGZZZARA ESEAZZZAAS . . Desired output file: (9 Replies)
Discussion started by: patrick87
9 Replies

5. Shell Programming and Scripting

How to fetch rows based on line numbers or based on the beginning of a word?

I have a file which will have rows like shown below, ST*820*316054716 RMR*IV*11333331009*PO*40.31 REF*IV*22234441009*xsss471-2762 DTM*003*091016 ENT*000006 RMR*IV*2222234444*PO*239.91 REF*IV*1234445451009*LJhjlkhkj471-2762 </SPAN> DTM*003* 091016 RMR*IV*2223344441009*PO*40.31... (18 Replies)
Discussion started by: Muthuraj K
18 Replies

6. Shell Programming and Scripting

Renaming Files Based on Contents

Hello everyone, I currently have a situation which is causing me some issues for keeping up with certain files. I will explain this to the best of my abilities. I have a list of files as follows 50_REPORT_1111 - file contains the word Car 50_REPORT_2222 - file contains the word House... (15 Replies)
Discussion started by: DerangedNick
15 Replies

7. Shell Programming and Scripting

Merging files based on the contents

Hi, I have a file f1 having the contents as below select (<condn>) from dual I have another file f2 having the contents as below 1, 2, 3 I want to replace <condn> in f1 with the contents of f2 I tried using sed like this sed "s:<condn>:`cat f2`:g" f1 The above command resulted in sed:... (3 Replies)
Discussion started by: mr_manii
3 Replies

8. Shell Programming and Scripting

parsing a file and manipulating the contents

Hi I have a text file as follows BOB 14/14 TOM 94/94 SAM 3/3 CRIS 13/13 TOM 6/6 CRIS 27/27 SAM 2/2 JACK 25/25 CRIS (6 Replies)
Discussion started by: shellignorant
6 Replies

9. Shell Programming and Scripting

need help with finding a word in file's contents

Hi, I need to check if a particular name is already in the file or not and i am using following code for this... match=$(grep -n -e "$output1" outputfiles.txt ) where output1 is the variable name having names in it and outputfiles.txt is the file name ..and i am using ksh can anybosy... (6 Replies)
Discussion started by: manmeet
6 Replies

10. Shell Programming and Scripting

Using first word and print their contents using awk

suppose u hava a file G1354R tGGC-CGC D1361N cGAC-AAC I1424T ATC-ACC R768W gCGG-TGG Q1382R CAG-CGG Q178E gCAG-GAG Y181C TAC-TGC .........cont. So the question is By searching for first word i.e.character say R output shud be R768W gCGG-TGG R182P CGG-CCG R189W ... (6 Replies)
Discussion started by: cdfd123
6 Replies
Login or Register to Ask a Question