Visit Our UNIX and Linux User Community


ITERATION: remove row based on string value


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting ITERATION: remove row based on string value
# 1  
Old 09-02-2010
ITERATION: remove row based on string value

It is my first post, hoping to get help from the forum.

In a directory, I have 5000 multiple files that contains around 4000 rows with 10 columns in each file containing a unique string 'AT' located at 4th column.
Code:
OM   3328   O     BT   268       5.800      7.500      4.700      0.000     1.400
OM   3329   O     BT   723       8.500      8.900      3.600      8.500     1.400
OM   3330   O     AT   231       6.700      5.500      7.600      0.000     1.400
OM   3331   O     AT   234       1.200      7.700      5.500      8.500     1.400
OM   3332   O     AT   256       3.800      5.800      5.200      0.000     1.400

(Step-1)The bottom of the file needs entire few rows (only with string AT) to be removed ONLY if the 9th column is greater than a value of 0.10 . Then the kept rows in file shall be saved into a new file. An iteration command is required to do it on series of 5000 multiple files.

(Step-2)Next, a program 'calc' will be executed into this multiple new named files one by one. Again, if the 9th column is greater than value 0.10 (only for rows with string AT), then the corresponding row shall be removed from the file. Kept rows shall be renamed into new file.

I have written a short bash code below to execute the program 'calc' to series of multiple files in directory, and so far this small code for linux took me entire day to figure out because I dont have skill in writing any codes.

-------
Code:
#!/bin/sh
for d in $(\ls -d *.txt)
do
     ./calc  $d 
done

-------

(Step-3) Finally, every files that contains the same number of lines (ie, 3098, 3095, 3097 etc) shall be saved in single file, accordingly. In this case, from the original 5000 multiple files, the output file expected can be divided for example into:
Code:
3098 filename = containing all  files with 3098 lines
3095  filename = containing all files with 3095 lines
3097 filename =  containing all files with 3097 lines

Thank you so much for your time and attention.

-A

---------- Post updated at 02:08 PM ---------- Previous update was at 11:50 AM ----------

To tackle the problem in each step, first I need to remove matching lines by string and value.

In GNU/Linux x86_64:
Code:
awk  '($4 ~ /^AT$/){print}' newfile

The code above says that 4th column with matching string AT, will print into newfile. BUT, I need to tell the script that ONLY if the 9th column has value in between 0.00-0.10 ? How to do that in bash shell ?

Code:
> cat file
OM   3328   O     BT   268       5.800      7.500      4.700      0.000      1.400
OM   3329   O     BT   723       8.500      8.900      3.600      8.500      1.400
OM   3330   O     AT   231       6.700      5.500      7.600      0.000      1.400
OM   3331   O     AT   234       1.200      7.700      5.500      8.500      1.400
OM   3332   O     AT   256       3.800      5.800      5.200      0.100      1.400

Code:
> cat newfile
 OM   3330   O     AT   231       6.700      5.500      7.600      0.000      1.400
 OM   3332   O     AT   256       3.800      5.800      5.200      0.100      1.400

Please help.

-A

Last edited by Franklin52; 09-02-2010 at 03:17 AM.. Reason: Please use code tags, thank you!
# 2  
Old 09-02-2010
Quote:
Originally Posted by asanjuan
To tackle the problem in each step, first I need to remove matching lines by string and value.

In GNU/Linux x86_64:
Code:
awk  '($4 ~ /^AT$/){print}' newfile

The code above says that 4th column with matching string AT, will print into newfile. BUT, I need to tell the script that ONLY if the 9th column has value in between 0.00-0.10 ? How to do that in bash shell ?

Code:
> cat file
OM   3328   O     BT   268       5.800      7.500      4.700      0.000      1.400
OM   3329   O     BT   723       8.500      8.900      3.600      8.500      1.400
OM   3330   O     AT   231       6.700      5.500      7.600      0.000      1.400
OM   3331   O     AT   234       1.200      7.700      5.500      8.500      1.400
OM   3332   O     AT   256       3.800      5.800      5.200      0.100      1.400

Code:
> cat newfile
 OM   3330   O     AT   231       6.700      5.500      7.600      0.000      1.400
 OM   3332   O     AT   256       3.800      5.800      5.200      0.100      1.400

Please help.

-A
Try this:
Code:
awk '$4=="AT" && $9 <= 0.1 && $9 >= 0.0' file

This User Gave Thanks to Franklin52 For This Post:
# 3  
Old 09-02-2010
ITERATION: remove row based on string value

Make myself clear:

Code:
> cat file
OM   3328   O     BT   268       5.800      7.500      4.700      0.000      1.400
OM   3329   O     BT   723       8.500      8.900      3.600      8.500      1.400
OM   3330   O     AT   231       6.700      5.500      7.600      0.000      1.400
OM   3331   O     AT   234       1.200      7.700      5.500      8.500      1.400
OM   3332   O     AT   256       3.800      5.800      5.200      0.100      1.400

Code:
> cat newfile
OM   3328   O     BT   268       5.800      7.500      4.700      0.000      1.400
OM   3329   O     BT   723       8.500      8.900      3.600      8.500      1.400
OM   3330   O     AT   231       6.700      5.500      7.600      0.000      1.400
OM   3332   O     AT   256       3.800      5.800      5.200      0.100      1.400

Using awk the output newfile should ONLY delete lines with AT string (4th column) with value greater than 0.10 at (9th column).


Moderator's Comments:
Mod Comment Please use code tags, thank you


---------- Post updated at 03:09 PM ---------- Previous update was at 02:48 PM ----------

Thank you moderator for the kind reply.

awk '$4=="AT" && $9 <= 0.1 && $9 >= 0.0' file

Code above you gave prints line with string AT (4th column) and with value 0.0-0.10 (9th column).

Since one file contains 4000 lines with about 30 different strings at 4th column, it would be better that instead of above task, I would create a code (with help from forum) to just delete rows with AT string and >0.10 value (9th column). This way, rest of the lines with non "AT" string will kept inside the file.
# 4  
Old 09-02-2010
Quote:
Originally Posted by asanjuan
Make myself clear:

Code:
> cat file
OM   3328   O     BT   268       5.800      7.500      4.700      0.000      1.400
OM   3329   O     BT   723       8.500      8.900      3.600      8.500      1.400
OM   3330   O     AT   231       6.700      5.500      7.600      0.000      1.400
OM   3331   O     AT   234       1.200      7.700      5.500      8.500      1.400
OM   3332   O     AT   256       3.800      5.800      5.200      0.100      1.400

Code:
> cat newfile
OM   3328   O     BT   268       5.800      7.500      4.700      0.000      1.400
OM   3329   O     BT   723       8.500      8.900      3.600      8.500      1.400
OM   3330   O     AT   231       6.700      5.500      7.600      0.000      1.400
OM   3332   O     AT   256       3.800      5.800      5.200      0.100      1.400

Using awk the output newfile should ONLY delete lines with AT string (4th column) with value greater than 0.10 at (9th column).


Moderator's Comments:
Mod Comment Please use code tags, thank you


---------- Post updated at 03:09 PM ---------- Previous update was at 02:48 PM ----------

Thank you moderator for the kind reply.

awk '$4=="AT" && $9 <= 0.1 && $9 >= 0.0' file

Code above you gave prints line with string AT (4th column) and with value 0.0-0.10 (9th column).

Since one file contains 4000 lines with about 30 different strings at 4th column, it would be better that instead of above task, I would create a code (with help from forum) to just delete rows with AT string and >0.10 value (9th column). This way, rest of the lines with non "AT" string will kept inside the file.
Something like this?
Code:
awk '!($4=="AT" && $9 > 0.10)' file

# 5  
Old 09-02-2010
ITERATION: remove row based on string value

It work !!!! Step-1 is almost done;

I tried to create iteration loop:
---------------
Code:
#!/bin/bash
for d in $(\ls -d *.asa)
do
  awk '!($4=="AT" && $9 > 0.10)' > file
done

-----------------

Question is how to printout an output for each files in d ? The 'file' written above needs to be changed such that each output is written into separate files accordingly.

Thank you for your kind help.

Last edited by Franklin52; 09-02-2010 at 05:48 AM.. Reason: Please use code tags and indent your code
# 6  
Old 09-02-2010
just execute the same code on your files.so 1st and last step will be performed.
1.
Code:
 
#!/bin/ksh
for i in `ls *.txt`
do
nawk '!($4=="AT" && $9>0.10 )' $i > newfile_$i.txt
done

3.

Code:
 
#!/bin/ksh
wc -l newfile*|grep -v total >records.txt
nawk '{print $1}' records.txt|sort -u >uniq.txt
for i in `cat uniq.txt`
do 
 for j in `cat records.txt|grep "$i "| nawk '{print$2}'`
 do
 cat $j >> $i_output
 done
done

Try the above one

Last edited by malikshahid85; 09-02-2010 at 06:18 AM..
This User Gave Thanks to malikshahid85 For This Post:
# 7  
Old 09-02-2010
ITERATION: remove row based on string value

malikshahid85, Thanks so much for great help. Code-1 works well but Code-3 has shell issue or error:

line 9: $i_output: ambiguous redirect

I have tried searching in forum but not successful. Would you please further help me.

Thanks again. There are lot of wonderful people helping in this forum.

Previous Thread | Next Thread
Test Your Knowledge in Computers #684
Difficulty: Medium
According to both NetMarketShare and WikiMedia, Safari claimed a mobile market share of over 40% in 2019.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting single row into multiple rows based on for every 10 digits of last field of the row

Hi ALL, We have requirement in a file, i have multiple rows. Example below: Input file rows 01,1,102319,0,0,70,26,U,1,331,000000113200000011920000001212 01,1,102319,0,1,80,20,U,1,241,00000059420000006021 I need my output file should be as mentioned below. Last field should split for... (4 Replies)
Discussion started by: kotra
4 Replies

2. Shell Programming and Scripting

Trying to remove duplicates based on field and row

I am trying to see if I can use awk to remove duplicates from a file. This is the file: -==> Listvol <== deleting /vol/eng_rmd_0941 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_1006 deleting /vol/eng_rmd_1012 rearrange /vol/eng_rmd_0943 ... (6 Replies)
Discussion started by: newbie2010
6 Replies

3. Programming

String array iteration causing segfault

I am populating an array of string and print it. But it going in infinite loop and causing segfault. char Name = { "yahoo", "rediff", "facebook", NULL }; main(int argc, char* argv) { int j = 0; ... (7 Replies)
Discussion started by: rupeshkp728
7 Replies

4. UNIX for Dummies Questions & Answers

Remove lines in a positional file based on string value

Gurus, I am relatively new to Unix scripting and am struck with a problem in my script. I have positional input file which has a FLAG indicator in at position 11 in every record of the file. If the Flag has value =Y, then the record from the input needs to be written to a new file.However if... (3 Replies)
Discussion started by: gsam
3 Replies

5. Shell Programming and Scripting

Remove line based on string and put new line with parameter

Hi Folks, I am new to ksh, i have informatica parameter file that i need to update everyday with shell script. i need your help updating this file with new parameters. sample data $$TABLE1_DATE=04-27-2011 $$TABLE2_DATE=04-23-2011 $$TABLE3_DATE=03-19-2011 .......Highligned... (4 Replies)
Discussion started by: victor369
4 Replies

6. Shell Programming and Scripting

String manipulation row by row

Dear masters, I stuck again in a very tricky situation,so need your valuable inputs. I have a file having rows as below: _Db _Database 1023 1 1 1 17.0B 0.2 1.0 _Field _Field-Name 3 2 2 11 56.2K 64.1 ... (5 Replies)
Discussion started by: patric2326
5 Replies

7. Shell Programming and Scripting

remove characters from string based on occurrence of a string

Hello Folks.. I need your help .. here the example of my problem..i know its easy..i don't all the commands in unix to do this especiallly sed...here my string.. dwc2_dfg_ajja_dfhhj_vw_dec2_dfgh_dwq desired output is.. dwc2_dfg_ajja_dfhhj it's a simple task with tail... (5 Replies)
Discussion started by: victor369
5 Replies

8. UNIX for Dummies Questions & Answers

How to remove duplicated based on longest row & largest value in a column

Hii i have a file with data as shown below. Here i need to remove duplicates of the rows in such a way that it just checks for 2,3,4,5 column for duplicates.When deleting duplicates,retain largest row i.e with many columns with values should be selected.Then it must remove duplicates such that by... (11 Replies)
Discussion started by: reva
11 Replies

9. Shell Programming and Scripting

remove row if string is same as previous row

I have data like: Blue Apple 6 Red Apple 7 Yellow Apple 8 Green Banana 2 Purple Banana 8 Orange Pear 11 What I want to do is if $2 in a row is the same as $2 in the previous row remove that row. An identical $2 may exist more than one time. So the out file would look like: Blue... (4 Replies)
Discussion started by: dcfargo
4 Replies

10. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

Featured Tech Videos