It is my first post, hoping to get help from the forum.
In a directory, I have 5000 multiple files that contains around 4000 rows with 10 columns in each file containing a unique string 'AT' located at 4th column.
Code:
OM 3328 O BT 268 5.800 7.500 4.700 0.000 1.400
OM 3329 O BT 723 8.500 8.900 3.600 8.500 1.400
OM 3330 O AT 231 6.700 5.500 7.600 0.000 1.400
OM 3331 O AT 234 1.200 7.700 5.500 8.500 1.400
OM 3332 O AT 256 3.800 5.800 5.200 0.000 1.400
(Step-1)The bottom of the file needs entire few rows (only with string AT) to be removed ONLY if the 9th column is greater than a value of 0.10 . Then the kept rows in file shall be saved into a new file. An iteration command is required to do it on series of 5000 multiple files.
(Step-2)Next, a program 'calc' will be executed into this multiple new named files one by one. Again, if the 9th column is greater than value 0.10 (only for rows with string AT), then the corresponding row shall be removed from the file. Kept rows shall be renamed into new file.
I have written a short bash code below to execute the program 'calc' to series of multiple files in directory, and so far this small code for linux took me entire day to figure out because I dont have skill in writing any codes.
-------
Code:
#!/bin/sh
for d in $(\ls -d *.txt)
do
./calc $d
done
-------
(Step-3) Finally, every files that contains the same number of lines (ie, 3098, 3095, 3097 etc) shall be saved in single file, accordingly. In this case, from the original 5000 multiple files, the output file expected can be divided for example into:
Code:
3098 filename = containing all files with 3098 lines
3095 filename = containing all files with 3095 lines
3097 filename = containing all files with 3097 lines
Thank you so much for your time and attention.
-A
---------- Post updated at 02:08 PM ---------- Previous update was at 11:50 AM ----------
To tackle the problem in each step, first I need to remove matching lines by string and value.
In GNU/Linux x86_64:
Code:
awk '($4 ~ /^AT$/){print}' newfile
The code above says that 4th column with matching string AT, will print into newfile. BUT, I need to tell the script that ONLY if the 9th column has value in between 0.00-0.10 ? How to do that in bash shell ?
Code:
> cat file
OM 3328 O BT 268 5.800 7.500 4.700 0.000 1.400
OM 3329 O BT 723 8.500 8.900 3.600 8.500 1.400
OM 3330 O AT 231 6.700 5.500 7.600 0.000 1.400
OM 3331 O AT 234 1.200 7.700 5.500 8.500 1.400
OM 3332 O AT 256 3.800 5.800 5.200 0.100 1.400
Code:
> cat newfile
OM 3330 O AT 231 6.700 5.500 7.600 0.000 1.400
OM 3332 O AT 256 3.800 5.800 5.200 0.100 1.400
Please help.
-A
Last edited by Franklin52; 09-02-2010 at 03:17 AM..
Reason: Please use code tags, thank you!
To tackle the problem in each step, first I need to remove matching lines by string and value.
In GNU/Linux x86_64:
Code:
awk '($4 ~ /^AT$/){print}' newfile
The code above says that 4th column with matching string AT, will print into newfile. BUT, I need to tell the script that ONLY if the 9th column has value in between 0.00-0.10 ? How to do that in bash shell ?
Code:
> cat file
OM 3328 O BT 268 5.800 7.500 4.700 0.000 1.400
OM 3329 O BT 723 8.500 8.900 3.600 8.500 1.400
OM 3330 O AT 231 6.700 5.500 7.600 0.000 1.400
OM 3331 O AT 234 1.200 7.700 5.500 8.500 1.400
OM 3332 O AT 256 3.800 5.800 5.200 0.100 1.400
Code:
> cat newfile
OM 3330 O AT 231 6.700 5.500 7.600 0.000 1.400
OM 3332 O AT 256 3.800 5.800 5.200 0.100 1.400
Please help.
-A
Try this:
Code:
awk '$4=="AT" && $9 <= 0.1 && $9 >= 0.0' file
This User Gave Thanks to Franklin52 For This Post:
> cat file
OM 3328 O BT 268 5.800 7.500 4.700 0.000 1.400
OM 3329 O BT 723 8.500 8.900 3.600 8.500 1.400
OM 3330 O AT 231 6.700 5.500 7.600 0.000 1.400
OM 3331 O AT 234 1.200 7.700 5.500 8.500 1.400
OM 3332 O AT 256 3.800 5.800 5.200 0.100 1.400
Code:
> cat newfile
OM 3328 O BT 268 5.800 7.500 4.700 0.000 1.400
OM 3329 O BT 723 8.500 8.900 3.600 8.500 1.400
OM 3330 O AT 231 6.700 5.500 7.600 0.000 1.400
OM 3332 O AT 256 3.800 5.800 5.200 0.100 1.400
Using awk the output newfile should ONLY delete lines with AT string (4th column) with value greater than 0.10 at (9th column).
Moderator's Comments:
Please use code tags, thank you
---------- Post updated at 03:09 PM ---------- Previous update was at 02:48 PM ----------
Thank you moderator for the kind reply.
awk '$4=="AT" && $9 <= 0.1 && $9 >= 0.0' file
Code above you gave prints line with string AT (4th column) and with value 0.0-0.10 (9th column).
Since one file contains 4000 lines with about 30 different strings at 4th column, it would be better that instead of above task, I would create a code (with help from forum) to just delete rows with AT string and >0.10 value (9th column). This way, rest of the lines with non "AT" string will kept inside the file.
> cat file
OM 3328 O BT 268 5.800 7.500 4.700 0.000 1.400
OM 3329 O BT 723 8.500 8.900 3.600 8.500 1.400
OM 3330 O AT 231 6.700 5.500 7.600 0.000 1.400
OM 3331 O AT 234 1.200 7.700 5.500 8.500 1.400
OM 3332 O AT 256 3.800 5.800 5.200 0.100 1.400
Code:
> cat newfile
OM 3328 O BT 268 5.800 7.500 4.700 0.000 1.400
OM 3329 O BT 723 8.500 8.900 3.600 8.500 1.400
OM 3330 O AT 231 6.700 5.500 7.600 0.000 1.400
OM 3332 O AT 256 3.800 5.800 5.200 0.100 1.400
Using awk the output newfile should ONLY delete lines with AT string (4th column) with value greater than 0.10 at (9th column).
Moderator's Comments:
Please use code tags, thank you
---------- Post updated at 03:09 PM ---------- Previous update was at 02:48 PM ----------
Thank you moderator for the kind reply.
awk '$4=="AT" && $9 <= 0.1 && $9 >= 0.0' file
Code above you gave prints line with string AT (4th column) and with value 0.0-0.10 (9th column).
Since one file contains 4000 lines with about 30 different strings at 4th column, it would be better that instead of above task, I would create a code (with help from forum) to just delete rows with AT string and >0.10 value (9th column). This way, rest of the lines with non "AT" string will kept inside the file.
#!/bin/bash
for d in $(\ls -d *.asa)
do
awk '!($4=="AT" && $9 > 0.10)' > file
done
-----------------
Question is how to printout an output for each files in d ? The 'file' written above needs to be changed such that each output is written into separate files accordingly.
Thank you for your kind help.
Last edited by Franklin52; 09-02-2010 at 05:48 AM..
Reason: Please use code tags and indent your code
just execute the same code on your files.so 1st and last step will be performed.
1.
Code:
#!/bin/ksh
for i in `ls *.txt`
do
nawk '!($4=="AT" && $9>0.10 )' $i > newfile_$i.txt
done
3.
Code:
#!/bin/ksh
wc -l newfile*|grep -v total >records.txt
nawk '{print $1}' records.txt|sort -u >uniq.txt
for i in `cat uniq.txt`
do
for j in `cat records.txt|grep "$i "| nawk '{print$2}'`
do
cat $j >> $i_output
done
done
Try the above one
Last edited by malikshahid85; 09-02-2010 at 06:18 AM..
This User Gave Thanks to malikshahid85 For This Post:
Hi ALL,
We have requirement in a file, i have multiple rows.
Example below:
Input file rows
01,1,102319,0,0,70,26,U,1,331,000000113200000011920000001212
01,1,102319,0,1,80,20,U,1,241,00000059420000006021
I need my output file should be as mentioned below. Last field should split for... (4 Replies)
I am trying to see if I can use awk to remove duplicates from a file. This is the file:
-==> Listvol <==
deleting /vol/eng_rmd_0941
deleting /vol/eng_rmd_0943
deleting /vol/eng_rmd_0943
deleting /vol/eng_rmd_1006
deleting /vol/eng_rmd_1012
rearrange /vol/eng_rmd_0943
... (6 Replies)
I am populating an array of string and print it.
But it going in infinite loop and causing segfault.
char Name = {
"yahoo",
"rediff",
"facebook",
NULL
};
main(int argc, char* argv)
{
int j = 0;
... (7 Replies)
Gurus,
I am relatively new to Unix scripting and am struck with a problem in my script. I have positional input file which has a FLAG indicator in at position 11 in every record of the file.
If the Flag has value =Y, then the record from the input needs to be written to a new file.However if... (3 Replies)
Hi Folks,
I am new to ksh, i have informatica parameter file that i need to update everyday with shell script. i need your help updating this file with new parameters.
sample data
$$TABLE1_DATE=04-27-2011
$$TABLE2_DATE=04-23-2011
$$TABLE3_DATE=03-19-2011
.......Highligned... (4 Replies)
Dear masters,
I stuck again in a very tricky situation,so need your valuable inputs. I have a file having rows as below:
_Db
_Database 1023 1 1 1 17.0B 0.2 1.0
_Field
_Field-Name 3 2 2 11 56.2K 64.1 ... (5 Replies)
Hello Folks..
I need your help ..
here the example of my problem..i know its easy..i don't all the commands in unix to do this especiallly sed...here my string..
dwc2_dfg_ajja_dfhhj_vw_dec2_dfgh_dwq
desired output is..
dwc2_dfg_ajja_dfhhj
it's a simple task with tail... (5 Replies)
Hii i have a file with data as shown below. Here i need to remove duplicates of the rows in such a way that
it just checks for 2,3,4,5 column for duplicates.When deleting duplicates,retain largest row i.e with many columns with values should be selected.Then it must remove duplicates such that by... (11 Replies)
I have data like:
Blue Apple 6
Red Apple 7
Yellow Apple 8
Green Banana 2
Purple Banana 8
Orange Pear 11
What I want to do is if $2 in a row is the same as $2 in the previous row remove that row. An identical $2 may exist more than one time.
So the out file would look like:
Blue... (4 Replies)
Hi
I have been struggling with a script for removing duplicate messages from a shared mailbox.
I would like to search for duplicate messages based on the “Message-ID” string within the messages files.
I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
09-02-2010
