Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to filter out data file...?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to filter out data file...?
# 1  
Old 12-08-2012
How to filter out data file...?
Hi...

I would like to filter out my data file....in two different way

1st way is like this, I will take one example..here...

The script should ask like this.

Code:
Enter min value in first column 

Enter max value in first column

Enter min value in second column

Enter max value in second column

Enter min value in third column

Enter max value in third column

After getting input from user end...data need to be filtered within the range specified..

Code:
suppose 1 column range is -5 to 5
                          2nd column range is 10 to 25
                          3rd column range is 15 to 35

output :
Code:


Code:



column 1   column 2  column 3    data   data   data






-5                10              15              12     225           25
-4                11              16              25     356           663
---------------------------------------------------------------------------------
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
 5                 25               35             255     2568     258

data should be within specified range..

Code:
2nd way

Enter first column value 

Enter second column value

Enter third column value 

Data should match the value given by user

suppose say I am giving 1st column value 10, 2nd column values as 25, third as 30

It should display like this

Code:
column1 column2 column 3 data        data
10             25           30          12            35 
10             25           30          25            69 
10             25           30          26            39

data_filter.txt (842 Bytes) 
# 2  
Old 12-08-2012
Hi, what have you tried so far and where are you stuck?
# 3  
Old 12-08-2012
Code:
c1_min=0
c1_max=0
c2_min=0
c2_max=0
c3_min=0
c3_max=0

echo -n "Enter column 1 min value > "
read c1_min
echo -n "Enter column 1 max value > "
read c1_max
echo -n "Enter column 2 min value > "
read c2_min
echo -n "Enter column 2 max value > "
read c2_max
echo -n "Enter column 3 min value > "
read c3_min
echo -n "Enter column 3 max value > "
read c3_max

for file in *.txt; do

I am beginner wants to learn googling..and trying...

---------- Post updated at 08:57 AM ---------- Previous update was at 04:00 AM ----------

Code:
how can I apply this algorithm for my data file in shell
1st way

if column1 <=c1_min and >=c1_max and 
if column2 <=c2_min and >=c2_max and
if column3 <=c3_min and >=c3_max 
display column data

2nd way
if column1=c1_x and
if column2=c2_x and
if column3=c3_x 
display column

# 4  
Old 12-09-2012
I think its impossible..in shell
# 5  
Old 12-09-2012
If you have ksh93 that has floating point arithmetic, so you can do something like this:
Code:
{
  read header
  printf "%s\n" "$header"
  while read col1 col2 col3 col4 col5; do
    if [ $c1_min -lt $col1 ] && [ $c1_max -gt $col1 ] && 
       [ $c2_min -lt $col2 ] && [ $c2_max -gt $col2 ] &&
       [ $c3_min -lt $col3 ] && [ $c3_max -gt $col3 ]
    then 
      printf "%s\t%s\t%s\t%s\t%s\n" $col1 $col2 $col3 $col4 $col5
    fi
  done 
} < data_filter.txt

If you are using ksh93 then there are better ways of writing this, but you get the idea...


In other shells you would need to convert the values to integer first or use an external program like for example bc

For example, using parameter expansion:
Code:
{
  read header
  printf "%s\n" "$header"
  while read col1 col2 col3 col4 col5; do
    if [ $c1_min -lt ${col1%%.*} ] && [ $c1_max -gt ${col1%%.*} ] && 
       [ $c2_min -lt ${col2%%.*} ] && [ $c2_max -gt ${col2%%.*} ] &&
       [ $c3_min -lt ${col3%%.*} ] && [ $c3_max -gt ${col3%%.*} ]
    then 
      printf "%s\t%s\t%s\t%s\t%s\n" $col1 $col2 $col3 $col4 $col5
    fi
  done 
} < data_filter.txt

Last edited by Scrutinizer; 12-09-2012 at 05:05 AM..
# 6  
Old 12-09-2012
Code:
echo -n "Enter column 1 min value > "
read c1_min
echo -n "Enter column 1 max value > "
read c1_max
echo -n "Enter column 2 min value > "
read c2_min
echo -n "Enter column 2 max value > "
read c2_max
echo -n "Enter column 3 min value > "
read c3_min
echo -n "Enter column 3 max value > "
read c3_max

{
  read header
  printf "%s\n" "$header"
  while read col1 col2 col3 col4 col5; do
    if [ $c1_min -lt $col1 ] && [ $c1_max -gt $col1 ] && 
       [ $c2_min -lt $col2 ] && [ $c2_max -gt $col2 ] &&
       [ $c3_min -lt $col3 ] && [ $c3_max -gt $col3 ]
    then 
      printf "%s\t%s\t%s\t%s\t%s\n" $col1 $col2 $col3 $col4 $col5
    fi
  done 
} < data_filter.txt

I tried like this...not working..

---------- Post updated at 04:24 AM ---------- Previous update was at 04:19 AM ----------

when I tried with ksh file.sh


Code:
column1	column2	column3	data1	data2
-0.018	64.008	4	28.18	36.43
-0.018	64.008	5	28.18	36.44
-0.018	64.008	6	28.19	36.45
-0.018	64.008	7	28.19	36.44
-0.018	64.008	8	28.94	34.97
data_filter.sh[30]: [: -1: unknown operator

I got this error

---------- Post updated at 04:36 AM ---------- Previous update was at 04:24 AM ----------

something wrong in if statement...
Last edited by Scott; 12-09-2012 at 06:09 AM.. Reason: Code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Shell script to read lines in a text file and filter user data Shell Programming and Scripting

sxsaaas (3 Replies)
Discussion started by: VikrantD
3 Replies

2. Shell Programming and Scripting

Filter a .kml file (xml) with data set from text file

I have a .kml file. So I want filter the .kml to get only the tags that have this numeric codes that they are in a text file 11951 11952 74014 11964 11965 11969 11970 11971 11972 60149 74018 74023 86378 11976 11980 11983 11984 11987 (5 Replies)
Discussion started by: pcoj33
5 Replies

3. UNIX for Dummies Questions & Answers

Need to filter data from a file

Hi, I have a file with hundreds of records. There are four fields on each line, separated by semicolons. Name Height (meters) Country Continent (Africa,Asia,Europe,North America,Oceania,South America,The Poles) I need to Write the command to find display how many mountains appear... (1 Reply)
Discussion started by: erora
1 Replies

4. Shell Programming and Scripting

How to filter required data from file using bash script?

Hi All , I have one file like below , Owner name = abu2-kxy-m29.hegen.app Item_id = AX1981, special_id = *NULL*, version = 1 Created = 09/01/2010 12:56:56 (1283389016) Enddate = 03/31/2011 00:00:00 (1301554800) From the above file I need to get the output in the below format ,i need... (3 Replies)
Discussion started by: gnanasekar_beem
3 Replies

5. UNIX for Advanced & Expert Users

filter last 24 hour data and put in new file

i have file server 1 (filesvr01acess.log) and disc server 1 (discsvr01acess.log) in unix box(say ip adress of the box 10.39.66.81) Similiarly i have file server 2 (filesvr01acess.log) and disc server 2(discsvr01acess.log) in another unix box(say ip adress of the box 10.39.66.82). Now my... (1 Reply)
Discussion started by: nripa1
1 Replies

6. Shell Programming and Scripting

Shell script to read lines in a text file and filter user data

hi all, I have this file with some user data. example: $cat myfile.txt FName|LName|Gender|Company|Branch|Bday|Salary|Age aaaa|bbbb|male|cccc|dddd|19900814|15000|20| eeee|asdg|male|gggg|ksgu|19911216||| aara|bdbm|male|kkkk|acke|19931018||23| asad|kfjg|male|kkkc|gkgg|19921213|14000|24|... (4 Replies)
Discussion started by: srimal
4 Replies

7. Shell Programming and Scripting

extract data from a data matrix with filter criteria

Here is what old matrix look like, IDs X1 X2 Y1 Y2 10914061 -0.364613333 -0.362922333 0.001691 -0.450094667 10855062 0.845956333 0.860396667 0.014440333 1.483899333... (7 Replies)
Discussion started by: ssshen
7 Replies

8. UNIX for Dummies Questions & Answers

Using input file to filter data from another file

I have a data file: abc Text Text Text Unique Text 123 Text word Line Unique Text fgh Text data Line Unique Text 789 Text Text Line Unique Text 543 Text Text Data Unique Text and a filter file 123 789 I want to extract out from the data file the two records that contain the keys... (1 Reply)
Discussion started by: tumblez
1 Replies

9. Windows & DOS: Issues & Discussions

Filter data from text file

Hi All We have got a text file, which has data dumped from 60 tables. From these 60 tables of data we need data from 4 tables only. I tried assigning line numbers to filter out data, but it is not working as intended. below is the sample file ----Table1----- 3,dfs,43,df 4,sd,5,edd... (18 Replies)
Discussion started by: b_sri
18 Replies

10. UNIX for Dummies Questions & Answers

filter data

Hi I have two files and their names are donotcall.txt, filter_it.txt. I want output.txt file with all data that is in filter_it.txt file but not in donotcall.txt. Can anybody help me to write 1-2 lines unix code for this. donotcall.txt contains following data. Each row represent phone... (3 Replies)
Discussion started by: deep.singh
3 Replies
Login or Register to Ask a Question