Solaris - Filter columns in text file and adding new column


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Solaris - Filter columns in text file and adding new column
# 1  
Old 09-19-2013
Solaris - Filter columns in text file and adding new column

Hello,
I am very now to this, hope you can help,
I am looking into editing a file in Solaris, with dinamic collums (lenght varies) and I need 2 things to be made, the fist is to filter the first column and third column from the file bellow file.txt, and create a new file with the 2 filtered collums, plus a third column with a value (0, 1 or 2) depending on the values of the second collum (0 for values 0-10, 1 for values 11-100, and 2 for values above 100)
EXAMPLE:
ORIGINAL FILE:
Code:
EH.ERROR.ADB_INSERT --------- 5* 0 11 0.0 Kb
EH.ERROR.ADB_UPDATE --------- 5* 0 0 0.0 Kb
ERP.SAP-PI.bpmMensagemSMS.EH ---+----- 5* 0 234 0.0 Kb
ERRORHANDLER.UI.CFG.POLICY.REQUEST ---+----- 5* 0 1000 0.0 Kb

RESULT FILE:
Code:
EH.ERROR.ADB_INSERT 11 1
EH.ERROR.ADB_UPDATE 0 0
ERP.SAP-PI.bpmMensagemSMS.EH 234 2
ERRORHANDLER.UI.CFG.POLICY.REQUEST 1000 2

NOTE: Collumns are alligned in original file, here in fórum they got out of place

---------- Post updated at 12:02 PM ---------- Previous update was at 11:59 AM ----------

As preiously stated, in original file collumns are alligned.
In bold the values to filter:
EH.ERROR.ADB_INSERT --------- 5* 0 11 0.0 Kb


Last edited by vbe; 09-19-2013 at 02:14 PM..
# 2  
Old 09-19-2013
columns format and indetation in code are kept if you use (and are asked to...) code tags!
Code tags are to be use for code AND data...
This User Gave Thanks to vbe For This Post:
# 3  
Old 09-19-2013
Thank you vbe!

Code:
ORIGINAL FILE:
EH.ERROR.ADB_INSERT                    ---------    5*     0       11     0.0 Kb
EH.ERROR.ADB_UPDATE                    ---------    5*     0        0     0.0 Kb
ERP.SAP-PI.bpmMensagemSMS.EH           ---+-----    5*     0      234     0.0 Kb
ERRORHANDLER.UI.CFG.POLICY.REQUEST     ---+-----    5*     0     1000     0.0 Kb

RESULT FILE:
EH.ERROR.ADB_INSERT                      11     1
EH.ERROR.ADB_UPDATE                       0     0
ERP.SAP-PI.bpmMensagemSMS.EH            234     2
ERRORHANDLER.UI.CFG.POLICY.REQUEST     1000     2

# 4  
Old 09-19-2013
Code:
awk '($5>=0&&$5<=10){a=0}($5>=11&&$5<=100){a=1}($5>100){a=2}{printf("%-40s %-10s %-10s\n",$1,$5,a)}' file

This User Gave Thanks to in2nix4life For This Post:
# 5  
Old 10-02-2013
Thank you very much for the reply, it has helped me a lot!!!

However I need an extra upgrade I guess shouldn´t be too complicated to implement:

The awk command gave me the result I needed, however I will need to change thresholds depending on the domain (eg: EH, ERP, etc), and above that add 2 extra columns with the min and max thresholds that are set for that row.

NOW:
Code:
 
awk '($5>=0&&$5<=10){a=0}($5>=11&&$5<=100){a=1}($5>100){a=2}{printf("%-40s %-10s %-10s\n",$1,$5,a)}' file
 
RESULT FILE:
EH.ERROR.ADB_INSERT                      11     1
EH.ERROR.ADB_UPDATE                       0     0
ERP.SAP-PI.bpmMensagemSMS.EH            234     2
ERRORHANDLER.UI.CFG.POLICY.REQUEST     1000     2
 
 
Now what I am trying to do is some if condition to distinguish the thresholds by domain, and add the 2 columns with the set low and max thresholds, like (example EH with lower thresholds and ERP with higher)
 
ORIGINAL FILE:
 
EH.ERROR.ADB_INSERT                    ---------    5*     0       11     0.0 Kb
EH.ERROR.ADB_UPDATE                    ---------    5*     0        5     0.0 Kb
ERP.SAP-PI.bpmMensagemSMS.EH           ---+-----    5*     0      234     0.0 Kb
ERRORHANDLER.UI.CFG.POLICY.REQUEST     ---+-----    5*     0     1000     0.0 Kb
 
if EH do  awk '($5>=0&&$5<=10){a=0}($5>=10&&$5<=20){a=1}($5>20){a=2}{printf("%-40s %-10s %-10s\n",$1,$5,a)}' 
if ERP do  awk '($5>=0&&$5<=300){a=0}($5>=301&&$5<=500){a=1}($5>500){a=2}{printf("%-40s %-10s %-10s\n",$1,$5,a)}' 
else do  awk '($5>=0&&$5<=10){a=0}($5>=11&&$5<=100){a=1}($5>100){a=2}{printf("%-40s %-10s %-10s\n",$1,$5,a)}' 
file
 
RESULTS:
EH.ERROR.ADB_INSERT                      11     2   10  20
EH.ERROR.ADB_UPDATE                       0     2   10  20
ERP.SAP-PI.bpmMensagemSMS.EH            234     0    300   500
ERRORHANDLER.UI.CFG.POLICY.REQUEST     1000     2     10   100

Any help is very much appreciatted!! Thank you in advance for your time! Smilie
# 6  
Old 10-02-2013
Being given requirements little bits at a time, instead of giving us all of the requirements up front wastes our time and is likely to introduce significant delays in getting the results you want...

You explicitly state that the lower bound for range 1 is 0 and in2nix4life's script assumes that $5 will never contain a negative number. (If a negative number is given, in2nix4life's script will print the range value for the previous input line.)

You said that field widths will vary but in2nix4life's script assumed upper bounds of 40 characters for column 1 and 10 characters for column 5.

The following script seems to meet your updated requirements (and it should be clear how to add additional domains with different sets of ranges). This script reads the file twice; the 1st pass gets the maximum field widths present in your input, and the 2nd pass prints the results. If column 5's value is less than zero, this script will print -1 for the range code. You can ignore that test if you know no input will ever have a negative value, or you can do something else where that test occurs if you want to print an error message, just remove that line's data from the output, or take some other action.

Code:
/usr/xpg4/bin/awk '
BEGIN {#ERE to match domain     Range 0 max     Range 1 max
        dom[0] = "^EH\.";       rm[0,0] = 10;   rm[0,1] = 20     
        dom[1] = "^ERP\.";      rm[1,0] = 300;  rm[1,1] = 500     
        dom[2] = ".*";          rm[2,0] = 10;   rm[2,1] = 100       
        ndom=3
        # determine # of columns needed to print range values...  
        for(i = 0; i < ndom; i++) {    
                if(length(rm[i,0]) > mr0) mr0 = length(rm[i,0])  
                if(length(rm[i,1]) > mr1) mr1 = length(rm[i,1])   
        }
}
FNR == NR {
        # 1st pass through the file; get max lengths of fields 1 and 5    
        if(length($1) > m1) m1 = length($1)
        if(length($5) > m5) m5 = length($5)
        next
}
FNR == 1 {
        # 1st line in 2nd pass; create format string to be used...  
        fmt = sprintf("%%-%ds%%%dd%%3d%%%dd%%%dd\n",
                m1, m5 + 1, mr0 + 1, mr1 + 1)
}
{       # 2nd pass; print the results...
        for(i = 0; i < ndom; i++) {
                if($1 ~ dom[i]) {
                        r = ($5 > rm[i,0]) + ($5 > rm[i,1])
                        break
                }        }
        if($5 < 0) r = -1
        printf(fmt, $1, $5, r, rm[i,0], rm[i,1])
}' file file

Since you said you're doing this on a Solaris system, I specified /usr/xpg4/bin/awk (instead of the default /bin/awk or /usr/bin/awk) as the utility to use to run this awk script. If you want to run it on another system, change the first line of the script from:
Code:
/usr/xpg4/bin/awk '

to just:
Code:
awk '

This User Gave Thanks to Don Cragun For This Post:
# 7  
Old 10-03-2013
Don thank you very very much for your help, it is working nearly to perfection

My appologies if I don´t give all info at once, I am quite newbie to this as I mention, and just learning as I go along.....

Your solution works perfect, the only thing I am not very clear is how I can add a specific range for a specific line, is that possible?

Say I would only want to change thresholds for EH.ERROR.ADB_INSERT but not EH.ERROR.ADB_UPDATE, I tried this but does not work Smilie

Code:
 
EH.ERROR.ADB_INSERT --------- 5* 0 11 0.0 Kb
EH.ERROR.ADB_UPDATE --------- 5* 0 0 0.0 Kb
ERP.SAP-PI.bpmMensagemSMS.EH ---+----- 5* 0 234 0.0 Kb
ERRORHANDLER.UI.CFG.POLICY.REQUEST ---+----- 5* 0 1000 0.0 Kb


I tried this but did not work....

Code:
BEGIN {#ERE to match domain            Range 0 max        Range 1 max
        dom[0] = ".*";                 rm[0,0] = 1000;    rm[0,1] = 2000     
        dom[1] = "^EH.ERROR.ADB_INSERT\.";        rm[1,0] = 15000;      rm[1,1] = 20000     
        dom[2] = "^ERP\.";       rm[2,0] = 10000;      rm[2,1] = 20000
        ndom=3
        # determine # of columns needed to print range values...  
        for(i = 0; i < ndom; i++) {    
                if(length(rm[i,0]) > mr0) mr0 = length(rm[i,0])  
                if(length(rm[i,1]) > mr1) mr1 = length(rm[i,1])   
        }


Again, thank you very much for your time and patience with this newbie :P
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help in adding text before columns in shell script

Hello, Can someone please help in below requirement. My requirement is to add date before to first column,some text before 1st,2nd coulmns and insert a new column in between 2 and 3 columns. input file. aa 123 dddd aa 667 kdkdk ddj 738 kkkk aa 123 dddd aa 667 ... (5 Replies)
Discussion started by: Cva2568
5 Replies

2. UNIX for Dummies Questions & Answers

Filter records in a huge text file from a filter text file

Hi Folks, I have a text file with lots of rows with duplicates in the first column, i want to filter out records based on filter columns in a different filter text file. bash scripting is what i need. Data.txt Name OrderID Quantity Sam 123 300 Jay 342 498 Kev 78 2500 Sam 420 50 Vic 10... (3 Replies)
Discussion started by: tech_frk
3 Replies

3. Shell Programming and Scripting

Reading columns from a text file and to make an array for each column

Hi, I am not so familiar with bash scripting and would appreciate your help here. I have a text file 'input.txt' like this: 2 3 4 5 6 7 8 9 10 I want to store each column in an array like this a ={2 5 8}, b={3 6 9}, c={4 7 10} so that i can access any element, e.g b=6 for the later use. (1 Reply)
Discussion started by: Asif Siddique
1 Replies

4. UNIX for Dummies Questions & Answers

Adding a column to a text file with row numbers

Hi, I would like to add a new column containing the row numbers to a text file. How do I go about doing that? Thanks! Example input: A X B Y C D Output: A X 1 B Y 2 C D 3 (5 Replies)
Discussion started by: evelibertine
5 Replies

5. UNIX for Dummies Questions & Answers

Adding a column to a text file based on mathematical manipulation

Hi, I have a tab delimited text file with three different columns. I want to add an extra column to the text file. The extra column will be the second column and it will equal third column - 1. How do I go about doing that? Thanks! Input: chr1 788822 rs11240777 chr1 1008567 rs9442372... (2 Replies)
Discussion started by: evelibertine
2 Replies

6. UNIX for Dummies Questions & Answers

Adding tags to a specific column of a space delimited text file

I have a space delimited text file with two columns. I would like to add NA to the first column of the text file. Input: 19625 10.4791768259 19700 10.8146489183 19701 10.9084026759 19702 10.9861346978 19703 10.9304364984 Output: NA19625 10.4791768259 NA19700 10.8146489183... (1 Reply)
Discussion started by: evelibertine
1 Replies

7. Shell Programming and Scripting

Adding a new column in a file with other existing columns

Hi All , Kindly help me with this soln awk '{printf "%s %7s \n", $1,$c}' infile where value of variable c I am externally giving input But executing the above command shows all the columns of infile where as I want only 1st column of infile and 2nd column should print value c (8 Replies)
Discussion started by: Pratik4891
8 Replies

8. Shell Programming and Scripting

Suggestions for adding columns to text file

Good afternoon to everyone, I have some input and output from various widgets that I am trying to get to play nicely together. Basically I would like to stay out of excel and be able to automate the entire process. I have read some posts here about how to use awk, nawk, etc, to do similar... (9 Replies)
Discussion started by: LMHmedchem
9 Replies

9. Shell Programming and Scripting

Adding a column to a text based on file name

Dear all, Does anyone know how I could to add a column of numbers (1s, or 2s, or..., or 6s) to two-column text files (tab-delimited), where the specific number to be added varies as a function of the file naming? Currently, each of my text files has two columns, so the column with the... (12 Replies)
Discussion started by: rlapate
12 Replies

10. Shell Programming and Scripting

Changing the column for a row in a text file and adding another row

Hi, I want to write a shell script which increments a particular column in a row from a text file and then adds another row below the current row with the incremented value . For Eg . if the input file has a row : abc xyz lmn 89 lm nk o p I would like the script to create something like... (9 Replies)
Discussion started by: aYankeeFan
9 Replies
Login or Register to Ask a Question