Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Skipping rows based on columns

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Skipping rows based on columns
# 1  
Old 05-01-2012
Skipping rows based on columns
Hi,
suppose I have the following file and certain rows have missing columns,
how do i skip these rows and create an output file which has all the columns in it

Code:
E/N       Ko_exp   %err  Ko_calc  %err   diff  diff-  diff+  0.95
        ========  =======  ====  =======  ====  =====  =====  =====  ====
     1      4.00   2.8100   3.0   3.9502   0.5  -1.14  -1.31  -0.97    0
     2      8.00   2.8123   3.0   3.9668   0.5  -1.15  -1.32  -0.98    0
     3     12.00   2.8300   3.0   3.9920   0.5  -1.16  -1.33  -0.99    0
     4     16.00            3.0   4.0201   0.5  -1.18  -1.35  -1.00    0
     5     20.00   2.8700   3.0   4.0473   0.5  -1.18  -1.35  -1.00    0
     6     24.00   2.9007   3.0            0.5  -1.17  -1.34  -0.99    0
     7     28.00   2.9437   3.0   4.0807   0.5  -1.14  -1.31  -0.96    0
     8     32.00   2.9983   3.0   4.0833   0.5  -1.08  -1.27  -0.90    0
     9     36.00   3.0567   3.0   4.0778   0.5  -1.02  -1.21  -0.84    0
    10     40.00   3.1100   3.0   4.0656   0.5  -0.96  -1.14  -0.77    0


I want my output file to look like this:

Code:
E/N       Ko_exp   %err  Ko_calc  %err   diff  diff-  diff+  0.95
        ========  =======  ====  =======  ====  =====  =====  =====  ====
     1      4.00   2.8100   3.0   3.9502   0.5  -1.14  -1.31  -0.97    0
     2      8.00   2.8123   3.0   3.9668   0.5  -1.15  -1.32  -0.98    0
     3     12.00   2.8300   3.0   3.9920   0.5  -1.16  -1.33  -0.99    0
     5     20.00   2.8700   3.0   4.0473   0.5  -1.18  -1.35  -1.00    0
     7     28.00   2.9437   3.0   4.0807   0.5  -1.14  -1.31  -0.96    0
     8     32.00   2.9983   3.0   4.0833   0.5  -1.08  -1.27  -0.90    0
     9     36.00   3.0567   3.0   4.0778   0.5  -1.02  -1.21  -0.84    0
    10     40.00   3.1100   3.0   4.0656   0.5  -0.96  -1.14  -0.77    0

# 2  
Old 05-01-2012
Hi

Code:
$ awk 'NR<=2 || NF==10' file

Guru.
# 3  
Old 05-01-2012
With awk you can print the first two lines, and then only lines which have 10 columns:

Code:
awk 'NR<3||NF==10' input

Had to include the header separately, since it has only 9 columns.
# 4  
Old 05-02-2012
Thankyou Guru and neutronscott.
this is working but, I see that it is skipping the rows eventhough they have 5 fields in them...

see my examples below

Here is the first 10 lines of my input file:

Code:
Country  Postal  Admin4  StreetBaseName  StreetType
HUN      2243    Kóka    Dózsa György   út
HUN      5475    Csépa   4511
HUN      9600    Sárvár  Ady Endre      utca
HUN      8705    Somogyszentpál  Kossuth        utca
HUN      7098    Magyarkeszi     H?sök  tere
HUN      2483    Gárdony
HUN      5100    Jászberény
HUN      5100    Jászberény      Lehel vezér    tér
HUN      5811    Végegyháza      Széchenyi István       út

I have used the following code:

Code:
awk 'NR<2||NF==5' HUN1.dat >HUN2.dat

Here are the First 10 lines of my output file:

Code:
Country  Postal  Admin4  StreetBaseName  StreetType
HUN      8705    Somogyszentpál  Kossuth        utca
HUN      7098    Magyarkeszi     H?sök  tere
HUN      2310    Szigetszentmiklós       Losonczi       utca
HUN      7142    Pörböly         Óvoda  utca
HUN      4025    Debrecen        Barna  utca
HUN      2040    Budaörs         Farkasréti     utca
HUN      2040    Budaörs         Szabadság      út
HUN      9373    Pusztacsalád    Új     utca
HUN      4262    Nyíracsád       Rákóczi        utca

Line 1,3,9 and 10 are skipped even though they have 5 fields in them.
# 5  
Old 05-02-2012
Problem there is, what defines a field? Are those tabs? Because line 1 is 6 columns if you use space delimiter because of the space in "Dózsa György"

If they are tabs: awk -F'\t' 'NF==5'
If they are spaces: awk -F' *' 'NF==5'
That's 3 spaces before the asterisks, then each field is split by 2 or more spaces..
# 6  
Old 05-02-2012
Now the problem is back to square one...
I did try it with -F'\t'; now i see lines with four fields the fifth field is empty.

I have tried the folloiwng code
Code:
awk -F'\t' 'NR<2||NF==5' HUN1.dat >HUN4.dat

here are the first 10 lines from the result file
Code:
Country  Postal  Admin4  StreetBaseName  StreetType
HUN      2243    Kóka    Dózsa György   út
HUN      5475    Csépa   4511
HUN      9600    Sárvár  Ady Endre      utca
HUN      8705    Somogyszentpál  Kossuth        utca
HUN      7098    Magyarkeszi     H?sök  tere
HUN      2483    Gárdony
HUN      5100    Jászberény
HUN      5100    Jászberény      Lehel vezér    tér
HUN      5811    Végegyháza      Széchenyi István       út

# 7  
Old 05-02-2012
@ramky, this is because your data sample was not representative of your actual data.
Try:
Code:
awk 'NF>4' infile

but that will give false positives for streets consisting of two words and a missing street type, so you would need to manually remove records..
Or you can try to tinker with the -F value like neutronscott suggested..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Keep only columns in first two rows based on partial header pattern.

I have this code below that only prints out certain columns from the first two rows (doesn't affect rows 3 and beyond). How can I do the same on a partial header pattern “G_TP” instead of having to know specific column numbers (e.g. 374-479)? I've tried many other commands within this pipe with no... (4 Replies)
Discussion started by: aachave1
4 Replies

2. UNIX for Advanced & Expert Users

Conversion of rows to columns using awk based om column value

HI, My Input file data is dn:adcfgeneral id:13343 Name:xxxxxx Password:iutyerwuitywue wpuwt tuiytruityrutyrwtyrwp dn:cdferwjyyyy id:3875 Name:yyyy Password :hgfdsjkfhdsfkdlshf dshfkldshfdklsfh interset:uiuiufj My output should be ... (6 Replies)
Discussion started by: dineshaila
6 Replies

3. Shell Programming and Scripting

Match in awk skipping header rows

I am trying to match $1-$7 between the two files and if a match is found then the contents of $8 in file2 and copied over. The awk I tried is below. There is also a header row in file2 that has the Chr Start End Ref Alt that does not need to be searched. Thank you :). awk awk... (3 Replies)
Discussion started by: cmccabe
3 Replies

4. Shell Programming and Scripting

Convert rows to columns based on key and count

Team, I am having requirement to convert rows to columns Input is: key ,count, id1, pulse1, id2, pulse2 ,id3, pulse3 12, 2 , 14 , 56 , 15, 65 13, 3, 12, 32, 14, 23, 18, 54 22, 1 , 32, 42 Expected Out put: key, id,pulse 12, 14, 56 12, 15, 65 13 ,12, 32 13, 14 ,23 13, 18 ,54 22 ,32,... (3 Replies)
Discussion started by: syam1406
3 Replies

5. Shell Programming and Scripting

Convert rows to columns based on condition

I have a file some thing like this: GN Name=YWHAB; RC TISSUE=Keratinocyte; RC TISSUE=Thymus; CC -!- FUNCTION: Adapter protein implicated in the regulation of a large CC spectrum of both general and specialized signaling pathways GN Name=YWHAE; RC TISSUE=Liver; RC ... (13 Replies)
Discussion started by: raj_k
13 Replies

6. Shell Programming and Scripting

Extracting rows and columns in a matrix based on condition

Hi I have a matrix with n rows and m columns like below example. i want to extract all the pairs with values <200. Input A B C D A 100 206 51 300 B 206 100 72 48 C 351 22 100 198 D 13 989 150 100 Output format A,A:200 A,C:51 B,B:100... (2 Replies)
Discussion started by: anurupa777
2 Replies

7. UNIX for Dummies Questions & Answers

Extracting rows from a text file based on the values of two columns (given ranges)

Hi, I have a tab delimited text file with multiple columns. The second and third columns include numbers that have not been sorted. I want to extract rows where the second column includes a value between -0.01 and 0.01 (including both numbers) and the first third column includes a value between... (1 Reply)
Discussion started by: evelibertine
1 Replies

8. Shell Programming and Scripting

Selecting rows based on values in columns

Hi My pipe delimited .txt file contains rows with 10 columns. Can anyone advise how I output to file only those rows with the letters ‘ci' as the first 2 characters in the 3rd column ? Many thanks (4 Replies)
Discussion started by: malts18
4 Replies

9. Shell Programming and Scripting

Arrange output based on rows into columns

Hi All, I would like to ask help on how can i achieve below output. Inputfile: Oct11,apa1-daily,01:25:01 Oct11,apa2-daily,01:45:23 Oct12,apa1-daily,02:30:11 Oct12,apa2-daily,01:55:01 Oct13,apa1-off,01:43:34 Oct13,apa2-off,01:22:04 Desired output: Clients ... (3 Replies)
Discussion started by: mars101
3 Replies

10. Shell Programming and Scripting

Binning rows while skipping the first column

Hi I have a file that I want to bin. I am using this code: awk -F'\t' -v r=40 '{for(i=r;i<=NF;i+=r){for(j=0;j<r;j++){sum+=$(i-j)}printf "%s ", sum/r;sum=0}; printf "\n"}' file1 > file2 So basically what this code does is that it will averaging every 40 columns (creating bins of 40). But... (2 Replies)
Discussion started by: phil_heath
2 Replies
Login or Register to Ask a Question