Shell Programming and Scripting

Hi,

I'd like to ask for some help with the following task, please:

there is a big file with a header (this is file.in):

what I need is to recognize a pattern in the header of this file (pattern is in another file) and delete the column with that header

for example, the file with the pattern looks like this (this is file.with.patterns)
so, it would recognize A_1 and will delete all the columns containing A_1; thus, the output would look like this (this is file.out):

I am not sure I've got the best approach. What I was thinking to do is to put all the columns whose header does not contain the specified pattern in one output file (so, those columns whose header does match the pattern will be let out, deleted):

one problem is that I would like to have all the columns whose header does not match the patterns in the file.with.patterns to be in the file.out and I am not sure if append sign (>>) would do that... it didn't really work well so far...


Another option I was thinking about is to establish the number of the columns whose header contains the pattern and then delete them with cut -f, but don't know how to do that.

Any ideas will be greatly appreciated!

Many thanks for your time!

From your post what i understood is you need a file which contains filtered content of header and rule for filter is given by a file.
let's say rule file is rule.txt (containing A_1 A_2 etc) and file with header is file.html

Code:

head -1 file.html >temp
 
for $pat `cat rule.txt`
grep -vPw '$pat\.[A-Z]' temp >temp1
cat temp1 >temp
done

try also:

Code:

awk '
NR==FNR {p[$0]=$0; next}
FNR==1 {for (i=1; i<=NF; i++) {s=$i; sub("[.].*","",s); if (p[s]) o[i]=s}}
{l=""; for (i=1; i<=NF; i++) if (!o[i]) l=l $i" "; $0=l;}
1
' file.with.patterns file.in > file.out

try

Code:

awk '
  FNR==NR{P[$1];next}
  FNR==1{
    for(i=1;i<=NF;i++) {
      c=$i
      sub(/\.[XYZ]$/,"",c)
      if(c in P)S[i]
    }
  }
  { a=x
    for(i=1;i<=NF;i++)
    if(!(i in S)) a=a " " $i;
    print substr(a,2)
  }' file.with.patterns file.in > file.out

Edit: Just a bit late with an awk solution, but this one doesn't put a space on the end of each output line.

Another awk:

Code:

awk '
  NR==FNR{
    A[$1]
    next
  }
  {
    s=$1
    for(i=2; i<=NF; i++){
      if(FNR==1) for(j in A) if($i~j) D[i]
      if( ! (i in D) ) s=s OFS $i
    }
    print s
  }
' file.with.patterns file.in

Thanks a lot for the scripts! They work perfectly on the example file. However, they do nothing for my big file. I am not sure, but may be it is the field separator? the big file's columns are separated by a space, not tab; could that affect the script?

Many thanks in advance!

Are there any carriage return characters in the 2 files? Post the output of:

Code:

head -2 file.with.patterns|od -bc

and

Code:

head -2 file.in|od -bc

.