Sponsored Content
Top Forums Shell Programming and Scripting Deleting columns by list or file Post 302376553 by jsmithstl on Tuesday 1st of December 2009 05:14:00 PM
Old 12-01-2009
This is very similar to a post the other day, except that person wanted to keep the columns identified in the column file.

This should work for you:

I created two test files. The first, column.dat, with 21 lines containing 300,000 columns of data delimited by spaces with a header line consisting of column headers:

column.dat
Code:
col_1 col_2 col_3 ... col_299998 col_299999 col_300000

1_1 1_2 1_3 ... 1_299998 col_299999 col_300000
2_1 2_2 2_3 ... 2_299998 col_299999 col_300000
...
...
20_1 20_2 20_3 ... 20_299998 20_299999 20_300000

The second, column.lst, contains the list of columns you want to delete, starting with col_300 and incrementing by 10 up to col_200290:
Code:
col_300
col_310
col_320
...
...
col_200270
col_200280
col_200290

The code is dependent on the columns in column.lst and the column headers in column.dat are in the same order.

Here's the perl code:
Code:
#!/usr/bin/perl

use strict;

my @a_column;
my @a_delcol;
my @a_allcol;
my @a_outcol;
my $datcol;
my $outline;
my $line;
my $date_stamp;
my $i;
my $d;

$date_stamp = localtime time;
print "START:  $date_stamp\n";

open COLFILE, "<column.lst"
  or die "can't open file: $!";

$i=0;
@a_delcol = <COLFILE>;

close COLFILE
  or die "can't close file: $!";

open DATFILE, "<column.dat"
  or die "can't open file: $!";

$datcol = <DATFILE>;
chomp ($datcol);

close DATFILE
  or die "can't close file: $!";

@a_allcol = (split ' ', $datcol);

$i=0;
$d=0;

while ( $a_allcol[$i] )
{
   chomp ($a_allcol[$i]);
   chomp ($a_delcol[$d]);

   if ( $a_allcol[$i] ne $a_delcol[$d] )
   {
      push (@a_column, $i);
      push (@a_outcol, "$a_allcol[$i]");
   }
   else
   {
      $d++;
   }
   $i++;
}

undef @a_allcol;
undef @a_delcol;

open DATFILE, "<column.dat"
  or die "can't open file: $!";

open OUTFILE, ">column.out"
  or die "can't open file: $!";

$outline = join(" ", @a_outcol);
print OUTFILE "$outline\n";

undef @a_outcol;

while($line = <DATFILE>)
{
   chomp($line);
   $outline = join(" ", (split ' ', $line) [@a_column]);
   print OUTFILE "$outline\n";
}

close DATFILE
  or die "can't close file: $!";

close OUTFILE
  or die "can't close file: $!";

$date_stamp = localtime time;
print "END:  $date_stamp\n";


Here's the timing of the script:
Code:
START:  Tue Dec  1 15:00:40 2009
    END:  Tue Dec  1 15:00:50 2009


and the output:
Code:
col_1 col_2 col_3 ... col_299 col_301 ... col_599 col_601 ... col_299999 col_300000
1_1 1_2 1_3 ... 1_299 1_301 ... 1_599 1_601 ... 1_299999 1_300000
...
...
20_1 20_2 20_3 ... 20_299 20_301 ... 20_599 20_601 ... 20_299999 20_300000

I would expect this would take about 12 to 15 minutes to complete depending on your system resources.

Good luck.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Deleting specific columns from a file

Hi Friends, I want to delete specific columns from a file. Say my file content is as follows: "1","a","ww1",1234" "2","b","wwr3","2222" "3","c","erre","3333" Now i want to delete the column 2 and 4 from this file. That is I want the file content to be: "1","ww1" "2","wwr3"... (11 Replies)
Discussion started by: premar
11 Replies

2. Shell Programming and Scripting

deleting rows & columns form a csv file

Hi , I want to delete some rows & columns from file. can someone please help me on this? Regards. (2 Replies)
Discussion started by: code19
2 Replies

3. Shell Programming and Scripting

Deleting columns from CSV file

Hi All, Am working on perl script which should delete columns in existing CSV file. If my file is : AA,BB,CC,DD 00,11,22,33 00,55,66,77 00,99,88,21 AA,BB... are all my headers can come in any order (e.g AA,CC,BB...) and rest are values. I want to delete column CC... Can anybody help... (2 Replies)
Discussion started by: darshakraut
2 Replies

4. Shell Programming and Scripting

Print columns in a file by number list in another file

This should follow with my last post but I think it's better to start a new one. Now I have a list of numbers stored in pos.txt: 2 6 7 . . . n . . . And I need to extract (2n-1, 2n) columns from matrix.txt: ind1 A B C D E F G H I J K L M N O P Q R ... ind2 B C D E F G H... (3 Replies)
Discussion started by: Zoho
3 Replies

5. UNIX for Dummies Questions & Answers

Deleting columns from a space delimited text file

I have a space delimited text file with 1,000,000+ columns and 100 rows. I want to delete columns 2 through 5 (2 and 5) included from the text file. How do I do that? Thanks. (3 Replies)
Discussion started by: evelibertine
3 Replies

6. UNIX for Dummies Questions & Answers

Deleting columns from a tab delimited text file?

I have a tab limited text file with 10000+ columns. I want to delete columns 6 through 23, how do I go about doing that? Thanks! (1 Reply)
Discussion started by: evelibertine
1 Replies

7. UNIX for Advanced & Expert Users

Help in Deleting columns and Renaming Mutliple columns in a .Csv File

Hi All, i have a .Csv file in the below format startTime, endTime, delta, gName, rName, rNumber, m2239max, m2239min, m2239avg, m100016509avg, m100019240max, metric3min, m100019240avg, propValues 11-Mar-2012 00:00:00, 11-Mar-2012 00:05:00, 300.0, vma3550a, a-1_CPU Index<1>, 200237463, 0.0,... (9 Replies)
Discussion started by: mahi_mayu069
9 Replies

8. Shell Programming and Scripting

Deleting specific columns

Hi group, Can you please tell how to delete specific columns from a file. I know something like awk -F, '{ print $1" "$2" "15 }' input.txt > output.txt will delete all other columns. But this is in a way to copy some particular columns. But is there any other way to select just some... (11 Replies)
Discussion started by: smitra
11 Replies

9. Shell Programming and Scripting

Deleting all the fields(columns) from a .csv file if all rows in that columns are blanks

Hi Friends, I have come across some files where some of the columns don not have data. Key, Data1,Data2,Data3,Data4,Data5 A,5,6,,10,, A,3,4,,3,, B,1,,4,5,, B,2,,3,4,, If we see the above data on Data5 column do not have any row got filled. So remove only that column(Here Data5) and... (4 Replies)
Discussion started by: ks_reddy
4 Replies

10. Shell Programming and Scripting

Deleting a list of words from a text file

Hello, I have a list of words separated by spaces I am trying to delete from a text file, and I could not figure out what is the best way to do this. what I tried (does not work) : delete="password key number verify" arr=($delete) for i in arr { sed "s/\<${arr}\>]*//g" in.txt } >... (5 Replies)
Discussion started by: Hawk4520
5 Replies
All times are GMT -4. The time now is 09:57 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy