The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
.
google unix.com



Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Unix Arithmatic operation issue , datatype issue thambi Shell Programming and Scripting 23 02-19-2008 07:19 AM
Unix command mmin issue Mohee UNIX for Dummies Questions & Answers 1 08-30-2007 11:41 PM
cp command issue. sai_nj UNIX for Dummies Questions & Answers 4 06-12-2007 01:15 PM
Tar command issue ajayyadavmca HP-UX 2 05-02-2007 11:42 AM
AIX 5.3 - Issue with using at command vigsgb AIX 2 08-03-2005 09:43 AM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 10-29-2007
RcR RcR is offline
Registered User
  
 

Join Date: Jun 2007
Posts: 10
Post Issue with Unix cat command

Hi Experts,

I am finding the performance of cat command is very wierd, it is taking more time to merge the files into a single file. We have a situation where we would be merging more than 100 files into a single file, but with cat command it is running slow. I tried doing with paste, join and cat, but cat is working faster than any of these. Please guide me if any one has faced similar issue, please help. How about sed and awk commands usage for merging bigger files.

Thanks,
Rajiv
  #2 (permalink)  
Old 10-29-2007
matrixmadhan matrixmadhan is offline Forum Advisor  
Technorati Master
  
 

Join Date: Mar 2005
Location: leaf node in B+ tree
Posts: 2,953
your statements are contradicting each other.

Could you please clarify that ?

Quote:
it is taking more time to merge the files into a single file.
Quote:
but with cat command it is running slow.
Quote:
I tried doing with paste, join and cat, but cat is working faster than any of these.
  #3 (permalink)  
Old 10-29-2007
RcR RcR is offline
Registered User
  
 

Join Date: Jun 2007
Posts: 10
Post Issue with Unix cat command

Matrixmadan,

We are currently using cat command to combine files, basically appending files together into a single file.

cat file1 file2 file3 .....file 100 > final_file

I tried creating a file list and ran cat command this way....

cat filelist | xargs cat >> final_file

There is no performance gain. The file1, file2 ...file 100 are of varying size, and the final_file is very big, somewhere around 15 GB. I wanted to know is there a better way to combine multiple files into one.

Thanks,
Rajiv
  #4 (permalink)  
Old 10-29-2007
matrixmadhan matrixmadhan is offline Forum Advisor  
Technorati Master
  
 

Join Date: Mar 2005
Location: leaf node in B+ tree
Posts: 2,953
Code:
#! /opt/third-party/bin/perl

use strict;

my @contents;
my @files_arr;

my $DIR1 = "/source/directory/";
my $big = "thebigfile.txt";

opendir(DIR, $DIR1) || die "Unable to open files in Dir : $DIR1 <$!> \n";
  @files_arr = readdir(DIR);
closedir(DIR);

open(BIG, ">", $big) or die "Unable to open file $big <$!> \n";

foreach (@files_arr) {
  if( ! /^\./ && ! /^\.\./ ) {
    print "$_\n";
    $_ = $DIR1 . $_;
    open(FILE, "<", $_) or die "Unable to open file $_ <$!> \n";
      @contents = <FILE>;
    close(FILE);
    print BIG "@contents";
  }
}

close(BIG);

exit 0
  #5 (permalink)  
Old 10-29-2007
RcR RcR is offline
Registered User
  
 

Join Date: Jun 2007
Posts: 10
Issue running cat command

MatrixMadan,

I tried running your script, but perl component is not available. I am getting error as "Perl not installed".

Thanks,
Rajiv
  #6 (permalink)  
Old 10-29-2007
porter porter is offline Forum Advisor  
Registered User
  
 

Join Date: Jan 2007
Posts: 2,965
cat is pretty efficient.

The operation is going to take a combination of CPU and IO. With the volumes you are talking about the IO will have more overhead that the CPU component.
Closed Thread

Bookmarks

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On




All times are GMT -4. The time now is 02:03 PM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0