Issue with Unix cat command


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Issue with Unix cat command
# 1  
Old 10-29-2007
Java Issue with Unix cat command

Hi Experts,

I am finding the performance of cat command is very wierd, it is taking more time to merge the files into a single file. We have a situation where we would be merging more than 100 files into a single file, but with cat command it is running slow. I tried doing with paste, join and cat, but cat is working faster than any of these. Please guide me if any one has faced similar issue, please help. How about sed and awk commands usage for merging bigger files.

Thanks,
Rajiv
# 2  
Old 10-29-2007
your statements are contradicting each other.

Could you please clarify that ? Smilie

Quote:
it is taking more time to merge the files into a single file.
Quote:
but with cat command it is running slow.
Quote:
I tried doing with paste, join and cat, but cat is working faster than any of these.
# 3  
Old 10-29-2007
Java Issue with Unix cat command

Matrixmadan,

We are currently using cat command to combine files, basically appending files together into a single file.

cat file1 file2 file3 .....file 100 > final_file

I tried creating a file list and ran cat command this way....

cat filelist | xargs cat >> final_file

There is no performance gain. The file1, file2 ...file 100 are of varying size, and the final_file is very big, somewhere around 15 GB. I wanted to know is there a better way to combine multiple files into one.

Thanks,
Rajiv
# 4  
Old 10-29-2007
Code:
#! /opt/third-party/bin/perl

use strict;

my @contents;
my @files_arr;

my $DIR1 = "/source/directory/";
my $big = "thebigfile.txt";

opendir(DIR, $DIR1) || die "Unable to open files in Dir : $DIR1 <$!> \n";
  @files_arr = readdir(DIR);
closedir(DIR);

open(BIG, ">", $big) or die "Unable to open file $big <$!> \n";

foreach (@files_arr) {
  if( ! /^\./ && ! /^\.\./ ) {
    print "$_\n";
    $_ = $DIR1 . $_;
    open(FILE, "<", $_) or die "Unable to open file $_ <$!> \n";
      @contents = <FILE>;
    close(FILE);
    print BIG "@contents";
  }
}

close(BIG);

exit 0

# 5  
Old 10-29-2007
Issue running cat command

MatrixMadan,

I tried running your script, but perl component is not available. I am getting error as "Perl not installed".

Thanks,
Rajiv
# 6  
Old 10-29-2007
cat is pretty efficient.

The operation is going to take a combination of CPU and IO. With the volumes you are talking about the IO will have more overhead that the CPU component.
# 7  
Old 10-29-2007
Java Issue with cat command

Porter,

I do agree with you. But still how far can we tune to speed up the process. Becuase of this file appending, whole load time is impacted. Let me know if you have any other thoughts on this...

Thanks,
Rajiv
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Issue with cat command on a for loop

Good day to all, I'd like to ask for your advice with regards to this. Scenario : I have here a file named TEST.tmp wherein the value inside is below; "ONE|TWO|FIVE|THREE|FOUR|SIX~SEVEN~EIGHT" "NINE" But when I'm trying to use this in a simple command like; for TESTING in $(cat... (4 Replies)
Discussion started by: asdfghjkl
4 Replies

2. Shell Programming and Scripting

Issue in executing cat (remote ssh)

Hi, I need to ssh remotely to a machine and cat a file assign the value to a variable Script: #!/bin/bash -x value=`cat config.txt` echo "$value" ssh me@xxx.host.com "valu='cat /export/home/test.md5'; echo "$valu"" | tee Execution: $ ./x ++ cat config.txt + value='touch me' +... (5 Replies)
Discussion started by: close2jay
5 Replies

3. Shell Programming and Scripting

Script with ^M causes display issue with cat or more

I have a script to do a couple simple but repetitive commands on files that are provided to us. One of the things is to get rid of the line feeds. This is the section that is causing problems, i even cut this section into its own file to make sure nothing else was affecting it. #!/usr/bin/bash... (4 Replies)
Discussion started by: oly_r
4 Replies

4. Shell Programming and Scripting

Unix Mget command issue

Hi, I am facing an issue with mget command, below are the details The following commands are in a batch file "abc.ctl" cd /mypath/mydirectory/ mget 'MYFILE_*.touch' bye I use this file as a parameter for an sftp command sftp -b abc.ctl -oIdentityFile=$myserver -oPort=$myport >>... (0 Replies)
Discussion started by: saurabh_mh
0 Replies

5. Shell Programming and Scripting

Issue in cat command

We have shell script (in ksh) which reads the records from a csv file line by line and does some operation. We have below command to read the file (CSV has the absolute path of files stored on a server something like SER/IMP/TEST/2010/12/123465.1). P_FILES=`cat $P_BATCH_FILE` for i in $P_FILES... (2 Replies)
Discussion started by: shreevatsau
2 Replies

6. UNIX for Dummies Questions & Answers

How to issue a long command on unix

Hi, Im trying to create a command template here. how can i issue this long command in one enter? echo -e "\nIssue or Request - Analysis Summary \n\nMemory Utilization Utilization Threshold\n\nResolution Summary\n\n1. check Server Info";echo "$(uname -a;uptime;date)"; echo -e "\n\n2. check... (9 Replies)
Discussion started by: jinslick25
9 Replies

7. Shell Programming and Scripting

simple for loop/cat issue

ok.. so problem is: I have a file that reads: cat 123 1 and 2 3 and 4 5 and 6 I was using for loops to run through this information. Code: for i in `cat 123` do echo $i done shouldn't the output come as 1 and 2 (3 Replies)
Discussion started by: foal_11
3 Replies

8. Emergency UNIX and Linux Support

cat issue

I have list of files in my current directory abc.txt 123.csv 234.csv 245.csv 145.csv 123_ex_c.sv I don't want to open first and last file. i.e (abc.txt and 123_ex_csv) I tried cat *.csv, but it won't work. Can anyone tell me the proper regex only in cat Thanks Pritish ... (2 Replies)
Discussion started by: pritish.sas
2 Replies

9. Shell Programming and Scripting

cat in the command line doesn't match cat in the script

Hello, So I sorted my file as I was supposed to: sort -n -r -k 2 -k 1 file1 | uniq > file2 and when I wrote > cat file2 in the command line, I got what I was expecting, but in the script itself ... sort -n -r -k 2 -k 1 averages | uniq > temp cat file2 It wrote a whole... (21 Replies)
Discussion started by: shira
21 Replies

10. UNIX for Dummies Questions & Answers

Unix command mmin issue

hi, Function 'mmin' is not in my unix, what is the other alternative to find files created/modified in last 10 minutes. Unix Command: find . -min -10 find: bad option -min find: path-list predicate-list Thank you (1 Reply)
Discussion started by: Mohee
1 Replies
Login or Register to Ask a Question