UNIX for Dummies Questions & Answers

Hello,

I am trying to add a line (usually just a word) to some text files in a directory that are already sorted. I just don't want to run the sort command again because it can take a long time when the text or log files are really huge. I have a bashscript that will take in the 1st argument ($1) and add it to the .txt files by appending the word to the end, then sorts it and the output goes to a temp file, it takes forever because some .txt files are very large, (over 1 gig). Any suggestions to add a word quicker? This is what I have:

Code:

#!/bin/bash 

for iFile in `ls *.txt`
do
echo $iFile

echo $1 >> $iFile | cat $iFile | sort > temp.txt
  rm $iFile
  mv temp.txt $iFile


done

Thanks

Code:

nawk -v new="$1" '!done && $0 > new { print new RS $0; done=1; next } 1' sorted

Thanks for the reply, but that doesn't seem to be working. It could be I am doing it wrong. Could you explain what that nawk command is doing?

Try this version:

Code:

#!/bin/sh

for iFile in *.txt; do
  echo $iFile
  { rm "$iFile"; { cat -; echo $1;} | sort > "$iFile";} < "$iFile" 
done

exit 0

Like this?

Code:

for iFile in `ls *.txt`
do
  echo $iFile
  nawk -v new="$1" '!done && $0 > new { print new RS $0; done=1; next } 1' $iFile >temp.txt
  rm $iFile
  mv temp.txt $iFile
done

It goes through each line until it finds the line which is lexicographically greater, then it prints $1 before it. Then it stops messing with the data and just prints the rest...

Hi.

Two thoughts:

1) If you have only one line to add, then it is, in effect, an already sorted file, so you can use the merge option in sort with your already-sorted large file. That's effectively a high-speed copy.

2) (Old-timers will remember this). You could patch in-place by replacing exactly one line in the file with a pointer to an auxiliary file that holds the line you have extracted from the main file, and the new line. You would need a special reading routine that recognizes the pointer, reads the auxiliary file, and comes back to read the main file. It would be fast, but tricky, because you have to re-write in-place. We essentially did that when we programmed in absolute octal and we needed to patch code.

Good luck ... cheers, drl

Do not use that approach unless it's acceptable to lose an entire file (either because the data is not of any importance or you have backups). If the pipeline is killed for some reason, you will lose data.

Personally, I would go with sort -m. As drl mentioned, this situation is dealing with two sorted files (the large pre-sorted file and a word which could easily be written to a temp file).

Regards,
Alister

---------- Post updated at 05:48 PM ---------- Previous update was at 05:46 PM ----------

Nothing good can come from using ls and command substitution in that manner. Only problems. Just use a simpler, safer, more efficient file glob.

Code:

for iFile in *.txt

Regards,
Alister