Sponsored Content
Full Discussion: Concatenate Numerous Files
Operating Systems Linux Fedora Concatenate Numerous Files Post 302723471 by bakunin on Tuesday 30th of October 2012 08:33:29 AM
Old 10-30-2012
Quote:
Originally Posted by sudon't
OK, I just want to be clear. In your example, it looks like you are pointing the script directly to the files, rather than a filelist. Is this correct?
No. alister is creating a list of values which he supposes to correspond to directory- and filenames. This is because of the way you laid out the problem in your previous posts.

First off, how to find your listfile:

Code:
-bash: ./OTlistfile.txt: No such file or directory

The same way you searched for all the other files:

Code:
find ~ -type f -name "OTlistfile.txt" -print

I will not really matter where it is stored. You could use full paths:

Code:
rm resultfile ; while read num file ; do cat $file spacerfile >> resultfile ; done </full/path/to/where/you/found/listfile.sorted

Second, here are some general tips, some of them digressing from the problem at hand to some more generalized angle:

Present your problem as concisely as possible.

Your description of the problem (the directory layout, how the files are organised, etc.) changed somewhat over the course of the thread. You didn't contradict yourself directly, but you left out critical information in your first description(s) which you gave out one at a time in your later posts.

Problems in shell scripting - and what you are attempting is shell-scripting, despite your claims it is way above your capabilities - are like any other programming problem mostly depending on a clear and precise definition. Once you have precisely defined what you want to do and how you want it to be done the solution is in most cases obvious and easy to implement. Have a look in the "Shell Programming and Scripting" forum and compare threads with many answers with the ones with few answers. One would expect the threads with many answers to be more interesting, but the opposite is the case: the ones with many answers are the ones which usually go like this:

Q: i need to produce X
A: do THIS
Q: ah, yes, fine, but i need the Xs to be different, more like Ys
A: modify THIS to be THAT to produce Ys
Q: many thanks, but my Ys should have a special quality of Z
A: *sigh* do THAT, but modify the FOO part to BAR
... rinse and repeat ad nauseam


The fifth answer was not at all more "complex" or "hard" to give than the first - it was just the realization of having come up with 4 answers completely unnecessarily that caused the sigh.

So, analyze the problem you have as exactly and meticulously as possible and you will be on the fast lane to programmers ascension. What we do is not an arcane art, but just this skill of defining problems precisely and abstractly, mixed with some common sense - trust me, i'm bakunin! ;-))

Second, you sure might want to know how alisters script works (which is, btw., based on a better idea than my own solution, so you should go with it).

Here is the short version of "Introduction to programming logic 101:

The core part is a loop, into which a "here-document" is fed. "Here-documents" are shell-constructs, which are similar to files but have fixed contents, so that they are incorporated into scripts directly. Cosider the following line:

Code:
cat x > y

A file "x" is read by "cat" and its contents are dumped into file "y". What exactly ends up in "y" depends on what was in "x" in first place. But if you want "y" to have a fixed content you could create a here-document replacing the file:

Code:
cat <<EOF > y
foo
bar
foobar
EOF

This says: treat everything you read until a line that reads "EOF" as the content of a (virtual) file. We could have the three lines in file "x" and used the above command to the same effect.

So lets see the relevant part of alisters script:

Code:
while IFS= read -r b; do
     .....
done <<'END_OF_DAYS'
Genesis
Exodus
...
Zechariah
Malachi

Matthew
Mark
...
Jude
Revelation
END_OF_DAYS

The core part into which this here-document is fed is this:

Code:
while read b ; do
     .....
done

This takes one line at a time, fills it into a variable named "b" and does whatever is between "do" and "done". Because with every loop the variable content of "b" changes we can use it for out purposes. Let us say we want to surround the name with equal signs. We could do this:

Code:
while read b; do
     echo == $b ==
done <<'END_OF_DAYS'
Genesis
Exodus
...
Zechariah
Malachi

Matthew
Mark
...
Jude
Revelation
END_OF_DAYS

The command echo == $b == does nothing else than print "==", than the content of variable b (this is what "$b" stands for) and then "==" again. Now, alister does something more sophisticated with "$b", but basically this is it. Let us see what he does:


Code:
    [ -z "$b" ] && t=$nt && continue

He is abbreviating here, so it is not that obvious. Let us write it in the long form and it will become clearer:

Code:
    if [ -z "$b" ] ; then
                   t=$nt
                   continue
              fi

The -z "$b" means: if "$b" is empty. This is true exactly one time: when the loop reads in the empty line in the middle of the document. In this case the variable "t" is filled with "$nt" (the contents of variable "nt") and the enclosing while-loop is immediately started over again ("continue").

Now, take stock: what are the various variables filled with:

$ot=path to old testament books, probably "./01_Old Testament"
$nt=path to old testament books, probably "./01_New Testament"
$t=either $ot (at start) or $nt (after the blank line is processed)

OK, on we go. What else does the while-loop do:

Code:
    i=1
    while cat "$t/$b/$b$i.txt" 2>/dev/null; do
        i=$((i+1))
    done

First, a variable "i" is set to "1". Then, there is another loop:

Code:
    while cat "$t/$b/$b$i.txt" 2>/dev/null; do
        i=$((i+1))
    done

I have to explain something about while-loops here: the general form is

Code:
while <command> ; do
     ......
done

This loop will run "<command>" and if this returns 0 (=TRUE) it will run the body of the loop. The same was true when we used:

Code:
while read b ; do
    ....
done

"read" is a command and it returns TRUE when there is something to read and FALSE if not - this is why the loop stops at the end of the list we feed into.

Inside this loop there is nothing special done, except for incrementing "i" by 1. This is simply counting: 1, 2, 3, 4, 5, .... Every time the command

Code:
cat "$t/$b/$b$i.txt"

is issued. Replacing the various variables with their content (see the list above), this is:

Code:
cat "./01_Old Testament/Genesis/Genesis1.txt"

(after incrementing i by 1)

Code:
cat "./01_Old Testament/Genesis/Genesis2.txt"

etc. at some point, this will give us a filename which doesn't exist. If Genesis has 51 chapters (haven't bothered to look), this would be:

Code:
cat "./01_Old Testament/Genesis/Genesis52.txt"

This time, "cat" would return a non-zero return value, meaning "FALSE" and the loop would stop.

Now there is one last question left: without redirection "cat" will display the content of the file to the screen (try it!). We haven't used any redirection, so why does the output not land on the screen?

Code:
sh bible.sh ~/Desktop/Old\ Testament ~/Desktop/New\ Testament > bible.txt

This is why: it is not only possible to redirect individual commands but also whole scripts. Without the last "> bible.txt" the text would indeed land on the screen. You could replace the redirection with a pipeline:

Code:
sh bible.sh ~/Desktop/Old\ Testament ~/Desktop/New\ Testament | more

Will seend the output to "more", which will display it on screen, but pagewise (hit any key to display another page, CTRL-C to end) or

Code:
sh bible.sh ~/Desktop/Old\ Testament ~/Desktop/New\ Testament | grep someword

to filter only for lines with "someword" in them.

I hope this helps.

bakunin

/PS: i refrained from giving you any practical solution because i figured you are here to learn foremost and to solve your problem at hand second. I hope to have served your intentions best in enabling you to understand and write scripts yourself instead of just throwing something miraculously working at your feet.

Once you overcome your reservations i am sure you will find neverending joy in programming the shell. Don't be shy, there may be only a few chosen, but an awful lot are invited. ;-)
These 4 Users Gave Thanks to bakunin For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Deleting numerous files

Hi there, I have numerous files in a directory (approx 2500) that I want to delete although I get the following:- Server> rm *.* Arguments too long Is there a proper way of deleting this rather than breaking it down further through the list of files rm *10.* rm *11.* rm *12.* ... (10 Replies)
Discussion started by: Hayez
10 Replies

2. UNIX for Dummies Questions & Answers

How to concatenate all files.

Hi, I'm totally new to Unix. I'm an MVS mainframer but ran into a situation where a Unix server I have available will help me. I want to be able to remotely connect to another server using FTP, login and MGET all files from it's root or home directory, logout, then login as a different user and do... (1 Reply)
Discussion started by: s80bob
1 Replies

3. Shell Programming and Scripting

Script to concatenate several files

I need a script to concatenate several files in one step, I have 3 header files say file.S, file.X and file.R, I need to concatenate these 3 header files to data files, say file1.S, file1.R, file1.X so that the header file "file.S" will be concatenated to all data files with .S extentions and so on... (3 Replies)
Discussion started by: docaia
3 Replies

4. Shell Programming and Scripting

Concatenate rows in to 2 files

I have 2 files FILEA 1232342 1232342 2344767 4576823 2325642 FILEB 3472328 2347248 1237123 1232344 8787890 I want the output to go into a 3rd file and look like: FILEC 1232342 3472328 (1 Reply)
Discussion started by: unxusr123
1 Replies

5. Shell Programming and Scripting

Concatenate files

I have directory structure sales_only under which i have multiple directories for each dealer example: ../../../Sales_Only/xxx_Dealer ../../../Sales_Only/yyy_Dealer ../../../Sales_Only/zzz_Dealer Every day i have one file produce under each directory when the process runs. The requirement... (3 Replies)
Discussion started by: mohanmuthu
3 Replies

6. Shell Programming and Scripting

Concatenate files

Hi, I want to create a batch(bash) file to combine 23 files together. These files have the same extension. I want the final file is save to a given folder. Once it is done it will delete the 23 files. Thanks for help. Need script. (6 Replies)
Discussion started by: zhshqzyc
6 Replies

7. Shell Programming and Scripting

Concatenate files

I have a file named "file1" which has the following data 10000 20000 30000 And I have a file named "file2" which has the following data ABC DEF XYZ My output should be 10000ABC 20000DEF (3 Replies)
Discussion started by: bobby1015
3 Replies

8. UNIX for Dummies Questions & Answers

Concatenate Several Files to One

Hi All, Need your help. I will need to concatenate around 100 files but each end of the file I will need to insert my name DIRT1228 on each of the file and before the next file is added and arrived with just one file for all the 100files. Appreciate your time. Dirt (6 Replies)
Discussion started by: dirt1228
6 Replies

9. UNIX for Dummies Questions & Answers

Concatenate files

Hi I am trying to learn linux step by step an i am wondering can i use cat command for concatenate files but i want to place context of file1 to a specific position in file2 place of file 2 and not at the end as it dose on default? Thank you. (3 Replies)
Discussion started by: iliya24
3 Replies

10. UNIX for Dummies Questions & Answers

Concatenate files and delete source files. Also have to add a comment.

- Concatenate files and delete source files. Also have to add a comment. - I need to concatenate 3 files which have the same characters in the beginning and have to remove those files and add a comment and the end. Example: cat REJ_FILE_ABC.txt REJ_FILE_XYZ.txt REJ_FILE_PQR.txt >... (0 Replies)
Discussion started by: eskay
0 Replies
All times are GMT -4. The time now is 08:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy