Sponsored Content
Full Discussion: Concatenate Numerous Files
Operating Systems Linux Fedora Concatenate Numerous Files Post 302723785 by sudon't on Wednesday 31st of October 2012 12:41:35 PM
Old 10-31-2012
I think I get it now. The first argument, (the path to the target dir), automatically becomes the first variable, and so on?

Here is the difference - to me - between scripting and the command line. With the CL, you have an application, maybe a flag or two, an argument, a target file. Very simple, and you can pipe that output to another app, etc...
With a script, there are all these strange symbols whose meanings I don't understand, and formatting whose purpose I don't understand. Why is there sometimes a bracket sitting on a line by itself? Why are some lines indented, and some not? I have no clue. Scripting is an area where I'll really have to start at the beginning. Not that I don't want to learn - I just haven't, yet. ; )

Here is where I'm at now. Turns out they were dos files. I checked this by opening a couple in vim. So I used find to pipe into dos2unix:
Code:
find ./01_Old\ Testament -name "*txt" -type f |dos2unix
find ./02_New\ Testament -name "*txt" -type f |dos2unix

I again used vi to check a couple of the files, and it no longer says [dos] on the bottom.
Then I wanted to see if all had an EOF newline. I'm not sure how to do this, but I opened a couple files in TextWrangler, showing invisibles, and they do seem to have a last newline. Anyway, the last lines of the files I looked at have that 'capital L laying on it's side' symbol, and the cursor will travel one line below that last line. How's that for scientific?
The way I invoked Alister's script is a little different than you show because I packed it away where I thought it should go:
Code:
which bible.sh
/usr/local/bin/bible.sh

But I think it's correct:
Code:
sh bible.sh ~/Sandbox/01_Old\ Testament ~/Sandbox/02_New\ Testament >Bible.txt

I even used the full paths, but I still end up with an empty file:
Code:
ls -l
total 120
drwxr-xr-x@ 41 rick  staff   1394 Oct 28 13:48 01_Old Testament
drwxr-xr-x@ 26 rick  staff    884 Oct 29 16:43 02_New Testament
-rw-r--r--   1 rick  staff      0 Oct 30 18:06 Bible.txt
-rw-r--r--   1 rick  staff  13054 Oct 29 16:55 NTfilelist.txt
-rw-r--r--   1 rick  staff  41377 Oct 29 16:47 OTfilelist.txt

It shouldn't be a problem with the eof newline - we would simply end up some lines stuck together. Did the fact that I changed directories affect the script? It doesn't seem like that should be the case. If I understand correctly, the script expects that input from the CL. I'm not sure how to proceed.

---------- Post updated 10-31-12 at 12:02 AM ---------- Previous update was 10-30-12 at 06:38 PM ----------

I also thought that, since we have proper list files, why not go back and try earlier solutions?

Elixir Sinari's solution:
Code:
rm resultfile ; while read num file ; do cat "$file" >> resultfile ; done <OTfilelist.txt
cat: : No such file or directory
cat: : No such file or directory
cat: : No such file or directory
. . .

And so on, all the way through.

And Bakunin's again:
Code:
rm resultfile ; while read num file ; do cat $file >> resultfile ; done <OTfilelist.txt

What happened here was that the first time, I had a blinking cursor as if it was working. Finally! - I thought. Then I noticed I wasn't using any processor. I looked, but cat did not seem to be a running process, but I let it go for about eight minutes, then killed it. I ended up with a 4 k empty file named 'resultfile'.
I ran it again, but it stopped itself after a couple moments, leaving me again with a 4 k file.

It occurred to me - why not just pipe find's output directly to cat? It worked with dos2unix, (and other programs), so why not?
Code:
find ./01_Old\ Testament -name "*txt" -type f |cat > OT.txt

But it only printed another file list named OT.txt

I don't know if this is useful, but here is what's in the directory, and where it's at. The two directories containing the files, and the two perfectly ordered and escaped file lists. I tossed the worthless files that were created by various attempts.
Code:
pwd
/Users/rick/Sandbox
ls -l
total 120
drwxr-xr-x@ 41 rick  staff   1394 Oct 28 13:48 01_Old Testament
drwxr-xr-x@ 26 rick  staff    884 Oct 29 16:43 02_New Testament
-rw-r--r--   1 rick  staff  13054 Oct 29 16:55 NTfilelist.txt
-rw-r--r--   1 rick  staff  42672 Oct 30 23:06 OTfilelist.txt

And here's a sample of what the filelist - OTfilelist.txt - looks like, in case there is a problem with it:
Code:
./01_Old\ Testament/21_Ecclesiastes/Ecclesiastes11.txt
./01_Old\ Testament/21_Ecclesiastes/Ecclesiastes12.txt
./01_Old\ Testament/22_Song\ of\ Solomon/Song\ of\ Solomon01.txt
./01_Old\ Testament/22_Song\ of\ Solomon/Song\ of\ Solomon02.txt

---------- Post updated at 11:43 AM ---------- Previous update was at 12:02 AM ----------

After much wailing and gnashing of teeth, I was struck with the inspiration that a big problem was with the file names, namely spaces. I replaced all spaces with connecting underlines:
Code:
find ./01_Old_Testament -name "*txt" -type f |rename 's/ /_/g'

Then I tossed in xargs:
Code:
find ./01_Old_Testament -name "*txt" -type f |xargs cat > OT.txt

Success! So we do the same with the New Testament directory, then concatenate those two files:
Code:
cat OT.txt NT.txt > TheeBible.txt

If it warms up a bit, I will make a burnt offering on the grill.
I thank everyone for their help and guidance and hope I didn't completely wear out my welcome. I really learned a lot!

---------- Post updated at 12:41 PM ---------- Previous update was at 11:43 AM ----------

Quote:
Originally Posted by elixir_sinari
Code:
rm resultfile ; while read num file ; do cat "$file" >> resultfile ; done <listfile.sorted

When you added double quotes to $file, was that an attempt to deal with the spaces in the file names?

Last edited by sudon't; 10-31-2012 at 01:09 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Deleting numerous files

Hi there, I have numerous files in a directory (approx 2500) that I want to delete although I get the following:- Server> rm *.* Arguments too long Is there a proper way of deleting this rather than breaking it down further through the list of files rm *10.* rm *11.* rm *12.* ... (10 Replies)
Discussion started by: Hayez
10 Replies

2. UNIX for Dummies Questions & Answers

How to concatenate all files.

Hi, I'm totally new to Unix. I'm an MVS mainframer but ran into a situation where a Unix server I have available will help me. I want to be able to remotely connect to another server using FTP, login and MGET all files from it's root or home directory, logout, then login as a different user and do... (1 Reply)
Discussion started by: s80bob
1 Replies

3. Shell Programming and Scripting

Script to concatenate several files

I need a script to concatenate several files in one step, I have 3 header files say file.S, file.X and file.R, I need to concatenate these 3 header files to data files, say file1.S, file1.R, file1.X so that the header file "file.S" will be concatenated to all data files with .S extentions and so on... (3 Replies)
Discussion started by: docaia
3 Replies

4. Shell Programming and Scripting

Concatenate rows in to 2 files

I have 2 files FILEA 1232342 1232342 2344767 4576823 2325642 FILEB 3472328 2347248 1237123 1232344 8787890 I want the output to go into a 3rd file and look like: FILEC 1232342 3472328 (1 Reply)
Discussion started by: unxusr123
1 Replies

5. Shell Programming and Scripting

Concatenate files

I have directory structure sales_only under which i have multiple directories for each dealer example: ../../../Sales_Only/xxx_Dealer ../../../Sales_Only/yyy_Dealer ../../../Sales_Only/zzz_Dealer Every day i have one file produce under each directory when the process runs. The requirement... (3 Replies)
Discussion started by: mohanmuthu
3 Replies

6. Shell Programming and Scripting

Concatenate files

Hi, I want to create a batch(bash) file to combine 23 files together. These files have the same extension. I want the final file is save to a given folder. Once it is done it will delete the 23 files. Thanks for help. Need script. (6 Replies)
Discussion started by: zhshqzyc
6 Replies

7. Shell Programming and Scripting

Concatenate files

I have a file named "file1" which has the following data 10000 20000 30000 And I have a file named "file2" which has the following data ABC DEF XYZ My output should be 10000ABC 20000DEF (3 Replies)
Discussion started by: bobby1015
3 Replies

8. UNIX for Dummies Questions & Answers

Concatenate Several Files to One

Hi All, Need your help. I will need to concatenate around 100 files but each end of the file I will need to insert my name DIRT1228 on each of the file and before the next file is added and arrived with just one file for all the 100files. Appreciate your time. Dirt (6 Replies)
Discussion started by: dirt1228
6 Replies

9. UNIX for Dummies Questions & Answers

Concatenate files

Hi I am trying to learn linux step by step an i am wondering can i use cat command for concatenate files but i want to place context of file1 to a specific position in file2 place of file 2 and not at the end as it dose on default? Thank you. (3 Replies)
Discussion started by: iliya24
3 Replies

10. UNIX for Dummies Questions & Answers

Concatenate files and delete source files. Also have to add a comment.

- Concatenate files and delete source files. Also have to add a comment. - I need to concatenate 3 files which have the same characters in the beginning and have to remove those files and add a comment and the end. Example: cat REJ_FILE_ABC.txt REJ_FILE_XYZ.txt REJ_FILE_PQR.txt >... (0 Replies)
Discussion started by: eskay
0 Replies
All times are GMT -4. The time now is 02:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy