Sponsored Content
Full Discussion: Concatenate Numerous Files
Operating Systems Linux Fedora Concatenate Numerous Files Post 302724487 by bakunin on Wednesday 31st of October 2012 04:05:57 PM
Old 10-31-2012
Quote:
Originally Posted by sudon't
I think I get it now. The first argument, (the path to the target dir), automatically becomes the first variable, and so on?
Correct.

Quote:
Originally Posted by sudon't
Here is the difference - to me - between scripting and the command line. With the CL, you have an application, maybe a flag or two, an argument, a target file. Very simple, and you can pipe that output to another app, etc...
With a script, there are all these strange symbols whose meanings I don't understand, and formatting whose purpose I don't understand.
Again: all these devices in scripts work (at least in principle) on the command line too and everything you write on teh commandline could come from a script. In fact every shell is a "command language" which you either type in by hand or have typed in from a script file. A script file is basically just a file with stored commands you don't have to type again should you need them a second time.


Quote:
Originally Posted by sudon't
Why is there sometimes a bracket sitting on a line by itself?
You mean "{"? It is for command grouping. Basically everything between "{ ... }" works like a single command from the outside. It is like you could write a script, give it a name and then use this script in another script like any other command. This works the same, just inside scripts.

Quote:
Originally Posted by sudon't
Why are some lines indented, and some not?
Simple: to be easier readable (for humans). If you have a loop (for ... done or while ... done) or a branch (if ... else ... fi or case ... esac) you indent the body of this to see easily what this body of the construct is. For the shell executing the command it makes no difference at all. You can write:

Code:
 while <condition> ; do
     command1
     command2
     command3
done

and the definition of the loop as well as the body immediately will stand out. This:

Code:
 while<condition> ; do command1; command2; command3; done

is syntactically the same but is a lot harder to find out what is loop definition and what is its body.


Quote:
Originally Posted by sudon't
So I used find to pipe into dos2unix:
Code:
find ./01_Old\ Testament -name "*txt" -type f |dos2unix
find ./02_New\ Testament -name "*txt" -type f |dos2unix

Very well. There is a "-exec" flag for find, which you could use. It takes a "command template" where the filename "find" has found is represented by "{}". Suppose you have a command

Code:
cmd /path/to/myfile.txt

and you want to execute this command with every txt-file in all subdirectories of "/there/too". You would write:

Code:
find /there/too -type f -name "*txt" -exec cmd {} \;

"find" will find all the filenames and for each filename found that way execute the command up to "\;" (this signifies the end of the template command) with "{}" replaced by the actual filename. You might find this device extremely useful and - i can promise - once you got the hang of it you can easily outperform every filemanager there might be. Unix aficionados are not command line fetishists because they are masochists, but because they can type in seconds what you can do with a mouse in hours.


Quote:
Originally Posted by sudon't
I again used vi to check a couple of the files, and it no longer says [dos] on the bottom.
Then I wanted to see if all had an EOF newline. I'm not sure how to do this,
We have a several "vi" resources here, threads where people tried to explain its usage in general or specific aspects of it. "vi" is most times a "love on third sight". First, you think it is overly complicated, the first half year you use it you think its giving you the third degree and after this time you start missing its features in every other program you type more than two keystrokes in.

Here is how to make the unvisible characters visible in "vi":

open the file. Hit "<ESC>" (maybe repeatedly, it can't hurt) to make sure you are in command mode. Press ":". A line with ":" at the beginning will appear in the last line of the screen. Enter "set list" and press <ENTER>. The line will disappear and you will see the non-printing characters now. Press ":" again and enter "set nolist" to switch that mode off or simply leave the editor. These characters will appear and have the following meaning:

Code:
^I     tab character
$      line end, Unix style
^M$    line end, DOS/Windows style

Notice that "^I" and "^M" are ONE character, as you can see when you pass over them with the cursor.

To remove the DOS line ends, you can do the following magic:

Again, from the command mode (<ESC>, you remember, will always take you there) press ":" and type:

Code:
1,$ s/^M$//

Enter the "^M" by pressing "<CTRL>-<V>" (take the next input character verbosely) and then either "<CTRL>-<M>" or "<ENTER>". You will notice that the "^M" appears.

You can do this with "list" mode set and you will see the "^M"s at the line ends (ALL line ends!) disappear. The command says: from the first to the last line ("1,$") substitute ("s") a ^M character, followed by a line end ("/^M$/") with nothing ("//").

Quote:
The way I invoked Alister's script is a little different than you show because I packed it away where I thought it should go:
Code:
which bible.sh
/usr/local/bin/bible.sh

Very good. in Unix every directory has its rationale and purpose and this is even (informally) standardized (we use the word "canonical" for this type of informal standard). Indeed "/usr/local/bin" is the directory where executable files which do not belong to the OS go (OS executables go to "/usr/bin"). "/usr", btw., is for "Unix Software Resources", even if it is usually pronounced like "user".

Quote:
If it warms up a bit, I will make a burnt offering on the grill.
LOL! Well, i think christians have burned already enough things in history, so it might even work if it doesn't warm up enough. Bishop Theophilus of Alexandria is quoted to have said, while burning down the library of Alexandria, that the books in there either agree with the bible and are superfluous or disagree with the bible and are rightfully burned. I am not divinely inspired to the same degree as this venerable bishop, but i can suggest a book about (Korn) shell and shell programming. It helps best, btw., if being read instead of being burned:

Barry Rosenberg, Korn Shell Programming Tutorial

You will find the read informative as well as very entertaining.

Quote:
When you added double quotes to $file, was that an attempt to deal with the spaces in the file names?
Exactly. This is the common method of dealing with blanks, because blanks are the shells default way of separating things. If one doesn't the blank (or any other) character to do that you use quoting.

I hope this helps.

bakunin
This User Gave Thanks to bakunin For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Deleting numerous files

Hi there, I have numerous files in a directory (approx 2500) that I want to delete although I get the following:- Server> rm *.* Arguments too long Is there a proper way of deleting this rather than breaking it down further through the list of files rm *10.* rm *11.* rm *12.* ... (10 Replies)
Discussion started by: Hayez
10 Replies

2. UNIX for Dummies Questions & Answers

How to concatenate all files.

Hi, I'm totally new to Unix. I'm an MVS mainframer but ran into a situation where a Unix server I have available will help me. I want to be able to remotely connect to another server using FTP, login and MGET all files from it's root or home directory, logout, then login as a different user and do... (1 Reply)
Discussion started by: s80bob
1 Replies

3. Shell Programming and Scripting

Script to concatenate several files

I need a script to concatenate several files in one step, I have 3 header files say file.S, file.X and file.R, I need to concatenate these 3 header files to data files, say file1.S, file1.R, file1.X so that the header file "file.S" will be concatenated to all data files with .S extentions and so on... (3 Replies)
Discussion started by: docaia
3 Replies

4. Shell Programming and Scripting

Concatenate rows in to 2 files

I have 2 files FILEA 1232342 1232342 2344767 4576823 2325642 FILEB 3472328 2347248 1237123 1232344 8787890 I want the output to go into a 3rd file and look like: FILEC 1232342 3472328 (1 Reply)
Discussion started by: unxusr123
1 Replies

5. Shell Programming and Scripting

Concatenate files

I have directory structure sales_only under which i have multiple directories for each dealer example: ../../../Sales_Only/xxx_Dealer ../../../Sales_Only/yyy_Dealer ../../../Sales_Only/zzz_Dealer Every day i have one file produce under each directory when the process runs. The requirement... (3 Replies)
Discussion started by: mohanmuthu
3 Replies

6. Shell Programming and Scripting

Concatenate files

Hi, I want to create a batch(bash) file to combine 23 files together. These files have the same extension. I want the final file is save to a given folder. Once it is done it will delete the 23 files. Thanks for help. Need script. (6 Replies)
Discussion started by: zhshqzyc
6 Replies

7. Shell Programming and Scripting

Concatenate files

I have a file named "file1" which has the following data 10000 20000 30000 And I have a file named "file2" which has the following data ABC DEF XYZ My output should be 10000ABC 20000DEF (3 Replies)
Discussion started by: bobby1015
3 Replies

8. UNIX for Dummies Questions & Answers

Concatenate Several Files to One

Hi All, Need your help. I will need to concatenate around 100 files but each end of the file I will need to insert my name DIRT1228 on each of the file and before the next file is added and arrived with just one file for all the 100files. Appreciate your time. Dirt (6 Replies)
Discussion started by: dirt1228
6 Replies

9. UNIX for Dummies Questions & Answers

Concatenate files

Hi I am trying to learn linux step by step an i am wondering can i use cat command for concatenate files but i want to place context of file1 to a specific position in file2 place of file 2 and not at the end as it dose on default? Thank you. (3 Replies)
Discussion started by: iliya24
3 Replies

10. UNIX for Dummies Questions & Answers

Concatenate files and delete source files. Also have to add a comment.

- Concatenate files and delete source files. Also have to add a comment. - I need to concatenate 3 files which have the same characters in the beginning and have to remove those files and add a comment and the end. Example: cat REJ_FILE_ABC.txt REJ_FILE_XYZ.txt REJ_FILE_PQR.txt >... (0 Replies)
Discussion started by: eskay
0 Replies
All times are GMT -4. The time now is 08:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy