Sponsored Content
Full Discussion: Merge files in a directory
Top Forums Shell Programming and Scripting Merge files in a directory Post 302980312 by rbatte1 on Friday 26th of August 2016 04:30:38 AM
Old 08-26-2016
The loop suggested seems odd to me. The sed is difficult to read and is a large process to start, so for multiple files will be slow. It also assumes that the target output files do not exist.

Would you be better with:-
Code:
filenames="$(for i in access*
do
  echo ${i%%.*}                     # Variable substitution removes everything after the first dot
done | sort -u)"                    # Building a list of required output files

for filename in $filenames
do
   cat ${filename}.* > $filename   # Write all matching files in one step
done

I know it is a two step process, but I've kept IO to a minimum and used internal functions rather than calling external processes.

Because of the trailing .* on the cat, this will not match the output file as an input file if it already exists. If you run this and then tidy away the input files, a re-run will match the output files as potential input, but then the cat will complain about missing input.




I hope that this helps,
Robin
This User Gave Thanks to rbatte1 For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

merge files

Hi, i have the files f1 and f2 like: files f1: c1 a1 c2 a2 c3 a3 file f2: c1 b1 c2 b2 c3 b3 i want merge the f1 and f2 file to f3 file like: c1 a1 b1 c2 a2 b3 c3 a3 b3........ .... . . please help me onthis..... (5 Replies)
Discussion started by: koti_rama
5 Replies

2. Shell Programming and Scripting

how to merge these two files?

I have two files, each of them has 12 lines, fileA has 3 columns, fileB has 1 column, like the following FileA a 1 b 2 c 3 ..blabla FileB A B C ..blabla Now I am trying to put the content of fileB as column 3 of fileA, e.g. a 1 A b 2 B c 3 C (3 Replies)
Discussion started by: fedora
3 Replies

3. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

4. Shell Programming and Scripting

Merge 2 files

Hello, i'd like a bash script to merge 2 files without duplicate lines. Example : file1 : toto titi file2 : toto tata Expected result, file3 : toto (5 Replies)
Discussion started by: Celmar
5 Replies

5. Shell Programming and Scripting

Merge files

Hello, I have a application software plink. It can merge files with some kinds of way. The command likes: plink --file 1 --merge 2 --recode --out merged That means merge file 1 and 2 then output file "merged". However I have 23 files (1,2,3,...22,23)to be merged together. How can I use... (2 Replies)
Discussion started by: zhshqzyc
2 Replies

6. UNIX for Dummies Questions & Answers

Merge files

Hi, I would like to know how can I merge files based on their coordinates, but mantaining the score of each file in the output file like: Note: 1st column is for chromosome, 2nd for start, 3rd for end of segment, 4th for score file1: 1 200 300 20 1 400 500 30 file2: 1 200 350 30 1... (1 Reply)
Discussion started by: fadista
1 Replies

7. Shell Programming and Scripting

Checking in a directory how many files are present and basing on that merge all the files

Hi, My requirement is,there is a directory location like: :camp/current/ In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated... (10 Replies)
Discussion started by: srikanth_sagi
10 Replies

8. Shell Programming and Scripting

Merge files and generate a resume in two files

Dear Gents, Please I need your help... I need small script :) to do the following. I have a thousand of files in a folder produced daily. I need first to merge all files called. txt (0009.txt, 0010.txt, 0011.txt) and and to output a resume of all information on 2 separate files in csv... (14 Replies)
Discussion started by: jiam912
14 Replies

9. Shell Programming and Scripting

How to merge two files?

Dear Frens, I have two files and need to merge into one file. Like File_1: Field1 Field2 1 4 File_2: Field1 Field2 3 5 I need one single output as File_1: Field1 Field2 1 4 3 5 This means taking header from either file. (8 Replies)
Discussion started by: manisha_singh
8 Replies

10. Shell Programming and Scripting

Directory containing files,Print names of the files in the directory that are exactly same content.

Given a directory containing say a few thousand files, please output a list of all the names of the files in the directory that are exactly the same, i.e. have the same contents. func(a_directory_name) output -> {“matches”: , ... ]} e.g. func(“/home/my/files”) where the directory... (7 Replies)
Discussion started by: anuragpgtgerman
7 Replies
cat(1)								   User Commands							    cat(1)

NAME
cat - concatenate and display files SYNOPSIS
cat [-nbsuvet] [file...] DESCRIPTION
The cat utility reads each file in sequence and writes it on the standard output. Thus: example% cat file prints file on your terminal, and: example% cat file1 file2 >file3 concatenates file1 and file2, and writes the results in file3. If no input file is given, cat reads from the standard input file. OPTIONS
The following options are supported: -n Precede each line output with its line number. -b Number the lines, as -n, but omit the line numbers from blank lines. -u The output is not buffered. (The default is buffered output.) -s cat is silent about non-existent files. -v Non-printing characters (with the exception of tabs, new-lines and form-feeds) are printed visibly. ASCII control characters (octal 000 - 037) are printed as ^n, where n is the corresponding ASCII character in the range octal 100 - 137 (@, A, B, C, . . ., X, Y, Z, [, , ], ^, and _); the DEL character (octal 0177) is printed ^?. Other non-printable characters are printed as M-x, where x is the ASCII character specified by the low-order seven bits. When used with the -v option, the following options may be used: -e A $ character will be printed at the end of each line (prior to the new-line). -t Tabs will be printed as ^I's and formfeeds to be printed as ^L's. The -e and -t options are ignored if the -v option is not specified. OPERANDS
The following operand is supported: file A path name of an input file. If no file is specified, the standard input is used. If file is `-', cat will read from the standard input at that point in the sequence. cat will not close and reopen standard input when it is referenced in this way, but will accept multiple occurrences of `-' as file. USAGE
See largefile(5) for the description of the behavior of cat when encountering files greater than or equal to 2 Gbyte ( 2**31 bytes). EXAMPLES
Example 1: Concatenating a file The following command: example% cat myfile writes the contents of the file myfile to standard output. Example 2: Concatenating two files into one The following command: example% cat doc1 doc2 > doc.all concatenates the files doc1 and doc2 and writes the result to doc.all. Example 3: Concatenating two arbitrary pieces of input with a single invocation The command: example% cat start - middle - end > file when standard input is a terminal, gets two arbitrary pieces of input from the terminal with a single invocation of cat. Note, however, that if standard input is a regular file, this would be equivalent to the command: cat start - middle /dev/null end > file because the entire contents of the file would be consumed by cat the first time `-' was used as a file operand and an end-of-file condition would be detected immediately when `-' was referenced the second time. ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of cat: LANG, LC_ALL, LC_CTYPE, LC_MES- SAGES, and NLSPATH. EXIT STATUS
The following exit values are returned: 0 All input files were output successfully. >0 An error occurred. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |CSI |enabled | +-----------------------------+-----------------------------+ |Interface Stability |Standard | +-----------------------------+-----------------------------+ SEE ALSO
touch(1), attributes(5), environ(5), largefile(5), standards(5) NOTES
Redirecting the output of cat onto one of the files being read will cause the loss of the data originally in the file being read. For exam- ple, example% cat filename1 filename2 >filename1 causes the original data in filename1 to be lost. SunOS 5.10 1 Feb 1995 cat(1)
All times are GMT -4. The time now is 06:15 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy