Concatenate 560 files in one


 
Thread Tools Search this Thread
# 1  
Concatenate 560 files in one

HI all,
could please help me in this code. I have 560 files containing the same columns but different rows. i want to concatenate all these files in one big file. i want to keep the header of the first file then add the dat from other files horizontally.
the name of my files contains 2 variables : chunk number and chromosome number. this an example

code:
IMP_chunk11-chr6_file

i have tried this code but it works only for one variable and 22 files. anu suggestion to adapt it to my case

code :


Code:
for p in 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
do
more +2 EUR.chr${p}.Fst > EUR.chr${p}.Fst.tmp
done

cp EUR.chr1.Fst EUR.chr1.Fst.tmp

cat EUR.chr*.Fst.tmp > EUR.all.Fst

thank you veru much

Moderator's Comments:
Mod Comment edit by bakunin: Please reread the rules, CODE-tags are not optional when posting code. Thank you.

Last edited by bakunin; 01-26-2013 at 06:29 PM..
# 2  
Code:
p=2
k=0
cat EUR.1.fst >EUR.all.fst
    while [ p -lt 561 ]
        do
             while read line
               do
                if [ k -eq 1 ]
                  then
                           echo "$line" >>EUR.all.fst
                  fi
            k=1
        done <EUR.${p}.Fst
       p=`expr $p + 1`
       k=0
done

Untested
This User Gave Thanks to jgt For This Post:
# 3  
You didn't tell us if there is any naming convention on the file names, so it is difficult to suggest a loop where you don't have to name the possible file names but instead generate them. In any case the loop you use should NOT be a for-loop, because it can break if a file glob expands to something you didn't foresee.

Instead, use a while-loop, like this:

Code:
ls <some-file-glob> | while read filename ; do
     # do something with "$filename"
done

To add a file to another without the first two lines you can use "sed":

Code:
sed -n '3,$ p' inputfile >> collectfile

So, putting it together, your script could look like this - you will have to supply the correct file glob instead of "*input", which i used for demonstration purposes:

Code:
#! /bin/ksh

typeset outfile="/path/to/output"
typeset infile=""

rm -rf "$outfile" 2>/dev/null >/dev/null

ls *input | while read infile ; do
     if [ -e "$outfile" ] ; then
          sed -n '3,$ p' "$infile" >> "$outfile"
     else
          cp "$infile" "$outfile"
     fi
done

This will copy the first input-file (preserving the 2-line header) and cut off the header of any following file. "if [ -e <filename> ]" means: if "<filename>" exists and is a regular file. Because any eventually existing output file is deleted prior to the loop this is not the case with the first file.

I hope this helps.

bakunin
This User Gave Thanks to bakunin For This Post:
# 4  
thanks so much Bakunin for your help,
i created a filelist with all the names for the files that i want to concatenate. as you can see attached in "files_name" the names of these files is doest fellow a rules.
how can you script be adjusted to this please
thanks again
# 5  
I think that you can change the above to:
Code:
p=2
k=0 
cat file1 >>EUR.all.Fst     
while read file         
do              
while read line                
do                 
if [ k -eq 1 ]                   
   then                            echo "$line" >>EUR.all.fst
 fi             
k=1         
done <$file         
k=0 
done <file.list

This User Gave Thanks to jgt For This Post:
# 6  
Quote:
Originally Posted by biopsy
how can you script be adjusted to this please
If you already have the list in a file, just "cat" it into the loop. The loop just takes one file name in every run, so as long as you provide this one filename all is fine. "ls" would have provided such a list of file names and "cat /path/to/yourlist" can too. Therefore replace:

Code:
# ls *input | while read infile ; do
cat /path/to/your/file | while read infile ; do

This will do the trick, just modify the script above.

I hope this helps.

bakunin
This User Gave Thanks to bakunin For This Post:
# 7  
Just shell and cat:
Code:
{
  read filename
  cat "$filename"
  while read filename
  do
     {
       read                 # use "read; read" to skip two header lines
       cat
     } < "$filename"
  done
} < file_name.txt > concatenated_file

This User Gave Thanks to Scrutinizer For This Post:
 

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Science: Mathematics
Difficulty: Hard
The binary number "101001101" is equivalent to the Decimal number "334"
True or False?

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Concatenate files and delete source files. Also have to add a comment.

- Concatenate files and delete source files. Also have to add a comment. - I need to concatenate 3 files which have the same characters in the beginning and have to remove those files and add a comment and the end. Example: cat REJ_FILE_ABC.txt REJ_FILE_XYZ.txt REJ_FILE_PQR.txt >... (0 Replies)
Discussion started by: eskay
0 Replies

2. UNIX for Dummies Questions & Answers

Concatenate files

Hi I am trying to learn linux step by step an i am wondering can i use cat command for concatenate files but i want to place context of file1 to a specific position in file2 place of file 2 and not at the end as it dose on default? Thank you. (3 Replies)
Discussion started by: iliya24
3 Replies

3. UNIX for Dummies Questions & Answers

Concatenate Several Files to One

Hi All, Need your help. I will need to concatenate around 100 files but each end of the file I will need to insert my name DIRT1228 on each of the file and before the next file is added and arrived with just one file for all the 100files. Appreciate your time. Dirt (6 Replies)
Discussion started by: dirt1228
6 Replies

4. Fedora

Concatenate Numerous Files

Hey! I wanted to find a text version of the Bible for purposes of grepping. The only files I could find, (in the translation I wanted), were Old Testament.txt and New Testament.txt. I thought, "fine, I'll just concatenate those two, no problemo." But when I unpacked them, turns out they had each... (22 Replies)
Discussion started by: sudon't
22 Replies

5. Shell Programming and Scripting

Concatenate files

I have a file named "file1" which has the following data 10000 20000 30000 And I have a file named "file2" which has the following data ABC DEF XYZ My output should be 10000ABC 20000DEF (3 Replies)
Discussion started by: bobby1015
3 Replies

6. Shell Programming and Scripting

Concatenate files

Hi, I want to create a batch(bash) file to combine 23 files together. These files have the same extension. I want the final file is save to a given folder. Once it is done it will delete the 23 files. Thanks for help. Need script. (6 Replies)
Discussion started by: zhshqzyc
6 Replies

7. Shell Programming and Scripting

Concatenate files

I have directory structure sales_only under which i have multiple directories for each dealer example: ../../../Sales_Only/xxx_Dealer ../../../Sales_Only/yyy_Dealer ../../../Sales_Only/zzz_Dealer Every day i have one file produce under each directory when the process runs. The requirement... (3 Replies)
Discussion started by: mohanmuthu
3 Replies

8. Shell Programming and Scripting

How to concatenate 2 files using awk?

I have to select a field ($8) from file1, then add an entry to the file2 in the correct format: /tmp> cat file1 3001-233 Invalid admin value in /etc/security/group for "imnadm" 3001-233 Invalid admin value in /etc/security/group for "perfmon" 3001-233 Invalid admin value in... (2 Replies)
Discussion started by: pdtak
2 Replies

9. UNIX for Dummies Questions & Answers

How to concatenate all files.

Hi, I'm totally new to Unix. I'm an MVS mainframer but ran into a situation where a Unix server I have available will help me. I want to be able to remotely connect to another server using FTP, login and MGET all files from it's root or home directory, logout, then login as a different user and do... (1 Reply)
Discussion started by: s80bob
1 Replies

Featured Tech Videos