Concatenate 560 files in one | Unix Linux Forums | UNIX for Dummies Questions & Answers

  Go Back    


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

Concatenate 560 files in one

UNIX for Dummies Questions & Answers


Tags
cat, concatenate

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 01-26-2013
biopsy biopsy is offline
Registered User
 
Join Date: Jun 2012
Last Activity: 23 April 2013, 4:52 AM EDT
Posts: 24
Thanks: 10
Thanked 0 Times in 0 Posts
Concatenate 560 files in one

HI all,
could please help me in this code. I have 560 files containing the same columns but different rows. i want to concatenate all these files in one big file. i want to keep the header of the first file then add the dat from other files horizontally.
the name of my files contains 2 variables : chunk number and chromosome number. this an example

code:
IMP_chunk11-chr6_file

i have tried this code but it works only for one variable and 22 files. anu suggestion to adapt it to my case

code :



Code:
for p in 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
do
more +2 EUR.chr${p}.Fst > EUR.chr${p}.Fst.tmp
done

cp EUR.chr1.Fst EUR.chr1.Fst.tmp

cat EUR.chr*.Fst.tmp > EUR.all.Fst

thank you veru much

Moderator's Comments:
edit by bakunin: Please reread the rules, CODE-tags are not optional when posting code. Thank you.

Last edited by bakunin; 01-26-2013 at 05:29 PM..
Sponsored Links
    #2  
Old 01-26-2013
jgt's Avatar
jgt jgt is offline Forum Advisor  
Registered User
 
Join Date: Apr 2007
Last Activity: 29 October 2014, 3:49 PM EDT
Location: 44.21.48N 80.50.15W
Posts: 1,560
Thanks: 1
Thanked 159 Times in 151 Posts

Code:
p=2
k=0
cat EUR.1.fst >EUR.all.fst
    while [ p -lt 561 ]
        do
             while read line
               do
                if [ k -eq 1 ]
                  then
                           echo "$line" >>EUR.all.fst
                  fi
            k=1
        done <EUR.${p}.Fst
       p=`expr $p + 1`
       k=0
done

Untested
The Following User Says Thank You to jgt For This Useful Post:
biopsy (01-26-2013)
Sponsored Links
    #3  
Old 01-26-2013
bakunin bakunin is offline Forum Staff  
Bughunter Extraordinaire
 
Join Date: May 2005
Last Activity: 31 October 2014, 2:56 PM EDT
Location: In the leftmost byte of /dev/kmem
Posts: 4,301
Thanks: 45
Thanked 827 Times in 654 Posts
You didn't tell us if there is any naming convention on the file names, so it is difficult to suggest a loop where you don't have to name the possible file names but instead generate them. In any case the loop you use should NOT be a for-loop, because it can break if a file glob expands to something you didn't foresee.

Instead, use a while-loop, like this:


Code:
ls <some-file-glob> | while read filename ; do
     # do something with "$filename"
done

To add a file to another without the first two lines you can use "sed":


Code:
sed -n '3,$ p' inputfile >> collectfile

So, putting it together, your script could look like this - you will have to supply the correct file glob instead of "*input", which i used for demonstration purposes:


Code:
#! /bin/ksh

typeset outfile="/path/to/output"
typeset infile=""

rm -rf "$outfile" 2>/dev/null >/dev/null

ls *input | while read infile ; do
     if [ -e "$outfile" ] ; then
          sed -n '3,$ p' "$infile" >> "$outfile"
     else
          cp "$infile" "$outfile"
     fi
done

This will copy the first input-file (preserving the 2-line header) and cut off the header of any following file. "if [ -e <filename> ]" means: if "<filename>" exists and is a regular file. Because any eventually existing output file is deleted prior to the loop this is not the case with the first file.

I hope this helps.

bakunin
The Following User Says Thank You to bakunin For This Useful Post:
biopsy (01-26-2013)
    #4  
Old 01-26-2013
biopsy biopsy is offline
Registered User
 
Join Date: Jun 2012
Last Activity: 23 April 2013, 4:52 AM EDT
Posts: 24
Thanks: 10
Thanked 0 Times in 0 Posts
thanks so much Bakunin for your help,
i created a filelist with all the names for the files that i want to concatenate. as you can see attached in "files_name" the names of these files is doest fellow a rules.
how can you script be adjusted to this please
thanks again
Attached Files
File Type: txt files_name.txt (11.6 KB, 15 views)
Sponsored Links
    #5  
Old 01-26-2013
jgt's Avatar
jgt jgt is offline Forum Advisor  
Registered User
 
Join Date: Apr 2007
Last Activity: 29 October 2014, 3:49 PM EDT
Location: 44.21.48N 80.50.15W
Posts: 1,560
Thanks: 1
Thanked 159 Times in 151 Posts
I think that you can change the above to:

Code:
p=2
k=0 
cat file1 >>EUR.all.Fst     
while read file         
do              
while read line                
do                 
if [ k -eq 1 ]                   
   then                            echo "$line" >>EUR.all.fst
 fi             
k=1         
done <$file         
k=0 
done <file.list

The Following User Says Thank You to jgt For This Useful Post:
biopsy (01-27-2013)
Sponsored Links
    #6  
Old 01-26-2013
bakunin bakunin is offline Forum Staff  
Bughunter Extraordinaire
 
Join Date: May 2005
Last Activity: 31 October 2014, 2:56 PM EDT
Location: In the leftmost byte of /dev/kmem
Posts: 4,301
Thanks: 45
Thanked 827 Times in 654 Posts
Quote:
Originally Posted by biopsy View Post
how can you script be adjusted to this please
If you already have the list in a file, just "cat" it into the loop. The loop just takes one file name in every run, so as long as you provide this one filename all is fine. "ls" would have provided such a list of file names and "cat /path/to/yourlist" can too. Therefore replace:


Code:
# ls *input | while read infile ; do
cat /path/to/your/file | while read infile ; do

This will do the trick, just modify the script above.

I hope this helps.

bakunin
The Following User Says Thank You to bakunin For This Useful Post:
biopsy (01-27-2013)
Sponsored Links
    #7  
Old 01-27-2013
Scrutinizer's Avatar
Scrutinizer Scrutinizer is offline Forum Staff  
Moderator
 
Join Date: Nov 2008
Last Activity: 1 November 2014, 3:43 AM EDT
Location: Amsterdam
Posts: 9,567
Thanks: 286
Thanked 2,432 Times in 2,179 Posts
Just shell and cat:

Code:
{
  read filename
  cat "$filename"
  while read filename
  do
     {
       read                 # use "read; read" to skip two header lines
       cat
     } < "$filename"
  done
} < file_name.txt > concatenated_file

The Following User Says Thank You to Scrutinizer For This Useful Post:
biopsy (01-27-2013)
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Concatenate files bobby1015 Shell Programming and Scripting 3 02-24-2012 11:38 AM
Concatenate files zhshqzyc Shell Programming and Scripting 6 02-19-2011 04:35 PM
Concatenate files mohanmuthu Shell Programming and Scripting 3 12-14-2010 09:49 AM
How to concatenate all files. s80bob UNIX for Dummies Questions & Answers 1 08-22-2005 04:20 PM



All times are GMT -4. The time now is 04:19 AM.