Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

New code for modifying text files in a folder

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting New code for modifying text files in a folder
# 1  
Old 09-23-2015
New code for modifying text files in a folder
Hi I want to create a code that can do this for all text files in a folder.

The filenames are all listed in the following syntax

Code:
UNIQUEID-LABID_[First Name 1][M] - [First Name 2][F].txt

Each file has a unique ID and a different name and the content in the file looks like this:

Code:
Kinship Analysis Report --- Likelihood Ratio
Marker	mmm.fsa			jjj.fsa			Parent/Child (LR)	
CSF1PO	10	12		10	12		1.90183	
TPOX	8	9		9	11		0.97656	
TH01	6	9.3		7	9.3		3.42466	
vWA	18			18	19		4.34783	
D16S539	10	12		9	10		2.33645	
D7S820	10	12		9	12		2.50000	
D13S317	11	12		11	12		1.48845	
D5S818	10	11		11	13		1.02459	
FGA	19	25		19	24		3.62319	
D8S1179	10	14		10	12		10.41667	
D18S51	12			12			17.24138	
D21S11	28	31		29	31		3.16456	
D3S1358	16	17		17			2.76243	
PENTA E	12	14		11	12		2.01613	
PENTA D	10	12		10	12		4.53612
							1.28E+07

I want the output to look like this:

Code:
Kinship Analysis Report --- Likelihood Ratio
Marker	First Name 1			First Name 2			Parent/Child (LR)	
CSF1PO	10	12		10	12		1.90183	
TPOX	8	9		9	11		0.97656	
TH01	6	9.3		7	9.3		3.42466	
vWA	18			18	19		4.34783	
D16S539	10	12		9	10		2.33645	
D7S820	10	12		9	12		2.50000	
D13S317	11	12		11	12		1.48845	
D5S818	10	11		11	13		1.02459	
FGA	19	25		19	24		3.62319	
D8S1179	10	14		10	12		10.41667	
D18S51	12			12			17.24138	
D21S11	28	31		29	31		3.16456	
D3S1358	16	17		17			2.76243	
PENTA E	12	14		11	12		2.01613	
PENTA D	10	12		10	12		4.53612	
AMEL	X	Y		X	X
							1.28E+07

Basically I want to replace the .fsa with the name on the filename and I want to add the line AMEL in each file where M would be X Y and F would be X X

Each folder contains about 50 plus files so a code to do it all at once would be best.

Thanks
# 2  
Old 09-24-2015
Please, test with a small sample of your given directory.

It creates a file.bk from the file that it finds in the directory, for safety reasons. If the result in the file.bk is correct, remove the
last commented command to rename the backup to the original name and run in the production/real directory

Run as:
Code:
perl kylle345_run.pl /path/to/directory/*.txt

Code:
#!/usr/bin/perl
# kylle345_run.pl

use strict;
use warnings;
use File::Copy qw(move);

# process each file given at the command line
for my $fname (@ARGV){
    my $fh;
    # open the current file    
    if(not open $fh, '<', $fname){
        # file could not be read; skip
        print STDERR "$fname could not be processed\n";
        next;
    }
    print "Processing file: $fname ...\n";

    # obtain substrings from file name if not skip
    my @names = $fname =~ /\[(.+?)\]\[\w\]/g or next;
    open my $fh_bk, '>', "$fname.bk" or next;
    my $before_last = "AMEL\tX\tY\t\tX\tX\n";

    # read the current file
    while(<$fh>){
         # only at second line substitute the xxx.fsa with parts of filename
         s/\w{3}\.fsa(\s+)\w{3}\.fsa/$names[0]$1$names[1]/ if $. == 2;
         # add a line before last
         if(eof) {
             print $fh_bk "$before_last$_";
             next;
         }
         # write to a copy ending in .bk
         print $fh_bk $_;
    }
    close $fh;
    close $fh_bk;
    # test first and if it does correctly, remove the
    # # from next line to rename backup to original name
    #move "$fname.bk", $fname;
}

Last edited by Aia; 09-24-2015 at 02:17 AM.. Reason: correct wording
# 3  
Old 09-24-2015
Try also
Code:
awk '
NR==1   {split(FILENAME, T, "[][]")
        }
NR==2   {sub ("...\.fsa", T[2])
         sub ("...\.fsa", T[6])
         EL="AMEL\tX\t" (T[4]=="M"?"Y":"X") "\t\tX\t" (T[8]=="M"?"Y":"X")
        }
NF==1   {print EL
        }

1

' 'UNIQUEID-LABID_[First Name 1][M] - [First Name 2][F].txt'

# 4  
Old 09-25-2015
Quote:
Originally Posted by Aia
Please, test with a small sample of your given directory.

It creates a file.bk from the file that it finds in the directory, for safety reasons. If the result in the file.bk is correct, remove the
last commented command to rename the backup to the original name and run in the production/real directory

Run as:
Code:
perl kylle345_run.pl /path/to/directory/*.txt

Code:
#!/usr/bin/perl
# kylle345_run.pl

use strict;
use warnings;
use File::Copy qw(move);

# process each file given at the command line
for my $fname (@ARGV){
    my $fh;
    # open the current file    
    if(not open $fh, '<', $fname){
        # file could not be read; skip
        print STDERR "$fname could not be processed\n";
        next;
    }
    print "Processing file: $fname ...\n";

    # obtain substrings from file name if not skip
    my @names = $fname =~ /\[(.+?)\]\[\w\]/g or next;
    open my $fh_bk, '>', "$fname.bk" or next;
    my $before_last = "AMEL\tX\tY\t\tX\tX\n";

    # read the current file
    while(<$fh>){
         # only at second line substitute the xxx.fsa with parts of filename
         s/\w{3}\.fsa(\s+)\w{3}\.fsa/$names[0]$1$names[1]/ if $. == 2;
         # add a line before last
         if(eof) {
             print $fh_bk "$before_last$_";
             next;
         }
         # write to a copy ending in .bk
         print $fh_bk $_;
    }
    close $fh;
    close $fh_bk;
    # test first and if it does correctly, remove the
    # # from next line to rename backup to original name
    #move "$fname.bk", $fname;
}


Thank you for the code but the error "could not be processed comes up"

---------- Post updated at 02:54 PM ---------- Previous update was at 02:54 PM ----------

Quote:
Originally Posted by RudiC
Try also
Code:
awk '
NR==1   {split(FILENAME, T, "[][]")
        }
NR==2   {sub ("...\.fsa", T[2])
         sub ("...\.fsa", T[6])
         EL="AMEL\tX\t" (T[4]=="M"?"Y":"X") "\t\tX\t" (T[8]=="M"?"Y":"X")
        }
NF==1   {print EL
        }

1

' 'UNIQUEID-LABID_[First Name 1][M] - [First Name 2][F].txt'


thanks is there a way to run the entire folder at once?

Thanks
# 5  
Old 09-25-2015
What be the output file names?

---------- Post updated at 21:51 ---------- Previous update was at 21:40 ----------

Anyhow, try
Code:
awk '
FNR==1  {if (fnm) close(fnm)
         fnm = FILENAME ".mod"
         split(FILENAME, T, "[][]")
        }
FNR==2  {sub ("...\.fsa", T[2])
         sub ("...\.fsa", T[6])
         EL="AMEL\tX\t" (T[4]=="M"?"Y":"X") "\t\tX\t" (T[8]=="M"?"Y":"X")
        }
NF==1   {print EL > fnm
        }

        {print > fnm
        }
' UNIQUEID*txt

# 6  
Old 09-25-2015
Quote:
Originally Posted by kylle345
Thank you for the code but the error "could not be processed comes up"
Yes, that's a warning message I built in to notify you for EACH files that could not be opened for reading for whatever reason (permissions, etc). Along with every notification the supposed file name is shown. Take a look and see why it cannot open it.
# 7  
Old 09-26-2015
Oddly enough I cannot seem to figure out what the issue is. Could it be the spacing (number of tabs)?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Modifying/combining two text outputs

greetings, I have what seems to be a fairly complex task at hand. I am dealing with 3 outputs from a database. a) Listing of tables. The first 3 characters are always an identifier of an application. See in part b - Example- select tablename from system.tables ABC1 ABC2 RFV1 ... (2 Replies)
Discussion started by: jeffs42885
2 Replies

2. Shell Programming and Scripting

How to add a text at the beginning of a text files in a folder?

how to add a text ( surya) at the beginning of a text files (so many) in folder text file: 111111 555555 666666 result: surya 111111 555555 666666 (3 Replies)
Discussion started by: suryanarayana
3 Replies

3. Shell Programming and Scripting

Extracting lines from text files in folder based on the numbers in another file

Hello, I have a file ff.txt that looks as follows *ABNA.txt 356 24 36 112 *AC24.txt 457 458 321 2 ABNA.txt and AC24.txt are the files in the folder named foo1. Based on the numbers in the ff.txt file, I want to extract the lines from the corresponding files in the foo1 folder and... (2 Replies)
Discussion started by: mohamad
2 Replies

4. Shell Programming and Scripting

Text Formating or Modifying

Hi Experts, I have a text exactly like below in a file: id item_id item_date prin_mkt_val --------------------------- --------------------------- ------------------------------- ------------------------ ... (1 Reply)
Discussion started by: apatil65
1 Replies

5. Shell Programming and Scripting

Need to build Shell Script to search content of a text file into a folder consist several files

Have to read one file say sourcefile containing several words and having another folder containing several files. Now read the first word of Sourcefile & search it into the folder consisting sevral files, and create another file with result. We hhave to pick the filename of the file in which... (3 Replies)
Discussion started by: mukesh.baranwal
3 Replies

6. Shell Programming and Scripting

Loop through text file > Copy Folder > Edit XML files in bulk?

I have a text file which contains lines in this format - it contains 105 lines in total, but I'm just putting 4 here to keep it short: 58571,east_ppl_ppla_por 58788,east_pcy_hd_por 58704,east_pcy_ga_por 58697,east_pcy_pcybs_por It's called id_key.txt I have a sample folder called... (9 Replies)
Discussion started by: biscuitcreek
9 Replies

7. UNIX for Dummies Questions & Answers

Separate text files in a folder by word count

Hi, been searching Google for like an hour and I doubt I got the right keywords so might as well ask here. What I need: Before: Article1.txt 564 Article2.txt 799 Article3.txt 349 Article4.txt 452 * Separate files over 400 wordcount * After: Article1.txt 564... (3 Replies)
Discussion started by: Richard2000
3 Replies

8. Shell Programming and Scripting

Find all text files in folder and then copy to a new folder

Hi all, *I use Uwin and Cygwin emulator. I´m trying to search for all text files in the current folder (C/Files) and its sub folders using find -depth -name "*.txt" The above command worked for me, but now I would like to copy all found text files to a new folder (C/Files/Text) with ... (4 Replies)
Discussion started by: cgkmal
4 Replies

9. Shell Programming and Scripting

Help need in modifying the text of .txt file

Hi All, I've written a shell script in which i defined two varibles for example: str=1.0.0.15 timeStamp=2008.03.08 now using this varibles i need to modify a text file. The text content looks like this ************************ * packageNumber : 1.0.0.14 * * date :... (2 Replies)
Discussion started by: vinna
2 Replies

10. UNIX for Dummies Questions & Answers

grep multiple text files in folder into 1 text file?

How do I use the grep command to take mutiple text files in a folder and make one huge text file out of them. I'm using Mac OS X and can not find a text tool that does it so I figured I'd resort to the BSD Unix CLI for a solution... there are 5,300 files that I want to write to one huge file so... (7 Replies)
Discussion started by: coppertone
7 Replies
Login or Register to Ask a Question