Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Find and modify a huge file
Top Forums Shell Programming and Scripting Find and modify a huge file Post 302995824 by GDC on Thursday 13th of April 2017 09:20:16 AM
Old 04-13-2017
Find and modify a huge file
Dear Forum,

I have a rather large file with a few million lines looking like this:

Code:
head -n 5 seq.txt
>KF1.8.1
010011001011100010101110000000
>DF1.6.1
0101000010111010101011111100
>XC1.3.7
010110101011101010110000011
>GG5.1.1
0100011010111010101110001101
>HK1.2.2
010000111011101101001110001010
0101011

In this file the lines can be split into different records with a name (starting with >) and the encoded information/sequence (001010...) associated with the header. Now, I need to add some code to the header according to the following file:

Code:
head -n 5 code.txt
>KF1.8.1;code=D0:B;D1:P;D2:E;D3:C;D4:H;D5:S_(1);
>DF1.6.1;code=D0:B;D1:D;D2:F;D3:C;D4:F;D5:S_(1);
>XC1.3.7;code=D0:A;D1:D;D2:E;D3:C;D4:H;D5:H;
>GG5.1.1;code=D0:A;D1:D;D2:E;D3:C;D4:F;D5:H;
>HK1.2.2;code=D0:A;D1:F;D2:F;D3:C;D4:H;D5:K_[23];

The results should look like this:

Code:
head -n 11 res.txt
>KF1.8.1;code=D0:B;D1:P;D2:E;D3:C;D4:H;D5:S_(1);
010011001011100010101110000000
>DF1.6.1;code=D0:B;D1:D;D2:F;D3:C;D4:F;D5:S_(1);
0101000010111010101011111100
>XC1.3.7;code=D0:A;D1:D;D2:E;D3:C;D4:H;D5:H;
0100011010111010101110001101
>GG5.1.1;code=D0:A;D1:D;D2:E;D3:C;D4:F;D5:H;
0100011010111010101110001101
>HK1.2.2;code=D0:A;D1:F;D2:F;D3:C;D4:H;D5:K_[23];
010000111011101101001110001010
0101011

The two files (seq.txt, code.txt) are not sorted but the number of records are identical.

I could use sed to change one record header at a time
Code:
sed 's/>KF1.8.1/>KF1.8.1;code=D0:B;D1:P;D2:E;D3:C;D4:H;D5:S_(1);/g' seq.txt

or maybe write it into a file an execute it

Code:
while read code
do
  record=`echo $code | cut -d';' -f 1`
  echo "sed 's/$record/$code/g' seq.txt" >> all.txt
done < code.txt

chmod a+x all.txt
./all.txt

but this might take some time. Does anybody have a faster and maybe more elegant way for me to modify the record headers?

Thanks for all your help!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

insert a header in a huge data file without using an intermediate file

I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract if i use sed, i have to redirect the output to a seperate file like sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat the same is true for awk and... (10 Replies)
Discussion started by: deepaktanna
10 Replies

2. Shell Programming and Scripting

Modify a perl script to find and count

Hello all !I have two sets of folders that have IP address from two sources.The below perl script I was working with needs some corrections.I am looking for the perl script to identify and count what IP address are found to be duplicated between both files.The format from both files are the same... (4 Replies)
Discussion started by: richsark
4 Replies

3. Shell Programming and Scripting

Help on splitting this huge file

Hi , i have files coming in my system which are very huge in MB and GBs, all these files are in a single line, there is no newline character. I need to get only last 700 bytes of these files, of this i am splitting the files by "split -b 700 filename" but this gives all the splitted... (2 Replies)
Discussion started by: Prateek007
2 Replies

4. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

5. AIX

find command modify the output

Hello All, I am new to this shell scripting , I wanted to modify the output of my find command such that it does not display the path but only file names , for example I am searching for the files which are modified in the last 24 hours which is find /usr/monitor/text/ -type f -mtime... (3 Replies)
Discussion started by: raokl
3 Replies

6. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

7. UNIX for Dummies Questions & Answers

My file system is 100%, can't find the huge file

Please help. My file system is 100%, I can't seem to find what is taking so much space. The total hard drive space is 150Gig free but I got nothing now. I did to this to find the big file but it's taking so much time. Is there any other way? du -ah / | more find ./ -size +200M... (3 Replies)
Discussion started by: samnyc
3 Replies

8. UNIX for Dummies Questions & Answers

Need to modify a delimited file using UNIX commands. Please find description

i have a '|' delimited file having 4 fields. now i want to sort the data by combination of first three fields without changing order of 4th field. input file looks like this: 3245|G|kop|45 1329|A|uty|76 9878|K|wer|12 3245|G|kop|15 1329|A|uty|56 9878|K|wer|2 3245|G|kop|105... (4 Replies)
Discussion started by: ankurgoyal2408
4 Replies

9. Shell Programming and Scripting

Find modify and delete files

hi every one. one of my friends has writen this script and send it to me. this script can find files that add-delete-modify and also send an alert by email i'm not catch all part of it. can anyone explain me how this work #!/bin/bash START="a.txt" END="b.txt" DIFF="c.txt" mv ${START}... (4 Replies)
Discussion started by: nimafire
4 Replies

10. UNIX for Beginners Questions & Answers

Search for word in huge logfile and need to continue to print few lines from that line til find date

Guys i need an idea for one logic..in shell scripting am struggling with a logic...So the thing is... i need to search for a word in a huge log file and i need to continue to print few more lines from that line and the consecutive line has to end when it finds the line with date..because i know... (1 Reply)
Discussion started by: Prathi
1 Replies

LEARN ABOUT CENTOS

locale::codes::langext

Locale::Codes::LangExt(3)				User Contributed Perl Documentation				 Locale::Codes::LangExt(3)

NAME

       Locale::Codes::LangExt - standard codes for language extension identification

SYNOPSIS

	  use Locale::Codes::LangExt;

	  $lext = code2langext('acm');		       # $lext gets 'Mesopotamian Arabic'
	  $code = langext2code('Mesopotamian Arabic'); # $code gets 'acm'

	  @codes   = all_langext_codes();
	  @names   = all_langext_names();

DESCRIPTION

       The "Locale::Codes::LangExt" module provides access to standard codes used for identifying language extensions, such as those as defined in
       the IANA language registry.

       Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default IANA language
       registry codes will be used.

SUPPORTED CODE SETS

       There are several different code sets you can use for identifying language extensions. A code set may be specified using either a name, or
       a constant that is automatically exported by this module.

       For example, the two are equivalent:

	  $lext = code2langext('acm','alpha');
	  $lext = code2langext('acm',LOCALE_LANGEXT_ALPHA);

       The codesets currently supported are:

       alpha
	   This is the set of three-letter (lowercase) codes from the IANA language registry, such as 'acm' for Mesopotamian Arabic.

	   This is the default code set.

ROUTINES

       code2langext ( CODE [,CODESET] )
       langext2code ( NAME [,CODESET] )
       langext_code2code ( CODE ,CODESET ,CODESET2 )
       all_langext_codes ( [CODESET] )
       all_langext_names ( [CODESET] )
       Locale::Codes::LangExt::rename_langext  ( CODE ,NEW_NAME [,CODESET] )
       Locale::Codes::LangExt::add_langext  ( CODE ,NAME [,CODESET] )
       Locale::Codes::LangExt::delete_langext  ( CODE [,CODESET] )
       Locale::Codes::LangExt::add_langext_alias  ( NAME ,NEW_NAME )
       Locale::Codes::LangExt::delete_langext_alias  ( NAME )
       Locale::Codes::LangExt::rename_langext_code  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangExt::add_langext_code_alias  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangExt::delete_langext_code_alias  ( CODE [,CODESET] )
	   These routines are all documented in the Locale::Codes::API man page.

SEE ALSO

       Locale::Codes
	   The Locale-Codes distribution.

       Locale::Codes::API
	   The list of functions supported by this module.

       http://www.iana.org/assignments/language-subtag-registry
	   The IANA language subtag registry.

AUTHOR

       See Locale::Codes for full author history.

       Currently maintained by Sullivan Beck (sbeck@cpan.org).

COPYRIGHT

	  Copyright (c) 2011-2013 Sullivan Beck

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.16.3							    2013-02-27						 Locale::Codes::LangExt(3)
All times are GMT -4. The time now is 02:19 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy