Sponsored Content
Full Discussion: Find and modify a huge file
Top Forums Shell Programming and Scripting Find and modify a huge file Post 302995824 by GDC on Thursday 13th of April 2017 09:20:16 AM
Old 04-13-2017
Find and modify a huge file

Dear Forum,

I have a rather large file with a few million lines looking like this:

Code:
head -n 5 seq.txt
>KF1.8.1
010011001011100010101110000000
>DF1.6.1
0101000010111010101011111100
>XC1.3.7
010110101011101010110000011
>GG5.1.1
0100011010111010101110001101
>HK1.2.2
010000111011101101001110001010
0101011

In this file the lines can be split into different records with a name (starting with >) and the encoded information/sequence (001010...) associated with the header. Now, I need to add some code to the header according to the following file:

Code:
head -n 5 code.txt
>KF1.8.1;code=D0:B;D1:P;D2:E;D3:C;D4:H;D5:S_(1);
>DF1.6.1;code=D0:B;D1:D;D2:F;D3:C;D4:F;D5:S_(1);
>XC1.3.7;code=D0:A;D1:D;D2:E;D3:C;D4:H;D5:H;
>GG5.1.1;code=D0:A;D1:D;D2:E;D3:C;D4:F;D5:H;
>HK1.2.2;code=D0:A;D1:F;D2:F;D3:C;D4:H;D5:K_[23];

The results should look like this:

Code:
head -n 11 res.txt
>KF1.8.1;code=D0:B;D1:P;D2:E;D3:C;D4:H;D5:S_(1);
010011001011100010101110000000
>DF1.6.1;code=D0:B;D1:D;D2:F;D3:C;D4:F;D5:S_(1);
0101000010111010101011111100
>XC1.3.7;code=D0:A;D1:D;D2:E;D3:C;D4:H;D5:H;
0100011010111010101110001101
>GG5.1.1;code=D0:A;D1:D;D2:E;D3:C;D4:F;D5:H;
0100011010111010101110001101
>HK1.2.2;code=D0:A;D1:F;D2:F;D3:C;D4:H;D5:K_[23];
010000111011101101001110001010
0101011

The two files (seq.txt, code.txt) are not sorted but the number of records are identical.

I could use sed to change one record header at a time
Code:
sed 's/>KF1.8.1/>KF1.8.1;code=D0:B;D1:P;D2:E;D3:C;D4:H;D5:S_(1);/g' seq.txt

or maybe write it into a file an execute it

Code:
while read code
do
  record=`echo $code | cut -d';' -f 1`
  echo "sed 's/$record/$code/g' seq.txt" >> all.txt
done < code.txt

chmod a+x all.txt
./all.txt

but this might take some time. Does anybody have a faster and maybe more elegant way for me to modify the record headers?

Thanks for all your help!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

insert a header in a huge data file without using an intermediate file

I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract if i use sed, i have to redirect the output to a seperate file like sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat the same is true for awk and... (10 Replies)
Discussion started by: deepaktanna
10 Replies

2. Shell Programming and Scripting

Modify a perl script to find and count

Hello all !I have two sets of folders that have IP address from two sources.The below perl script I was working with needs some corrections.I am looking for the perl script to identify and count what IP address are found to be duplicated between both files.The format from both files are the same... (4 Replies)
Discussion started by: richsark
4 Replies

3. Shell Programming and Scripting

Help on splitting this huge file

Hi , i have files coming in my system which are very huge in MB and GBs, all these files are in a single line, there is no newline character. I need to get only last 700 bytes of these files, of this i am splitting the files by "split -b 700 filename" but this gives all the splitted... (2 Replies)
Discussion started by: Prateek007
2 Replies

4. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

5. AIX

find command modify the output

Hello All, I am new to this shell scripting , I wanted to modify the output of my find command such that it does not display the path but only file names , for example I am searching for the files which are modified in the last 24 hours which is find /usr/monitor/text/ -type f -mtime... (3 Replies)
Discussion started by: raokl
3 Replies

6. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

7. UNIX for Dummies Questions & Answers

My file system is 100%, can't find the huge file

Please help. My file system is 100%, I can't seem to find what is taking so much space. The total hard drive space is 150Gig free but I got nothing now. I did to this to find the big file but it's taking so much time. Is there any other way? du -ah / | more find ./ -size +200M... (3 Replies)
Discussion started by: samnyc
3 Replies

8. UNIX for Dummies Questions & Answers

Need to modify a delimited file using UNIX commands. Please find description

i have a '|' delimited file having 4 fields. now i want to sort the data by combination of first three fields without changing order of 4th field. input file looks like this: 3245|G|kop|45 1329|A|uty|76 9878|K|wer|12 3245|G|kop|15 1329|A|uty|56 9878|K|wer|2 3245|G|kop|105... (4 Replies)
Discussion started by: ankurgoyal2408
4 Replies

9. Shell Programming and Scripting

Find modify and delete files

hi every one. one of my friends has writen this script and send it to me. this script can find files that add-delete-modify and also send an alert by email i'm not catch all part of it. can anyone explain me how this work #!/bin/bash START="a.txt" END="b.txt" DIFF="c.txt" mv ${START}... (4 Replies)
Discussion started by: nimafire
4 Replies

10. UNIX for Beginners Questions & Answers

Search for word in huge logfile and need to continue to print few lines from that line til find date

Guys i need an idea for one logic..in shell scripting am struggling with a logic...So the thing is... i need to search for a word in a huge log file and i need to continue to print few more lines from that line and the consecutive line has to end when it finds the line with date..because i know... (1 Reply)
Discussion started by: Prathi
1 Replies
fmt(1)							      General Commands Manual							    fmt(1)

NAME
fmt - format text SYNOPSIS
width] [file...] DESCRIPTION
The command is a simple text formatter that fills and joins lines to produce output lines of (up to) the number of characters specified in the width option. The default width is 72. concatenates the arguments. If none are given, formats text from the standard input. Blank lines are preserved in the output, as is the spacing between words. does not fill lines beginning with a period for compatibility with Nor does it fill lines starting with Indentation is preserved in the output and input lines with differing indentation are not joined (unless is used). can also be used as an in-line text filter for the command: reformats the text between the cursor location and the end of the paragraph. Options recognizes the following options: Crown margin mode. Preserve the indentation of the first two lines within a paragraph and align the left margin of each subsequent line with that of the second line. This is useful for tagged paragraphs. Split lines only. Do not join short lines to form longer ones. This prevents sample lines of code, and other such "formatted" text, from being unduly combined. Fill output lines to up to width columns. WARNINGS
The width option is acceptable for BSD compatibility, but it may go away in future releases. SEE ALSO
nroff(1), vi(1). fmt(1)
All times are GMT -4. The time now is 11:29 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy