Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Find and modify a huge file
Top Forums Shell Programming and Scripting Find and modify a huge file Post 302995830 by Corona688 on Thursday 13th of April 2017 11:34:32 AM
Old 04-13-2017
Yes, editing a huge file once as opposed to editing a huge file n times for n lines would certainly be preferable!

This should work efficiently for anywhere up to millions of sequences listed in code.txt :

Code:
$ awk -F';' 'NR==FNR { A[$1]=$0 ; next } ; /^>/ && ($1 in A) { $1=A[$1] } 1' code.txt seq.txt

>KF1.8.1;code=D0:B;D1:P;D2:E;D3:C;D4:H;D5:S_(1);
010011001011100010101110000000
>DF1.6.1;code=D0:B;D1:D;D2:F;D3:C;D4:F;D5:S_(1);
0101000010111010101011111100
>XC1.3.7;code=D0:A;D1:D;D2:E;D3:C;D4:H;D5:H;
010110101011101010110000011
>GG5.1.1;code=D0:A;D1:D;D2:E;D3:C;D4:F;D5:H;
0100011010111010101110001101
>HK1.2.2;code=D0:A;D1:F;D2:F;D3:C;D4:H;D5:K_[23];
010000111011101101001110001010
0101011

$

It works because awk has associative arrays, you can do ARRAY["something"]="ABCD". And NR==FNR means 'do this only for the first file listed'. So it reads the entire list into an associative array, then reads through the huge file hunting for relevant lines, substituting where appropriate, then printing everything.
This User Gave Thanks to Corona688 For This Post:
GDC
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

insert a header in a huge data file without using an intermediate file

I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract if i use sed, i have to redirect the output to a seperate file like sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat the same is true for awk and... (10 Replies)
Discussion started by: deepaktanna
10 Replies

2. Shell Programming and Scripting

Modify a perl script to find and count

Hello all !I have two sets of folders that have IP address from two sources.The below perl script I was working with needs some corrections.I am looking for the perl script to identify and count what IP address are found to be duplicated between both files.The format from both files are the same... (4 Replies)
Discussion started by: richsark
4 Replies

3. Shell Programming and Scripting

Help on splitting this huge file

Hi , i have files coming in my system which are very huge in MB and GBs, all these files are in a single line, there is no newline character. I need to get only last 700 bytes of these files, of this i am splitting the files by "split -b 700 filename" but this gives all the splitted... (2 Replies)
Discussion started by: Prateek007
2 Replies

4. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

5. AIX

find command modify the output

Hello All, I am new to this shell scripting , I wanted to modify the output of my find command such that it does not display the path but only file names , for example I am searching for the files which are modified in the last 24 hours which is find /usr/monitor/text/ -type f -mtime... (3 Replies)
Discussion started by: raokl
3 Replies

6. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

7. UNIX for Dummies Questions & Answers

My file system is 100%, can't find the huge file

Please help. My file system is 100%, I can't seem to find what is taking so much space. The total hard drive space is 150Gig free but I got nothing now. I did to this to find the big file but it's taking so much time. Is there any other way? du -ah / | more find ./ -size +200M... (3 Replies)
Discussion started by: samnyc
3 Replies

8. UNIX for Dummies Questions & Answers

Need to modify a delimited file using UNIX commands. Please find description

i have a '|' delimited file having 4 fields. now i want to sort the data by combination of first three fields without changing order of 4th field. input file looks like this: 3245|G|kop|45 1329|A|uty|76 9878|K|wer|12 3245|G|kop|15 1329|A|uty|56 9878|K|wer|2 3245|G|kop|105... (4 Replies)
Discussion started by: ankurgoyal2408
4 Replies

9. Shell Programming and Scripting

Find modify and delete files

hi every one. one of my friends has writen this script and send it to me. this script can find files that add-delete-modify and also send an alert by email i'm not catch all part of it. can anyone explain me how this work #!/bin/bash START="a.txt" END="b.txt" DIFF="c.txt" mv ${START}... (4 Replies)
Discussion started by: nimafire
4 Replies

10. UNIX for Beginners Questions & Answers

Search for word in huge logfile and need to continue to print few lines from that line til find date

Guys i need an idea for one logic..in shell scripting am struggling with a logic...So the thing is... i need to search for a word in a huge log file and i need to continue to print few more lines from that line and the consecutive line has to end when it finds the line with date..because i know... (1 Reply)
Discussion started by: Prathi
1 Replies

LEARN ABOUT REDHAT

dos2unix

dos2unix(1)						      General Commands Manual						       dos2unix(1)

NAME

       dos2unix - DOS/MAC to UNIX text file format converter

SYNOPSYS

       dos2unix [options] [-c convmode] [-o file ...] [-n infile outfile ...]

       Options:

       [-hkqV] [--help] [--keepdate] [--quiet] [--version]

DESCRIPTION

       This manual page documents dos2unix, the program that converts plain text files in DOS/MAC format to UNIX format.

OPTIONS

       The following options are available:

       -h --help
	      Print online help.

       -k --keepdate
	      Keep the date stamp of output file same as input file.

       -q --quiet
	      Quiet mode. Suppress all warning and messages.

       -V --version
	      Prints version information.

       -c --convmode convmode
	      Sets conversion mode. Simulates dos2unix under SunOS.

       -o --oldfile file ...
	      Old file mode. Convert the file and write output to it. The program default to run in this mode. Wildcard names may be used.

       -n --newfile infile outfile ...
	      New  file  mode.	Convert the infile and write output to outfile. File names must be given in pairs and wildcard names should NOT be
	      used or you WILL lost your files.

EXAMPLES

       Get input from stdin and write output to stdout.

	      dos2unix

       Convert and replace a.txt. Convert and replace b.txt.

	      dos2unix a.txt b.txt

	      dos2unix -o a.txt b.txt

       Convert and replace a.txt in ASCII conversion mode.  Convert and replace b.txt in ISO conversion mode.  Convert	c.txt  from  Mac  to  Unix
       ascii format.

	      dos2unix a.txt -c iso b.txt

	      dos2unix -c ascii a.txt -c iso b.txt

	      dos2unix -c mac a.txt b.txt

       Convert and replace a.txt while keeping original date stamp.

	      dos2unix -k a.txt

	      dos2unix -k -o a.txt

       Convert a.txt and write to e.txt.

	      dos2unix -n a.txt e.txt

       Convert a.txt and write to e.txt, keep date stamp of e.txt same as a.txt.

	      dos2unix -k -n a.txt e.txt

       Convert and replace a.txt. Convert b.txt and write to e.txt.

	      dos2unix a.txt -n b.txt e.txt

	      dos2unix -o a.txt -n b.txt e.txt

       Convert c.txt and write to e.txt. Convert and replace a.txt.  Convert and replace b.txt. Convert d.txt and write to f.txt.

	      dos2unix -n c.txt e.txt -o a.txt b.txt -n d.txt f.txt

DIAGNOSTICS

BUGS

       The program does not work properly under MSDOS in stdio processing mode.  If you know why is that so, please tell me.

AUTHORS

       Benjamin Lin - <blin@socs.uts.edu.au>

       Bernd Johannes Wuebben (mac2unix mode) <wuebben@kde.org>

MISCELLANY

       Tested environment:

	      Linux 1.2.0 with GNU C 2.5.8

	      SunOS 4.1.3 with GNU C 2.6.3

	      MS-DOS 6.20 with Borland C++ 4.02

       Suggestions and bug reports are welcome.

SEE ALSO

       unix2dos(1) mac2unix(1)

1995.03.31							   dos2unix v3.0						       dos2unix(1)
All times are GMT -4. The time now is 05:34 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy