Sponsored Content
Top Forums Shell Programming and Scripting How to quickly substitute pattern within certain range of a huge file? Post 303017418 by yifangt on Tuesday 15th of May 2018 04:26:30 PM
Old 05-15-2018
How to quickly substitute pattern within certain range of a huge file?

I have big files (some are >300GB!) that need substitution for some patterns, for example, change Multiple Spaces into Tab. I used this oneliner:
Code:
sed '1,18s/ \{1,\}/\t/g' infile_big.sam > outfile_big.sam

but it seems very slow as the job is still running after 24 hours! In this example, only the first 18 rows need be changed, and the rest is untouched.
Is there any better way to do the job quickly? I'm using GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu) on Linux 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3+deb9u1 (2017-12-23) x86_64 GNU/Linux.
Thanks a lot!

Last edited by yifangt; 05-15-2018 at 05:42 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

need solution for this quickly. please quickly.

Write a nawk script that will produce the following report: ***FIRST QUARTERLY REPORT*** ***CAMPAIGN 2004 CONTRIBUTIONS*** ------------------------------------------------------------------------- NAME PHONE Jan | ... (5 Replies)
Discussion started by: p.palakj.shah
5 Replies

2. Shell Programming and Scripting

print range between two patterns if it contains a pattern within the range

I want to print between the range two patterns if a particular pattern is present in between the two patterns. I am new to Unix. Any help would be greatly appreciated. e.g. Pattern1 Bombay Calcutta Delhi Pattern2 Pattern1 Patna Madras Gwalior Delhi Pattern2 Pattern1... (2 Replies)
Discussion started by: joyan321
2 Replies

3. Shell Programming and Scripting

sed pattern range

Hi guys, trying to replace a '#' with a ' ' (space) but only between the brackets '(' and ')' N="text1#text2#text3(var1#var2#var3)" N=`echo $N |sed '/(/,/) s/#. //'` echo $N Looking for an output of "text1#text2#text3(var1 var2 var3)" Any ideas? (15 Replies)
Discussion started by: mikepegg
15 Replies

4. Shell Programming and Scripting

How to combine lines within range of pattern

I've a file say having line 1 line 2 (NP line 3 line 4 line 5) line 6 I want to combine lines starting from (NP and ending with ) then it will look like line 1 line 2 (NP line3 line4 line5) line 6 I tried using sed '/(NP/,/)$/ s/\n/ /' but it's not working. Any help please? ... (8 Replies)
Discussion started by: neg
8 Replies

5. Shell Programming and Scripting

Removing tmp file too quickly?

Still trying to get the basics down and I would like a different solution to what I'm currently doing and a better understanding of why it's happening. I've written a simple backup script that tars individual directories and then dumps them to a NFS drive. STDERR is being dumped into a process... (2 Replies)
Discussion started by: mandelbrot333
2 Replies

6. Shell Programming and Scripting

Pattern Matchin Huge File

Hi Experts, I've issue with the huge file. My requirement is I need to search a pattern between the 155-156 position and if its match's to 31 or 36 then need to route that to a new separate files. The main file has around 1459328 line and 2 GB in size. I tired with the below code which take... (9 Replies)
Discussion started by: senthil.ak
9 Replies

7. Shell Programming and Scripting

Print pattern range to a new file

Hi Everyone! I really appreciate all of your help, I'm learning so much, can't wait until I get good enough to start answering questions! I have a problem ... from one large file, I'd like to create multiple new files for each pattern block beginning with /^ISA/ ending with /^IEA/ ... (2 Replies)
Discussion started by: verge
2 Replies

8. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

9. Shell Programming and Scripting

sed Range Pattern and 2 lines before Start Pattern

Hi all, I have been searching all over Google but I am unable to find a solution for a particular result that I am trying to achieve. Consider the following input: 1 2 3 4 5 B4Srt1--Variable-0000 B4Srt2--Variable-1111 Srt 6 7 8 9 10 End (3 Replies)
Discussion started by: y2jacky
3 Replies

10. Shell Programming and Scripting

Extract range from config file matching pattern

I have config file like this: server_name xx opt1 opt2 opt3 suboptions1 #suboptions - disabled suboptions2 pattern suboptions3 server_name yy opt1 opt2 opt3 suboptions1 pattern #suboptions - disabled suboptions2 So basically I want to extract the server... (1 Reply)
Discussion started by: nemesis911
1 Replies
SHTOOL-SUBST.TMP(1)					      GNU Portable Shell Tool					       SHTOOL-SUBST.TMP(1)

NAME
shtool-subst - GNU shtool sed(1) substitution operations SYNOPSIS
shtool subst [-v|--verbose] [-t|--trace] [-n|--nop] [-w|--warning] [-q|--quiet] [-s|--stealth] [-i|--interactive] [-b|--backup ext] [-e|--exec cmd] [-f|--file cmd-file] [file] [file ...] DESCRIPTION
This command applies one or more sed(1) substitution operations to stdin or any number of files. OPTIONS
The following command line options are available. -v, --verbose Display some processing information. -t, --trace Enable the output of the essential shell commands which are executed. -n, --nop No operation mode. Actual execution of the essential shell commands which would be executed is suppressed. -w, --warning Show warning on substitution operation resulting in no content change on every file. The default is to show a warning on substitution operations resulted in no content change on all files. -q, --quiet Suppress warning on substitution operation resulting in no content change. -s, --stealth Stealth operation. Preserve timestamp on file. -i, --interactive Enter interactive mode where the user has to approve each operation. -b, --backup ext Preserve backup of original file using file name extension ext. Default is to overwrite the original file. -e, --exec cmd Specify sed(1) command directly. -f, --file cmd-file Read sed(1) command from file. EXAMPLE
# shell script shtool subst -i -e 's;(c) ([0-9]*)-2000;(c) 1-2001;' *.[ch] # RPM spec-file %install shtool subst -v -n -e 's;^(prefix=).*;1 $RPM_BUILD_ROOT%{_prefix};g' -e 's;^(sysconfdir=).*;1 $RPM_BUILD_ROOT%{_prefix}/etc;g' `find . -name Makefile -print` make install HISTORY
The GNU shtool subst command was originally written by Ralf S. Engelschall <rse@engelschall.com> in 2001 for GNU shtool. It was prompted by the need to have a uniform and convenient patching frontend to sed(1) operations in the OpenPKG package specifications. SEE ALSO
shtool(1), sed(1). 18-Jul-2008 shtool 2.0.8 SHTOOL-SUBST.TMP(1)
All times are GMT -4. The time now is 06:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy