How to quickly substitute pattern within certain range of a huge file?
I have big files (some are >300GB!) that need substitution for some patterns, for example, change Multiple Spaces into Tab. I used this oneliner:
but it seems very slow as the job is still running after 24 hours! In this example, only the first 18 rows need be changed, and the rest is untouched.
Is there any better way to do the job quickly? I'm using GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu) on Linux 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3+deb9u1 (2017-12-23) x86_64 GNU/Linux.
Thanks a lot!
Write a nawk script that will produce the following report:
***FIRST QUARTERLY REPORT***
***CAMPAIGN 2004 CONTRIBUTIONS***
-------------------------------------------------------------------------
NAME PHONE Jan | ... (5 Replies)
I want to print between the range two patterns if a particular pattern is present in between the two patterns. I am new to Unix. Any help would be greatly appreciated.
e.g.
Pattern1
Bombay
Calcutta
Delhi
Pattern2
Pattern1
Patna
Madras
Gwalior
Delhi
Pattern2
Pattern1... (2 Replies)
Hi guys, trying to replace a '#' with a ' ' (space) but only between the brackets '(' and ')'
N="text1#text2#text3(var1#var2#var3)"
N=`echo $N |sed '/(/,/) s/#. //'`
echo $N
Looking for an output of "text1#text2#text3(var1 var2 var3)"
Any ideas? (15 Replies)
I've a file say having
line 1
line 2
(NP
line 3
line 4
line 5)
line 6
I want to combine lines starting from (NP and ending with ) then it will look like
line 1
line 2
(NP line3 line4 line5)
line 6
I tried using sed '/(NP/,/)$/ s/\n/ /' but it's not working. Any help please?
... (8 Replies)
Still trying to get the basics down and I would like a different solution to what I'm currently doing and a better understanding of why it's happening. I've written a simple backup script that tars individual directories and then dumps them to a NFS drive. STDERR is being dumped into a process... (2 Replies)
Hi Experts,
I've issue with the huge file.
My requirement is I need to search a pattern between the 155-156 position and if its match's to 31 or 36 then need to route that to a new separate files.
The main file has around 1459328 line and 2 GB in size. I tired with the below code which take... (9 Replies)
Hi Everyone!
I really appreciate all of your help, I'm learning so much, can't wait until I get good enough to start answering questions!
I have a problem ... from one large file, I'd like to create multiple new files for each pattern block
beginning with /^ISA/
ending with /^IEA/
... (2 Replies)
Hi,
I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each.
Please help me as Split command cannot work here as it might miss tags..
Format of the file is as below
<!--###### ###### START-->... (6 Replies)
Hi all,
I have been searching all over Google but I am unable to find a solution for a particular result that I am trying to achieve.
Consider the following input:
1
2
3
4
5
B4Srt1--Variable-0000
B4Srt2--Variable-1111
Srt
6
7
8
9
10
End (3 Replies)
I have config file like this:
server_name xx opt1 opt2 opt3
suboptions1
#suboptions - disabled
suboptions2 pattern
suboptions3
server_name yy opt1 opt2 opt3
suboptions1 pattern
#suboptions - disabled
suboptions2
So basically I want to extract the server... (1 Reply)
Discussion started by: nemesis911
1 Replies
LEARN ABOUT DEBIAN
shtool-subst
SHTOOL-SUBST.TMP(1) GNU Portable Shell Tool SHTOOL-SUBST.TMP(1)NAME
shtool-subst - GNU shtool sed(1) substitution operations
SYNOPSIS
shtool subst [-v|--verbose] [-t|--trace] [-n|--nop] [-w|--warning] [-q|--quiet] [-s|--stealth] [-i|--interactive] [-b|--backup ext]
[-e|--exec cmd] [-f|--file cmd-file] [file] [file ...]
DESCRIPTION
This command applies one or more sed(1) substitution operations to stdin or any number of files.
OPTIONS
The following command line options are available.
-v, --verbose
Display some processing information.
-t, --trace
Enable the output of the essential shell commands which are executed.
-n, --nop
No operation mode. Actual execution of the essential shell commands which would be executed is suppressed.
-w, --warning
Show warning on substitution operation resulting in no content change on every file. The default is to show a warning on substitution
operations resulted in no content change on all files.
-q, --quiet
Suppress warning on substitution operation resulting in no content change.
-s, --stealth
Stealth operation. Preserve timestamp on file.
-i, --interactive
Enter interactive mode where the user has to approve each operation.
-b, --backup ext
Preserve backup of original file using file name extension ext. Default is to overwrite the original file.
-e, --exec cmd
Specify sed(1) command directly.
-f, --file cmd-file
Read sed(1) command from file.
EXAMPLE
# shell script
shtool subst -i -e 's;(c) ([0-9]*)-2000;(c) 1-2001;' *.[ch]
# RPM spec-file
%install
shtool subst -v -n
-e 's;^(prefix=).*;1 $RPM_BUILD_ROOT%{_prefix};g'
-e 's;^(sysconfdir=).*;1 $RPM_BUILD_ROOT%{_prefix}/etc;g'
`find . -name Makefile -print`
make install
HISTORY
The GNU shtool subst command was originally written by Ralf S. Engelschall <rse@engelschall.com> in 2001 for GNU shtool. It was prompted
by the need to have a uniform and convenient patching frontend to sed(1) operations in the OpenPKG package specifications.
SEE ALSO shtool(1), sed(1).
18-Jul-2008 shtool 2.0.8 SHTOOL-SUBST.TMP(1)