Sponsored Content
Top Forums Shell Programming and Scripting How to quickly substitute pattern within certain range of a huge file? Post 303017462 by rbatte1 on Wednesday 16th of May 2018 10:11:41 AM
Old 05-16-2018
Another thing to consider is the resources you have, so memory, disk devices and contention by other applications. If you run out of memory you may end up swapping/paging real memory to disk which is time consuming to write and to (later on) read back in.

For the disk, is it local disk or an attached SAN? I fear it might be an NFS or Samba mounted share which will be slow because another server is doing the real IO and shovelling it across the network.

If it is not NFS or Samba, is it local disk or SAN is still a question. Local simple disks (no RAID controller) will require writes to be committed before returning control to the program. You might find a high %SYS time on something like vmstat 3, ignoring the first line which is statistics since boot.

Local disk also may have IO contention for the physical devices.

Local hardware RAID disk or SAN provided disk LUNs (hopefully fibre attached), on the other hand should give better performance because they usually come with a large cache, to IO reads are anticipated and writes and written to disk-cache memory (and committed to real disk later) so the control goes back to the CPU again.


Can you tell us more about the resources you have?




Kind regards,
Robin
This User Gave Thanks to rbatte1 For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

need solution for this quickly. please quickly.

Write a nawk script that will produce the following report: ***FIRST QUARTERLY REPORT*** ***CAMPAIGN 2004 CONTRIBUTIONS*** ------------------------------------------------------------------------- NAME PHONE Jan | ... (5 Replies)
Discussion started by: p.palakj.shah
5 Replies

2. Shell Programming and Scripting

print range between two patterns if it contains a pattern within the range

I want to print between the range two patterns if a particular pattern is present in between the two patterns. I am new to Unix. Any help would be greatly appreciated. e.g. Pattern1 Bombay Calcutta Delhi Pattern2 Pattern1 Patna Madras Gwalior Delhi Pattern2 Pattern1... (2 Replies)
Discussion started by: joyan321
2 Replies

3. Shell Programming and Scripting

sed pattern range

Hi guys, trying to replace a '#' with a ' ' (space) but only between the brackets '(' and ')' N="text1#text2#text3(var1#var2#var3)" N=`echo $N |sed '/(/,/) s/#. //'` echo $N Looking for an output of "text1#text2#text3(var1 var2 var3)" Any ideas? (15 Replies)
Discussion started by: mikepegg
15 Replies

4. Shell Programming and Scripting

How to combine lines within range of pattern

I've a file say having line 1 line 2 (NP line 3 line 4 line 5) line 6 I want to combine lines starting from (NP and ending with ) then it will look like line 1 line 2 (NP line3 line4 line5) line 6 I tried using sed '/(NP/,/)$/ s/\n/ /' but it's not working. Any help please? ... (8 Replies)
Discussion started by: neg
8 Replies

5. Shell Programming and Scripting

Removing tmp file too quickly?

Still trying to get the basics down and I would like a different solution to what I'm currently doing and a better understanding of why it's happening. I've written a simple backup script that tars individual directories and then dumps them to a NFS drive. STDERR is being dumped into a process... (2 Replies)
Discussion started by: mandelbrot333
2 Replies

6. Shell Programming and Scripting

Pattern Matchin Huge File

Hi Experts, I've issue with the huge file. My requirement is I need to search a pattern between the 155-156 position and if its match's to 31 or 36 then need to route that to a new separate files. The main file has around 1459328 line and 2 GB in size. I tired with the below code which take... (9 Replies)
Discussion started by: senthil.ak
9 Replies

7. Shell Programming and Scripting

Print pattern range to a new file

Hi Everyone! I really appreciate all of your help, I'm learning so much, can't wait until I get good enough to start answering questions! I have a problem ... from one large file, I'd like to create multiple new files for each pattern block beginning with /^ISA/ ending with /^IEA/ ... (2 Replies)
Discussion started by: verge
2 Replies

8. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

9. Shell Programming and Scripting

sed Range Pattern and 2 lines before Start Pattern

Hi all, I have been searching all over Google but I am unable to find a solution for a particular result that I am trying to achieve. Consider the following input: 1 2 3 4 5 B4Srt1--Variable-0000 B4Srt2--Variable-1111 Srt 6 7 8 9 10 End (3 Replies)
Discussion started by: y2jacky
3 Replies

10. Shell Programming and Scripting

Extract range from config file matching pattern

I have config file like this: server_name xx opt1 opt2 opt3 suboptions1 #suboptions - disabled suboptions2 pattern suboptions3 server_name yy opt1 opt2 opt3 suboptions1 pattern #suboptions - disabled suboptions2 So basically I want to extract the server... (1 Reply)
Discussion started by: nemesis911
1 Replies
re(7)							 Miscellaneous Information Manual						     re(7)

NAME
re - SWXCR RAID interface SYNOPSIS
2100 Server Model A500MP DEC SWXCR controller xcrn at * vector xcintr device disk renn at xcrn drive nn DESCRIPTION
The re driver is for the SWXCR RAID Array controller. The following rules are used to determine the major and minor numbers that are associated with an re type disk. There are two major num- bers used to represent re disks. The major numbers are 11 for block devices and 44 for character (raw) devices. The minor number is used to represent both the unit number and partition. A disk partition refers to a designated portion of the physical disk. To accomplish this reference, the 20-bit minor number is divided into three parts. The lowest six bits of the minor number specify a disk partition. The partitions use a letter, a through h, for their name. The next three bits of the minor number specify the RE unit number for a unit attached to an SWXCR controller. The final 11 bits specify the controller number. The device special file names associated with re disks are based on conventions that are closely associated with the minor number assigned to the disk. The standard device names begin with re for block special files and rre for character (raw) special files. Following the re is the unit number and then a letter, a through h, that represents the partition. Throughout this reference page, the question mark (?) character represents the unit number in the name of the device special file. For example, re?b could represent re0b, re1b, and so on. The unit number can be calculated if the major and minor numbers of an re disk are provided. For example, suppose you have a device spe- cial file rre6a, with a major number of 44 and a minor number of 384. The partition is represented by the lower six bits of the number 384. These lower six bits of the number 384 are 0, which specifies the a partition. The next three bits of the minor number 384 specify the unit number, which is 6. The next eleven bits specify the controller number, which is zero. Putting these three pieces together reveals that the major/minor number pair 44/384 refers to the a partition of unit 6 attached to controller number 0. A disk can be accessed through either the block special file or the character special file. The block special file accesses the disk using the file system's normal buffering mechanism. Reads and writes to the block special file can specify any size. This capability avoids the need to limit data transfers to the size of physical disk records and to calculate offsets within disk records. The file system can break up large read and write requests into smaller fixed size transfers to the disk. The character special file provides a raw interface that allows for direct transmission between the disk and the user's read or write buf- fer. A single read or write to the raw interface results in exactly one I/O operation. Consequently, raw I/O may be considerably more efficient for large transfers. For systems with RE disks, the first software boot after the system is powered on may take longer than expected. This delay is normal and is caused by the software spinning up the RE disks. Disk Support The RE driver handles all disk drives that can be connected to the SWXCR controller. To determine which drives are supported for specific CPU types and hardware configurations, see the Installation and Configuration Guide for the StorageWorks RAID Array 200 Subsystem Family. SWXCR RAID Controllers are viewed in all cases as RE type disks. There are some notable differences that should be taken into considera- tion when configuring a RAID device: Currently only sector sizes of 512 bytes are supported. Logical Volume sizes are not fixed sizes as compared to other disk devices. The size of the Logical Volume is configurable based on needs. The dynamic nature of Logical Volume sizes is dealt with by defining RAID devices as DYNAMIC. Only partitions a, b, c, and g are defined. If necessary, the disklabel(8) command can be run to change and define partitions for RAID devices. Usually, the re?a partition is used for the root file system and the re?b partition as a paging area. The re?c partition can be used for disk-to-disk copying because it maps the entire disk. The starting location and length (in 512 byte sectors) of the disk partitions of each drive are shown in the following table. Partition sizes can be changed by using the disklabel(8) command. SWXCR (RAID) partitions for systems based on the Alpha AXP architecture disk start length re?a 0 131072 re?b 131072 262144 re?c 0 end of media re?d 0 0 re?e 0 0 re?f 0 0 re?g 393216 end of media re?h 0 0 FILES
/dev/re??? /dev/rre??? /etc/disktab RELATED INFORMATION
disklabel(8), MAKEDEV(8), uerf(8) delim off re(7)
All times are GMT -4. The time now is 12:36 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy