Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Script to break up file (write new files) in bash Post 302898677 by torchij on Wednesday 23rd of April 2014 09:39:38 AM
Old 04-23-2014
My apologies, here are some numbers. The file I'm using has 252 "pieces" and 49,134 lines total.
I like the csplit option, its clean and gives options:
Code:
time csplit -f test_ -n3 -s segments.test.txt "/^$/" {*}
real    0m2.112s
user    0m0.106s
sys     0m0.033s

awk command 1 (had to remove the quotes around the redirect ">")
Code:
time awk 'BEGIN{FN=1} NF==0 {FN++;next} {print $0 > "file_new_TEST"FN}' segments.test.txt
real    0m0.362s
user    0m0.054s
sys     0m0.034s

awk command 2:
Code:
time awk -F "\t" '{print $0 > "file_new" NR ".txt"}' RS= segments.test.txt
real    0m0.322s
user    0m0.012s
sys     0m0.035s

Awk seems to perform better...I'm quite the computing newb, it would be interesting to understand why this might be?

Thanks for all the help.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Break a file into separate files

Hello I am facing a scenario where I have a file with XML content and I am running shell script over it. But the problem is the XML is getting updated with new services. In the below scenario, my script takes values from the xml file from one service name say ABCD. Since there are multiple, it is... (8 Replies)
Discussion started by: chiru_h
8 Replies

2. Shell Programming and Scripting

shell script to remove old files and write to a log file

Hi, I have a script that works on a unix box but am trying to get it working on a linux box that uses shell. I am not a programmer so this is proving harder than I imagined. I made some changes and ended up with the script below but when I run it I get the following messages. Any help would be... (4 Replies)
Discussion started by: yabai
4 Replies

3. Shell Programming and Scripting

Can anybody write this bash script ?

hi, first congratulations on the nice forum! Can anybody write script, which can make copy of some or all files of the current directory in new directory (called "backups", which must be made in the current directory, if it's not already exist). And bring out a massage (report) with the count... (7 Replies)
Discussion started by: Cecko
7 Replies

4. Shell Programming and Scripting

How to write bash script to explode multiple zip files

I have a directory full of zip files. How would I write a bash script to enumerate all the zip files, remove the ".zip" from the file name, create a directory by that name and unzip each zip file into its corresponding directory? Thanks! Siegfried (3 Replies)
Discussion started by: siegfried
3 Replies

5. Shell Programming and Scripting

Write a new file from 2 files as input to the script

Hi- I am hoping someone can give me some pointers to get me started. I have a file which contains some dn's .e.g file 1 cn=bob,cn=user,dc=com cn=kev,cn=user,dc=com cn=john,cn=user,dc=com I have a second file e.g. file.template which looks something like :- dn: <dn> objectclass:... (5 Replies)
Discussion started by: sniper57
5 Replies

6. Homework & Coursework Questions

How to write script in bash.

I am very new to Linux/Unix. Kindly assist the following: I wish to write a bash shell script called how_many_to_go that calculates and prints the number of days, hours, minutes and/or seconds until the end of the current month (based on the output of the date command). Do ... (2 Replies)
Discussion started by: alobi
2 Replies

7. Shell Programming and Scripting

how to write bash script that will automatically extract zip file

i'm trying to write a bash script that that will automatically extract zip files after the download. i writed this script #!/bin/bash wget -c https://github.com/RonGokhle/kernel-downloader/zipball/master CURRENDIR=/home/kernel-downloader cd $CURRENDIR rm $CURRENDIR/zipfiles 2>/dev/null ... (2 Replies)
Discussion started by: ron gokhle
2 Replies

8. UNIX for Dummies Questions & Answers

Write pid and command name to a txt file while executing a bash script

Hi All, Just have a requirement, I am executing a bash shell script, my requirement is to catch the pid and job name to a txt file in the same directory, is there anyway to do it? please help me out. Regards Rahul ---------- Post updated at 08:42 AM ---------- Previous update was at... (2 Replies)
Discussion started by: rahulkalra9
2 Replies

9. Shell Programming and Scripting

Break output file into three files

Help! :) I am getting an output file that looks similar to below. EMAIL_ADDR ----------------------------------------------------------------------------------- user@gmail.com DATABASENAME ----------------------------------------------------------------------------------- db1 db2 db3... (6 Replies)
Discussion started by: cpolikowsky
6 Replies

10. UNIX for Beginners Questions & Answers

How To Write my Bash Script To Automate it?

Hello guys, I need some help. I am new in bash and I don't know how to automate the following script. head -2 out1 > 1.fasta sed ‘1,2 d' out1 > out2 rm out1 head -2 out2 > 2.fasta sed ‘1,2 d' out2 > out1 rm out2 head -2 out2 > 3.fasta sed '1,2 d' out2 > out1 rm out2 .......... (3 Replies)
Discussion started by: dellia222
3 Replies
SG_RBUF(8)							     SG3_UTILS								SG_RBUF(8)

NAME
sg_rbuf - reads data using SCSI READ BUFFER command SYNOPSIS
sg_rbuf [--buffer=EACH] [--dio] [--help] [--mmap] [--quick] [--size=OVERALL] [--test] [--verbose] [--version] DEVICE sg_rbuf [-b=EACH_KIB] [-d] [-m] [-q] [-s=OVERALL_MIB] [-t] [-v] [-V] DEVICE DESCRIPTION
This command reads data with the SCSI READ BUFFER command and then discards it. Typically the data being read is from a disk's memory cache. It is assumed that the data is sourced quickly (although this is not guaranteed by the SCSI standards) so that it is faster than reading data from the media. This command is designed for timing transfer speeds across a SCSI transport. To fetch the data with a SCSI READ BUFFER command and optionally decode it see the sg_read_buffer utility. There is also a sg_write_buffer utility useful for downloading firmware amongst other things. This utility supports two command line syntaxes, the preferred one is shown first in the synopsis and explained in this section. A later section on the old command line syntax outlines the second group of options. OPTIONS
Arguments to long options are mandatory for short options as well. -b, --buffer=EACH where EACH is the number of bytes to be transferred by each READ BUFFER command. The default is the actual available buffer size returned by the READ BUFFER (descriptor) command. The maximum is the same as the default, hence this argument can only be used to reduce the size of each transfer to less than the device's actual available buffer size. -d, --dio use direct IO if available. This option is only available if the DEVICE is a sg driver device node (e.g. /dev/sg1). In this case the sg driver will attempt to configure the DMA from the SCSI adapter to transfer directly into user memory. This will eliminate the copy via kernel buffers. If not available then this will be reported and indirect IO will be done instead. -h, --help print usage message then exit. -m, --mmap use memory mapped IO if available. This option is only available if the DEVICE is a sg driver device node (e.g. /dev/sg1). In this case the sg driver will attempt to configure the DMA from the SCSI adapter to transfer directly into user memory. This will elimi- nate the copy via kernel buffers. -O, --old switch to older style options. -q, --quick only transfer the data into kernel buffers (typically by DMA from the SCSI adapter card) and do not move it into the user space. This option is only available if the DEVICE is a sg driver device node (e.g. /dev/sg1). -s, --size=OVERALL where OVERALL is the size of total transfer in bytes. The default is 200 MiB (200*1024*1024 bytes). The actual number of bytes transferred may be slightly less than requested since all transfers are the same size (and an integer division is involved rounding towards zero). -t, --time times the bulk data transfer component of this command. The elapsed time is printed out plus a MB/sec calculation. In this case "MB" is 1,000,000 bytes. The gettimeofday() system call is used internally for the time calculation. -v, --verbose increase level of verbosity. Can be used multiple times. -V, --version print out version string then exit. NOTES
This command is typically used on modern SCSI disks which have a RAM cache in their drive electronics. If no IO to the magnetic media, or slower devices like flash RAM, is involved then the disk may be able to source data fast enough to saturate the bandwidth of the SCSI transport. The bottleneck may then be the DMA element in the HBA, the Linux drivers or the host machine's hardware (e.g. speed of RAM). Various numeric arguments (e.g. OVERALL) may include multiplicative suffixes or be given in hexadecimal. See the "NUMERIC ARGUMENTS" sec- tion in the sg3_utils(8) man page. EXAMPLES
On the test system /dev/sg0 corresponds to a fast disk on a U2W SCSI bus (max 80 MB/sec). The disk specifications state that its cache is 4 MB. $ time ./sg_rbuf /dev/sg0 READ BUFFER reports: buffer capacity=3434944, offset boundary=6 Read 200 MiB (actual 199 MiB, 209531584 bytes), buffer size=3354 KiB real 0m5.072s, user 0m0.000s, sys 0m2.280s So that is approximately 40 MB/sec at 40 % utilization. Now with the addition of the "-q" option this throughput improves and the utiliza- tion drops to 0%. $ time ./sg_rbuf -q /dev/sg0 READ BUFFER reports: buffer capacity=3434944, offset boundary=6 Read 200 MiB (actual 199 MiB, 209531584 bytes), buffer size=3354 KiB real 0m2.784s, user 0m0.000s, sys 0m0.000s EXIT STATUS
The exit status of sg_rbuf is 0 when it is successful. Otherwise see the sg3_utils(8) man page. OLDER COMMAND LINE OPTIONS
The options in this section were the only ones available prior to sg3_utils version 1.23 . In sg3_utils version 1.23 and later these older options can be selected by either setting the SG3_UTILS_OLD_OPTS environment variable or using '--old' (or '-O) as the first option. -b=EACH_KIB where EACH_KIB is the number of Kilobytes (i.e. 1024 byte units) to be transferred by each READ BUFFER command. Similar to the --buffer=EACH option in the main description but the units are different. -d use direct IO if available. Equivalent to the --dio option in the main description. -m use memory mapped IO if available. Equivalent to the --mmap option in the main description. -N switch to the newer style options. -q only transfer the data into kernel buffers (typically by DMA from the SCSI adapter card) and do not move it into the user space. Equivalent to the --quick option in the main description. -s=OVERALL_MIB where OVERALL_MIB is the size of total transfer in Megabytes (1048576 bytes). Similar to the --size=OVERALL option in the main description but the units are different. -t times the bulk data transfer component of this command. Equivalent to the --time option in the main description. -v increase level of verbosity. Can be used multiple times. -V print out version string then exit. AUTHOR
Written by Douglas Gilbert REPORTING BUGS
Report bugs to <dgilbert at interlog dot com>. COPYRIGHT
Copyright (C) 2000-2007 Douglas Gilbert This software is distributed under the GPL version 2. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PUR- POSE. SEE ALSO
sg_read_buffer, sg_write_buffer, sg_test_rwbuf(all in sg3_utils) sg3_utils-1.23 January 2007 SG_RBUF(8)
All times are GMT -4. The time now is 12:12 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy