Sponsored Content
Full Discussion: Which is faster AWK or CUT
Top Forums Shell Programming and Scripting Which is faster AWK or CUT Post 302227564 by redoubtable on Thursday 21st of August 2008 12:55:26 PM
Old 08-21-2008
Code:
Tsunami speed # du -h testfile 
54M     testfile
Tsunami speed # time awk -F":" '{print $1}' testfile >awk

real    0m5.687s
user    0m5.311s
sys     0m0.330s
Tsunami speed # time cut -d":" -f1 testfile >cut

real    0m0.730s
user    0m0.542s
sys     0m0.160s
Tsunami speed #

testfile has various repetitions of "AAA:BBB\n".
Like other posters said, if you can use cut for you problem you should choose it instead of awk, but there are situations where cut just isn't enough.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[grep awk cut] > awk

Hi, I'm very new to scripting. grep $s $filename | awk '{print $2}' | cut -c 1-8 How can I optimize this using a single awk? I tried: awk '/$s/ {print $2}' $filename | cut -c 1-8 However didn't work, I think the awk is not recognizing $s and the verbal is something else. (6 Replies)
Discussion started by: firdousamir
6 Replies

2. UNIX for Dummies Questions & Answers

Help please awk or cut

Hi I'm new to unix programming so struggling with something thats probably simple to many of you I have data files of the format : ID, date, value1, value2, blank on each line either value1 or value2 will be zero. I need my output file to contain ID, date, non-zero value The input... (3 Replies)
Discussion started by: thewench
3 Replies

3. Shell Programming and Scripting

awk help to make my work faster

hii everyone , i have a file in which i have line numbers.. file name is file1.txt aa bb cc "12" qw xx yy zz "23" we bb qw we "123249" jh here 12,23,123249. is the line number now according to this line numbers we have to print lines from other file named... (11 Replies)
Discussion started by: kumar_amit
11 Replies

4. Shell Programming and Scripting

**HELP** need to split this line faster than cut-command

Hi, A datafile containing lines such as below needs to be split: 500000000000932491683600000000000000000000000000016800000GS0000000000932491683600*HOME I need to get the 2-5, 11-20, and 35-40 characters and I can do it via cut command. cut -c 2-5 file > temp1.txt cut -c 11-20 file >... (9 Replies)
Discussion started by: daytripper1021
9 Replies

5. Shell Programming and Scripting

Multi thread awk command for faster performance

Hi, I have a script below for extracting xml from a file. for i in *.txt do echo $i awk '/<.*/ , /.*<\/.*>/' "$i" | tr -d '\n' echo -ne '\n' done . I read about using multi threading to speed up the script. I do not know much about it but read it on this forum. Is it a... (21 Replies)
Discussion started by: chetan.c
21 Replies

6. Shell Programming and Scripting

Faster way to use this awk command

awk "/May 23, 2012 /,0" /var/tmp/datafile the above command pulls out information in the datafile. the information it pulls is from the date specified to the end of the file. now, how can i make this faster if the datafile is huge? even if it wasn't huge, i feel there's a better/faster way to... (8 Replies)
Discussion started by: SkySmart
8 Replies

7. Shell Programming and Scripting

Making a faster alternative to a slow awk command

Hi, I have a large number of input files with two columns of numbers. For example: 83 1453 99 3255 99 8482 99 7372 83 175 I only wish to retain lines where the numbers fullfil two requirements. E.g: =83 1000<=<=2000 To do this I use the following... (10 Replies)
Discussion started by: s052866
10 Replies

8. Shell Programming and Scripting

awk changes to make it faster

I have script like below, who is picking number from one file and and searching in another file, and printing output. Bu is is very slow to be run on huge file.can we modify it with awk #! /bin/ksh while read line1 do echo "$line1" a=`echo $line1` if then echo "$num" cat file1|nawk... (6 Replies)
Discussion started by: mirwasim
6 Replies

9. Shell Programming and Scripting

How to make awk command faster?

I have the below command which is referring a large file and it is taking 3 hours to run. Can something be done to make this command faster. awk -F ',' '{OFS=","}{ if ($13 == "9999") print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12 }' ${NLAP_TEMP}/hist1.out|sort -T ${NLAP_TEMP} |uniq>... (13 Replies)
Discussion started by: Peu Mukherjee
13 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
SG_RBUF(8)							     SG3_UTILS								SG_RBUF(8)

NAME
sg_rbuf - reads data using SCSI READ BUFFER command SYNOPSIS
sg_rbuf [--buffer=EACH] [--dio] [--help] [--mmap] [--quick] [--size=OVERALL] [--test] [--verbose] [--version] DEVICE sg_rbuf [-b=EACH_KIB] [-d] [-m] [-q] [-s=OVERALL_MIB] [-t] [-v] [-V] DEVICE DESCRIPTION
This command reads data with the SCSI READ BUFFER command and then discards it. Typically the data being read is from a disk's memory cache. It is assumed that the data is sourced quickly (although this is not guaranteed by the SCSI standards) so that it is faster than reading data from the media. This command is designed for timing transfer speeds across a SCSI transport. To fetch the data with a SCSI READ BUFFER command and optionally decode it see the sg_read_buffer utility. There is also a sg_write_buffer utility useful for downloading firmware amongst other things. This utility supports two command line syntaxes, the preferred one is shown first in the synopsis and explained in this section. A later section on the old command line syntax outlines the second group of options. OPTIONS
Arguments to long options are mandatory for short options as well. -b, --buffer=EACH where EACH is the number of bytes to be transferred by each READ BUFFER command. The default is the actual available buffer size returned by the READ BUFFER (descriptor) command. The maximum is the same as the default, hence this argument can only be used to reduce the size of each transfer to less than the device's actual available buffer size. -d, --dio use direct IO if available. This option is only available if the DEVICE is a sg driver device node (e.g. /dev/sg1). In this case the sg driver will attempt to configure the DMA from the SCSI adapter to transfer directly into user memory. This will eliminate the copy via kernel buffers. If not available then this will be reported and indirect IO will be done instead. -h, --help print usage message then exit. -m, --mmap use memory mapped IO if available. This option is only available if the DEVICE is a sg driver device node (e.g. /dev/sg1). In this case the sg driver will attempt to configure the DMA from the SCSI adapter to transfer directly into user memory. This will elimi- nate the copy via kernel buffers. -O, --old switch to older style options. -q, --quick only transfer the data into kernel buffers (typically by DMA from the SCSI adapter card) and do not move it into the user space. This option is only available if the DEVICE is a sg driver device node (e.g. /dev/sg1). -s, --size=OVERALL where OVERALL is the size of total transfer in bytes. The default is 200 MiB (200*1024*1024 bytes). The actual number of bytes transferred may be slightly less than requested since all transfers are the same size (and an integer division is involved rounding towards zero). -t, --time times the bulk data transfer component of this command. The elapsed time is printed out plus a MB/sec calculation. In this case "MB" is 1,000,000 bytes. The gettimeofday() system call is used internally for the time calculation. -v, --verbose increase level of verbosity. Can be used multiple times. -V, --version print out version string then exit. NOTES
This command is typically used on modern SCSI disks which have a RAM cache in their drive electronics. If no IO to the magnetic media, or slower devices like flash RAM, is involved then the disk may be able to source data fast enough to saturate the bandwidth of the SCSI transport. The bottleneck may then be the DMA element in the HBA, the Linux drivers or the host machine's hardware (e.g. speed of RAM). Various numeric arguments (e.g. OVERALL) may include multiplicative suffixes or be given in hexadecimal. See the "NUMERIC ARGUMENTS" sec- tion in the sg3_utils(8) man page. EXAMPLES
On the test system /dev/sg0 corresponds to a fast disk on a U2W SCSI bus (max 80 MB/sec). The disk specifications state that its cache is 4 MB. $ time ./sg_rbuf /dev/sg0 READ BUFFER reports: buffer capacity=3434944, offset boundary=6 Read 200 MiB (actual 199 MiB, 209531584 bytes), buffer size=3354 KiB real 0m5.072s, user 0m0.000s, sys 0m2.280s So that is approximately 40 MB/sec at 40 % utilization. Now with the addition of the "-q" option this throughput improves and the utiliza- tion drops to 0%. $ time ./sg_rbuf -q /dev/sg0 READ BUFFER reports: buffer capacity=3434944, offset boundary=6 Read 200 MiB (actual 199 MiB, 209531584 bytes), buffer size=3354 KiB real 0m2.784s, user 0m0.000s, sys 0m0.000s EXIT STATUS
The exit status of sg_rbuf is 0 when it is successful. Otherwise see the sg3_utils(8) man page. OLDER COMMAND LINE OPTIONS
The options in this section were the only ones available prior to sg3_utils version 1.23 . In sg3_utils version 1.23 and later these older options can be selected by either setting the SG3_UTILS_OLD_OPTS environment variable or using '--old' (or '-O) as the first option. -b=EACH_KIB where EACH_KIB is the number of Kilobytes (i.e. 1024 byte units) to be transferred by each READ BUFFER command. Similar to the --buffer=EACH option in the main description but the units are different. -d use direct IO if available. Equivalent to the --dio option in the main description. -m use memory mapped IO if available. Equivalent to the --mmap option in the main description. -N switch to the newer style options. -q only transfer the data into kernel buffers (typically by DMA from the SCSI adapter card) and do not move it into the user space. Equivalent to the --quick option in the main description. -s=OVERALL_MIB where OVERALL_MIB is the size of total transfer in Megabytes (1048576 bytes). Similar to the --size=OVERALL option in the main description but the units are different. -t times the bulk data transfer component of this command. Equivalent to the --time option in the main description. -v increase level of verbosity. Can be used multiple times. -V print out version string then exit. AUTHOR
Written by Douglas Gilbert REPORTING BUGS
Report bugs to <dgilbert at interlog dot com>. COPYRIGHT
Copyright (C) 2000-2007 Douglas Gilbert This software is distributed under the GPL version 2. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PUR- POSE. SEE ALSO
sg_read_buffer, sg_write_buffer, sg_test_rwbuf(all in sg3_utils) sg3_utils-1.23 January 2007 SG_RBUF(8)
All times are GMT -4. The time now is 09:27 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy