Sponsored Content
Top Forums Shell Programming and Scripting Big data file - sed/grep/awk? Post 302204272 by era on Wednesday 11th of June 2008 06:38:23 AM
Old 06-11-2008
@fpmurphy: I don't think it's growing, it's just looping over the fourth field. As far as I can tell, the root cause would seem to be that the shell's arrays are not scaling nicely.

Most of what you're doing can be accomplished in awk directly just as well. typeset -Z4 appears to be a kshism to pad a number with leading zeros to the specified width, correct?

I don't know if this captures all the nuances of your script, but perhaps it can be refined to do what you need.

Code:
awk -F '|' '"/^\[$STARTHOUR/"'{
    if ($4 > 9999) $4=10000; ++m[int($4/1000)]}
  END { for (i=0; i<=10; ++i) printf ("NUMOFSECONDS[%i]=%04i\n", i, m[i]) }' ANYOLDFILE


Last edited by era; 06-11-2008 at 07:41 AM.. Reason: Use int() to truncate division
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

filter parts of a big file using awk or sed script

I need an assistance in file generation using awk, sed or anything... I have a big file that i need to filter desired parts only. The objective is to select (and print) the report # having the string "apple" on 2 consecutive lines in every report. Please note that the "apple" line has a HEX... (1 Reply)
Discussion started by: apalex
1 Replies

2. Shell Programming and Scripting

Should I use sed/ grep/awk for wrap file?

Hi, This is my first time post a new thread. I have been trying to work on this for the past 2 days and could not find any good solution. I have 1 long long line ( EDI wrapped file) like below: NEW*SR*04411763447*279*278*Q~*ZR*AAV*SR*04511763460*SQ*21B37F04~HL*305*304*Q~K~SN1*1*1*SR*05511763461*... (6 Replies)
Discussion started by: vanda_25
6 Replies

3. Shell Programming and Scripting

sed or awk to extract data from Xml file

Hi, I want to get data from Xml file by using sed or awk command. I want to get the following result : mon titre 1;Createur1;Dossier1 mon titre 1;Createur1;Dossier1 and save it in cvs file (fichier.cvs). FROM this Xml file (test.xml): <playlist version="1"> <trackList> <track>... (1 Reply)
Discussion started by: yeclota
1 Replies

4. Shell Programming and Scripting

How to cut some data from big file

How to cut data from big file my file around 30 gb I tried "head -50022172 filename > newfile.txt ,and tail -5454283 newfile.txt. It's slowy. afer that I tried sed -n '46467831,50022172p' filename > newfile.txt ,also slow Please recommend me , faster command to cut some data from... (4 Replies)
Discussion started by: almanto
4 Replies

5. UNIX for Dummies Questions & Answers

awk and grep to search a data file

Hi everyone, I cannot figure out how I can do a search in a file that has Names, Surnames, Addresses and telephone number of a number of people. Here is an example of the data file Daisy:Hunter:490 London Road:07313196347 Richard:Murphy:983 Main Road:07002625997 Isobel:Magnusson:133 London... (1 Reply)
Discussion started by: philipisaia
1 Replies

6. Shell Programming and Scripting

formatting data file with awk or sed

Hi, I have a (quite large) data file which looks like: _____________ header part.. more header part.. x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 ... ... x59 x60 y1 y2 y3 y4... ... y100 ______________ where x1, x2,...,x60 and y1, y2,...y100 are numbers of 10 digits (so each line... (5 Replies)
Discussion started by: lego
5 Replies

7. Shell Programming and Scripting

Sort a big data file

Hello, I have a big data file (160 MB) full of records with pipe(|) delimited those fields. I`m sorting the file on the first field. I'm trying to sort with "sort" command and it brings me 6 minutes. I have tried with some transformation methods in perl but it results "Out of memory". I was... (2 Replies)
Discussion started by: rubber08
2 Replies

8. Shell Programming and Scripting

Router ping log extract data from it Awk/Sed/grep

Hi, I am new to this world.. Using expect i loging to router and checking ping response to my links. I need to genarate report using this output and that report contains only three file link name, packet loss, latency. my output of script is like below: -bash-3.00$ monmw/mwbkp... (2 Replies)
Discussion started by: jkmistry
2 Replies

9. Shell Programming and Scripting

awk - sed / reading from a data file and doing algebraic operations

Hi everyone, I am trying to write a bash script which reads a data file and does some algebraic operations. here is the structure of data.xml file that I have; 1 <data> 2 . 3 . 4 . 5 </data> 6 <data> 7 . 8 . 9 . 10</data> etc. Each data block contains same number of lines (say... (4 Replies)
Discussion started by: hayreter
4 Replies

10. Shell Programming and Scripting

How to grep/sed selected data from a command or file?

Below is the output of a DB2 command. Now I have 2 requirements... Database Partition 0 -- Database TESTDB1 -- Active Standby -- Up 213 days 02:33:07 -- Date 02/22/2016 17:04:50 HADR Information: Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes) Standby ... (2 Replies)
Discussion started by: rlokesh27
2 Replies
typeset(1)                                                         User Commands                                                        typeset(1)

NAME
typeset, whence - shell built-in functions to set/get attributes and values for shell variables and functions SYNOPSIS
typeset [ +- HLRZfilrtux [n]] [ name [ = value]]... whence [-pv] name... DESCRIPTION
typeset sets attributes and values for shell variables and functions. When typeset is invoked inside a function, a new instance of the variables name is created. The variables value and type are restored when the function completes. The following list of attributes may be specified: -H This flag provides UNIX to host-name file mapping on non-UNIX machines. -L Left justify and remove leading blanks from value. If n is non-zero it defines the width of the field; otherwise, it is determined by the width of the value of first assignment. When the variable is assigned to, it is filled on the right with blanks or trun- cated, if necessary, to fit into the field. Leading zeros are removed if the -Z flag is also set. The -R flag is turned off. -R Right justify and fill with leading blanks. If n is non-zero it defines the width of the field, otherwise it is determined by the width of the value of first assignment. The field is left filled with blanks or truncated from the end if the variable is reas- signed. The -L flag is turned off. -Z Right justify and fill with leading zeros if the first non-blank character is a digit and the -L flag has not been set. If n is non-zero it defines the width of the field; otherwise, it is determined by the width of the value of first assignment. -f The names refer to function names rather than variable names. No assignments can be made and the only other valid flags are -t, -u and -x. The flag -t turns on execution tracing for this function. The flag -u causes this function to be marked undefined. The FPATH variable will be searched to find the function definition when the function is referenced. The flag -x allows the func- tion definition to remain in effect across shell procedures invoked by name. -i Parameter is an integer. This makes arithmetic faster. If n is non-zero it defines the output arithmetic base; otherwise, the first assignment determines the output base. -l All upper-case characters are converted to lower-case. The upper-case flag, -u is turned off. -r The given names are marked readonly and these names cannot be changed by subsequent assignment. -t Tags the variables. Tags are user definable and have no special meaning to the shell. -u All lower-case characters are converted to upper-case characters. The lower-case flag, -l is turned off. -x The given names are marked for automatic export to the environment of subsequently-executed commands. The -i attribute can not be specified along with -R, -L, -Z, or -f. Using + rather than - causes these flags to be turned off. If no name arguments are given but flags are specified, a list of names (and optionally the values) of the variables which have these flags set is printed. (Using + rather than - keeps the values from being printed.) If no names and flags are given, the names and attributes of all variables are printed. For each name, whence indicates how it would be interpreted if used as a command name. The -v flag produces a more verbose report. The -p flag does a path search for name even if name is an alias, a function, or a reserved word. On this man page, ksh(1) commands that are preceded by one or two * (asterisks) are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a vari- able assignment. This means that tilde substitution is performed after the = sign and word splitting and file name generation are not performed. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ SEE ALSO
ksh(1), set(1), sh(1), attributes(5) SunOS 5.10 1 Feb 1995 typeset(1)
All times are GMT -4. The time now is 11:35 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy