Sponsored Content
Top Forums Shell Programming and Scripting Big data file - sed/grep/awk? Post 302204239 by dlam on Wednesday 11th of June 2008 04:10:31 AM
Old 06-11-2008
Thanks, I'll implement that bit now and see if that makes much difference. The "stuff" part is:

for results in `grep "^\[$STARTHOUR" ANYOLDFILE | awk -F'|' '{print $4}'`
do
if [ $results -gt 9999 ]
then
(( NUMOFSECONDS[10]=NUMOFSECONDS[10]+1 ))
else
typeset -Z4 results
AMOUNTOFSECONDSFINDER=`echo $results| cut -c1`
(( NUMOFSECONDS[AMOUNTOFSECONDSFINDER]=NUMOFSECONDS[AMOUNTOFSECONDSFINDER]+1 ))
fi
done


... so I'm not sure what fat can be trimmed from here. Feel free to tell me if I'm missing anything obvious!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

filter parts of a big file using awk or sed script

I need an assistance in file generation using awk, sed or anything... I have a big file that i need to filter desired parts only. The objective is to select (and print) the report # having the string "apple" on 2 consecutive lines in every report. Please note that the "apple" line has a HEX... (1 Reply)
Discussion started by: apalex
1 Replies

2. Shell Programming and Scripting

Should I use sed/ grep/awk for wrap file?

Hi, This is my first time post a new thread. I have been trying to work on this for the past 2 days and could not find any good solution. I have 1 long long line ( EDI wrapped file) like below: NEW*SR*04411763447*279*278*Q~*ZR*AAV*SR*04511763460*SQ*21B37F04~HL*305*304*Q~K~SN1*1*1*SR*05511763461*... (6 Replies)
Discussion started by: vanda_25
6 Replies

3. Shell Programming and Scripting

sed or awk to extract data from Xml file

Hi, I want to get data from Xml file by using sed or awk command. I want to get the following result : mon titre 1;Createur1;Dossier1 mon titre 1;Createur1;Dossier1 and save it in cvs file (fichier.cvs). FROM this Xml file (test.xml): <playlist version="1"> <trackList> <track>... (1 Reply)
Discussion started by: yeclota
1 Replies

4. Shell Programming and Scripting

How to cut some data from big file

How to cut data from big file my file around 30 gb I tried "head -50022172 filename > newfile.txt ,and tail -5454283 newfile.txt. It's slowy. afer that I tried sed -n '46467831,50022172p' filename > newfile.txt ,also slow Please recommend me , faster command to cut some data from... (4 Replies)
Discussion started by: almanto
4 Replies

5. UNIX for Dummies Questions & Answers

awk and grep to search a data file

Hi everyone, I cannot figure out how I can do a search in a file that has Names, Surnames, Addresses and telephone number of a number of people. Here is an example of the data file Daisy:Hunter:490 London Road:07313196347 Richard:Murphy:983 Main Road:07002625997 Isobel:Magnusson:133 London... (1 Reply)
Discussion started by: philipisaia
1 Replies

6. Shell Programming and Scripting

formatting data file with awk or sed

Hi, I have a (quite large) data file which looks like: _____________ header part.. more header part.. x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 ... ... x59 x60 y1 y2 y3 y4... ... y100 ______________ where x1, x2,...,x60 and y1, y2,...y100 are numbers of 10 digits (so each line... (5 Replies)
Discussion started by: lego
5 Replies

7. Shell Programming and Scripting

Sort a big data file

Hello, I have a big data file (160 MB) full of records with pipe(|) delimited those fields. I`m sorting the file on the first field. I'm trying to sort with "sort" command and it brings me 6 minutes. I have tried with some transformation methods in perl but it results "Out of memory". I was... (2 Replies)
Discussion started by: rubber08
2 Replies

8. Shell Programming and Scripting

Router ping log extract data from it Awk/Sed/grep

Hi, I am new to this world.. Using expect i loging to router and checking ping response to my links. I need to genarate report using this output and that report contains only three file link name, packet loss, latency. my output of script is like below: -bash-3.00$ monmw/mwbkp... (2 Replies)
Discussion started by: jkmistry
2 Replies

9. Shell Programming and Scripting

awk - sed / reading from a data file and doing algebraic operations

Hi everyone, I am trying to write a bash script which reads a data file and does some algebraic operations. here is the structure of data.xml file that I have; 1 <data> 2 . 3 . 4 . 5 </data> 6 <data> 7 . 8 . 9 . 10</data> etc. Each data block contains same number of lines (say... (4 Replies)
Discussion started by: hayreter
4 Replies

10. Shell Programming and Scripting

How to grep/sed selected data from a command or file?

Below is the output of a DB2 command. Now I have 2 requirements... Database Partition 0 -- Database TESTDB1 -- Active Standby -- Up 213 days 02:33:07 -- Date 02/22/2016 17:04:50 HADR Information: Role State SyncMode HeartBeatsMissed LogGapRunAvg (bytes) Standby ... (2 Replies)
Discussion started by: rlokesh27
2 Replies
integer(3perl)						 Perl Programmers Reference Guide					    integer(3perl)

NAME
integer - Perl pragma to use integer arithmetic instead of floating point SYNOPSIS
use integer; $x = 10/3; # $x is now 3, not 3.33333333333333333 DESCRIPTION
This tells the compiler to use integer operations from here to the end of the enclosing BLOCK. On many machines, this doesn't matter a great deal for most computations, but on those without floating point hardware, it can make a big difference in performance. Note that this only affects how most of the arithmetic and relational operators handle their operands and results, and not how all numbers everywhere are treated. Specifically, "use integer;" has the effect that before computing the results of the arithmetic operators (+, -, *, /, %, +=, -=, *=, /=, %=, and unary minus), the comparison operators (<, <=, >, >=, ==, !=, <=>), and the bitwise operators (|, &, ^, <<, >>, |=, &=, ^=, <<=, >>=), the operands have their fractional portions truncated (or floored), and the result will have its fractional portion truncated as well. In addition, the range of operands and results is restricted to that of familiar two's complement integers, i.e., -(2**31) .. (2**31-1) on 32-bit architectures, and -(2**63) .. (2**63-1) on 64-bit architectures. For example, this code use integer; $x = 5.8; $y = 2.5; $z = 2.7; $a = 2**31 - 1; # Largest positive integer on 32-bit machines $, = ", "; print $x, -$x, $x + $y, $x - $y, $x / $y, $x * $y, $y == $z, $a, $a + 1; will print: 5.8, -5, 7, 3, 2, 10, 1, 2147483647, -2147483648 Note that $x is still printed as having its true non-integer value of 5.8 since it wasn't operated on. And note too the wrap-around from the largest positive integer to the largest negative one. Also, arguments passed to functions and the values returned by them are not affected by "use integer;". E.g., srand(1.5); $, = ", "; print sin(.5), cos(.5), atan2(1,2), sqrt(2), rand(10); will give the same result with or without "use integer;" The power operator "**" is also not affected, so that 2 ** .5 is always the square root of 2. Now, it so happens that the pre- and post- increment and decrement operators, ++ and --, are not affected by "use integer;" either. Some may rightly consider this to be a bug -- but at least it's a long-standing one. Finally, "use integer;" also has an additional affect on the bitwise operators. Normally, the operands and results are treated as unsigned integers, but with "use integer;" the operands and results are signed. This means, among other things, that ~0 is -1, and -2 & -5 is -6. Internally, native integer arithmetic (as provided by your C compiler) is used. This means that Perl's own semantics for arithmetic operations may not be preserved. One common source of trouble is the modulus of negative numbers, which Perl does one way, but your hardware may do another. % perl -le 'print (4 % -3)' -2 % perl -Minteger -le 'print (4 % -3)' 1 See "Pragmatic Modules" in perlmodlib, "Integer Arithmetic" in perlop perl v5.14.2 2010-12-30 integer(3perl)
All times are GMT -4. The time now is 08:12 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy