Visit Our UNIX and Linux User Community


calculation on a large file


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers calculation on a large file
# 1  
Old 08-18-2009
calculation on a large file

Trying to do some simple maths on a large file. Excel works fine but I have 1 million entries:

If the difference between a number in column 2 and the one above it is more than 100 the insert a new line

8 4001 4100
8 4101 4200
8 4201 4300
8 15901 16000
8 15910 16100

For column 2
if 4101-4001 is > than 100 then insert a new line
else 'move onto next number'
end if
exit loop

The result for the example given should be:
8 4001 4100
8 4101 4200
8 4201 4300

8 15901 16000
8 15910 16100

Thanks for any help

L
# 2  
Old 08-18-2009
Code:
$ cat file
8 4001 4100
8 4101 4200
8 4201 4300
8 15901 16000
8 15910 16100
8 16910 16100

Code:
awk '$2-s > 100 && s {print RS $0; s=$2; next} s=$2' file


Output:

Code:
8 4001 4100
8 4101 4200
8 4201 4300

8 15901 16000
8 15910 16100

8 16910 16100

 

Previous Thread | Next Thread
Test Your Knowledge in Computers #219
Difficulty: Easy
When a data packet comes in to a port on a router, the router reads the network address information in the packet footer to determine the ultimate destination.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with awk percentage calculation from a file

i have a file say test with the below mentioned details Folder Name Total space Space used /test/test1 500.1GB 112.0 GB /test/test2 3.2 TB 5TB /test/test3 3TB 100GB i need to calculate percentage of each row based on total space and space used and copy... (9 Replies)
Discussion started by: venkitesh
9 Replies

2. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

3. Shell Programming and Scripting

How to get full width of screen, when using large calculation with bc ?

Experts, When doing large calculation the digits in the screen are limiting to 68 digit and then with a \ ( backspace) next line comes. example: ubuntu# echo "123456789 ^ 50 " | bc 37648602365872212683379005814670372328125515868188009630652959693316\... (4 Replies)
Discussion started by: rveri
4 Replies

4. Shell Programming and Scripting

How do I include the file being compared into calculation?

nawk -F, 'NR==FNR{file=FILENAME;a++;next} a{if(FILENAME~file)next;b++;} END{ for(i in a){if(a && !b){print "NEW: "i}} for(i in b){if(b)print i"\t\t"b}}' OFS=, 123.csv *.csv I need to include 123.csv into the equation for the total output currently it compares whatever is on 123.csv against... (27 Replies)
Discussion started by: llcooljatt
27 Replies

5. Shell Programming and Scripting

math calculation for a txt file

Hi All, I have a text file which is only one column. I want to multiply all these values in file1.txt by 0.01 and get the output.txt file1.txt 65 85 90 ... output.txt 0.65 0.85 0.90 ... Thanks. Please use code tags when posting data and code samples! (2 Replies)
Discussion started by: senayasma
2 Replies

6. Shell Programming and Scripting

Calculation from a file

Hi , I have a file with below format 6/1/200618:00:011348797339524635352499218493964052971 6/1/200617:00:011348672070311735349025833693964052971 6/1/200616:00:011348546711070035345679259193964052971 I want a output as below 6/1/200618:00:011348797339524635352499218493964052971... (1 Reply)
Discussion started by: manojit123
1 Replies

7. Shell Programming and Scripting

KSH SHELL: problem calculation number of lines inside compressed file

Hi Gurus, I am working with a korn shell script to simplify some operations of calculation number of lines inside compressed file. The called function (inside a cycle) is the following: ######################################### # F.ne: CheckCount #########################################... (3 Replies)
Discussion started by: GERMANICO
3 Replies

8. Shell Programming and Scripting

File Size calculation with AWK

Hello Friends, Im calculating file sizes with below AWK script. I do this before some spesific files are transferred. I run the script it works but after several running it stuck with a limit of 2147483647 (2 Gbytes -1 byte) and cant exceed this. Something is wrong and I can't proceed, would... (1 Reply)
Discussion started by: EAGL€
1 Replies

9. Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Hello Gurus, We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this . Problem Definition: /Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies

10. Shell Programming and Scripting

Help In Calculation of large values in loop

Hi Gurus, I am writing a shell script in which i need to strip out the numbers from file the values are unknown i. e. the range cannot be predicted.. and in my current program the sum of values is not coming as desired i think the value of calculation is crossing the range i.e. after some... (6 Replies)
Discussion started by: sandeepb
6 Replies

Featured Tech Videos