Sponsored Content
Full Discussion: Complex awk problem
Top Forums Shell Programming and Scripting Complex awk problem Post 302770505 by RudiC on Saturday 16th of February 2013 04:03:52 PM
Old 02-16-2013
You are right, you seem not understand the script. Did you try it at all?
file1 is read exactly once, populating the array from 0 to (the highest value in file1) * 1000. On the first line of file2, before any action on file2, the rest of the array (previous + 1 up to 999) is populated. Then, for every line (of the millions) in file2, field 1 is multiplied by 1000, truncated, and the match in the array is retrieved and printed. One single action per one single line.

I'd appreciate if you could attach (in advanced edit) a representative sample of each of your files, not necessarily millions of values, and anonymized if need be, so I can test and time the scripts above by myself. Thank you.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Complex Pipeline/Redirection/Regular Expression problem

This is a very hard unix command which I could not perform..... :( ......here it is: The file ~unx122/public/data.txt contains over 18,000 lines of text. Here are the first ten lines of the file: pagination 20657 2740 28416 31090 18145 shiner 1695 2507 9964 14512 13122 cool 29210... (1 Reply)
Discussion started by: netmaster
1 Replies

2. Shell Programming and Scripting

Complex problem about nested for loops

Hey, I'm writing this bash script that will test print me many copies of the same program but with different combos of 4 variables being between 1 and 100. Here's the code: #! /bin/bash x=0 for ((a=1; a < 101; a++)) do for ((b=1; b < 101; b++)) do for ((c=1; c < 101; c++)) do for... (4 Replies)
Discussion started by: Silverlining
4 Replies

3. Shell Programming and Scripting

Complex file count problem

Hi all! I have a question regarding possibilities to do line counts. SEARCH_VAR=TEX rsh $REM_HOST -l $REM_USER "cd $REM_DIR; ls *$SEARCH_VAR* 2> /dev/null" | sort -n | awk 'BEGIN { FS = "-" } ; { print $1"\t"$0 }' Will produce an output on the screen like this: 483 483-SOME-TEXT-1... (1 Reply)
Discussion started by: bbergstrom74
1 Replies

4. IP Networking

Need to solve complex network problem

I have a Red Hat linux server X on a x.x.0.0 network. This machine also has to communicate with another server Y on a network called y.y.0.0 Server X has two network interfaces. eth0 is configured on the x.x.0.0 network and has a default gateway on the x.x.0.0 network. In order to... (4 Replies)
Discussion started by: soliberus
4 Replies

5. Shell Programming and Scripting

Complex use with awk

Hi , I have file named docs.txt The content of the file look like this: DOC disk location Size ======= ===== ============= ========= TXT A /dev/dm-1 10 TXT B /dev/dm-2 10 BIN C ... (3 Replies)
Discussion started by: yoavbe
3 Replies

6. Shell Programming and Scripting

complex Awk Question

Hi, I have a file look likes this : --->start hir Trace file: pudwh_ora_9998.trc Sort options: fchela exeela ***************************************************************count = number of times OCI procedure was executed cpu = cpu time in seconds executing elapsed = elapsed... (3 Replies)
Discussion started by: yoavbe
3 Replies

7. Shell Programming and Scripting

Get values from 2 files - Complex "for loop and if" awk problem

Hi everyone, I've been thinking and trying/changing all day long the below code, maybe some awk expert could help me to fix the for loop I've thought, I think I'm very close to the correct output. file1 is: <boxes content="Grapes and Apples"> <box No.="Box MT. 53"> <quantity... (8 Replies)
Discussion started by: Ophiuchus
8 Replies

8. Shell Programming and Scripting

awk script (complex)

picked this up from another thread. echo 1st_file.csv; nawk -F, 'NR==FNR{a++;next} a{b++} END{for(i in b){if(b-1&&a!=b){print i";\t\t"b}else{print "NEW:"i";\t\t"b} } }' OFS=, 1st_file.csv *.csv | sort -r i need to use the above but with a slight modification.. 1.compare against 3 month... (25 Replies)
Discussion started by: slashbash
25 Replies

9. Shell Programming and Scripting

Help with Complex Awk.

Hi, I have a file. In this file when ever the word "ABC" occurs at position from 25 and 34 I would like to replace the value at postion 100 to 5 for the first 1000 rows only. I have written the following Awk command. nawk 'substr($0,25,9)=="ABC" {print $0}' filename The above command... (4 Replies)
Discussion started by: pinnacle
4 Replies

10. Shell Programming and Scripting

Complex text parsing with speed/performance problem (awk solution?)

I have 1.6 GB (and growing) of files with needed data between the 11th and 34th line (inclusive) of the second column of comma delimited files. There is also a lot of stray white space in the file that needs to be trimmed. They have DOS-like end of lines. I need to transpose the 11th through... (13 Replies)
Discussion started by: Michael Stora
13 Replies
JOIN(1) 						      General Commands Manual							   JOIN(1)

NAME
join - relational database operator SYNOPSIS
join [-an] [-e s] [-o list] [-tc] file1 file2 DESCRIPTION
Join forms, on the standard output, a join of the two relations specified by the lines of file1 and file2. If file1 is `-', the standard input is used. File1 and file2 must be sorted in increasing ASCII collating sequence on the fields on which they are to be joined, normally the first in each line. There is one line in the output for each pair of lines in file1 and file2 that have identical join fields. The output line normally con- sists of the common field, then the rest of the line from file1, then the rest of the line from file2. Fields are normally separated by blank, tab or newline. In this case, multiple separators count as one, and leading separators are dis- carded. These options are recognized: -an In addition to the normal output, produce a line for each unpairable line in file n, where n is 1 or 2. -e s Replace empty output fields by string s. -o list Each output line comprises the fields specified in list, each element of which has the form n.m, where n is a file number and m is a field number. -tc Use character c as a separator (tab character). Every appearance of c in a line is significant. SEE ALSO
sort(1), comm(1), awk(1). BUGS
With default field separation, the collating sequence is that of sort -b; with -t, the sequence is that of a plain sort. The conventions of join, sort, comm, uniq, look and awk(1) are wildly incongruous. 7th Edition April 29, 1985 JOIN(1)
All times are GMT -4. The time now is 07:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy