This looks awesome!! I have a really dumb question though, in the variables - is that expecting 2 files? one for just the products with counts and one just with skews, products and counts?
No, the variables PCF and SCF in the shell and pcf and scf in awk are the names of temp files used by the script to store sorted lists of products and sorted lists of skews for a selected product, respectively, while producing output in the END actions. The input data comes from a single file named file as marked in red in this excerpt from about eight lines from the end of the shell script:
Obviously, you can change file to any other filename you want to use. Or you could change it to "$1" and pass the name of the file you want to process as the only argument to the shell script.
This script doesn't need to sort the entire input file, but with millions of input lines these temporary sort result files could still be large. If they are too big to save in the directory where you run this script, you could add an option to the shell script to specify a different directory for these temp files.
Let me know if you're still confused.
---------- Post updated at 19:08 ---------- Previous update was at 18:45 ----------
Quote:
Originally Posted by JoshCrosby
... ... ... ---------- Post updated at 07:35 PM ---------- Previous update was at 07:12 PM ----------
Works perfect!!!
For those who want to know.
To create the .Skew_Count use this one-liner:
To create the .Product_Counts use this:
Don't forget to change file to products.txt at the end of the awk command
Don - HUGE THANK YOU!!!!!!!
By the way, using a MAC, so I don't have KornShell, used Bash without issue.
The code you have above will create the .Product_Counts file used by the script (before sorting it in reverse order by the number of hits for the product and increasing order of product name), but the script produces a .Skew_Counts file for each product entry in the top 3 list. The script never produces the entire list of skews. (You did say that some skews could appear with more than one product.) The lists of skews produced by my script, only show the skew counts for each of the displayed products; skipping all occurrences of the skew for other products.
Note that I developed and tested this on my MacBook Pro. You should have the KornShell available as /bin/ksh on any recent version of OS X.
No kidding, I wasn't aware of that ksh was on mac, never really looked though either.
1 question though for understanding. Would you mind explaining this piece, I know it's calling the products.txt file, just trying to understand it a bit better.
No kidding, I wasn't aware of that ksh was on mac, never really looked though either.
1 question though for understanding. Would you mind explaining this piece, I know it's calling the products.txt file, just trying to understand it a bit better.
Thanks again!
I added magenta line numbers to the code above to make it easier to refer to lines for this discussion.
Since there is no condition on line 1, lines 1 through 9 will be executed for each line read from the input file.
Line 3 adds 1 to p[$1]. $1 is the product from the input file. So, p[$1] is the number of times that the product specified in the first field of this input line has been seen so far in the input file.
Line 4 tests whether or not the skew listed in the 2nd column of this input line has been seen on any other line we have read from the input file before.
If this line is the first line that has this skew ($2) for this product ($1), line 6 increments the number of different skews that have been seen with this product (++plc[$1]) and saves this skew ($2) in the list of skews associated with this product (pl[$1, plc[$1]] = $2).
Then line 8 increments the number of times this skew has been seen with this product (s[$1, $2]++).
This User Gave Thanks to Don Cragun For This Post:
The awk below is supposed to count all the matching $5 strings and count how many $7 values is less than 20. I don't think I need the portion in bold as I do not need any decimal point or format, but can not seem to get the correct counts. Thank you :).
file
chr5 77316500 77316628 ... (6 Replies)
I have a data which looks like
1440993600|L|ABCDEF
1440993600|L|ABCD
1440993601|L|ABCDEF
1440993602|L|ABC
1440993603|L|ABCDE
.
.
.
1441015200|L|AB
1441015200|L|ABC
1441015200|L|ABCDEF
So basically, the $1 is epoch date, $2 and $3 is some application data
From one if the... (5 Replies)
Hello
Im new treat me nicely, I have a headache :)
I have a script that seemed to work now it doesnt anyway, the last part is adding counts of unique items in a csv file eg
05492U34 38
05492U34 47
two columns, (many different values like this in file)
i want... (7 Replies)
I have below inside a file.
11.22.33.44
user1
11.22.33.55
user2
I need this manipulated as
alias server1.domain.com='ssh user1@11.22.33.44'
alias server2.domain.com='ssh user2@11.22.33.55' (3 Replies)
Hello folks.
After awk, i have decided to start to learn perl, and i need some help.
I have following output :
1 a
1 b
2 k
2 f
3 s
3 p
Now with awk i get desired output by issuing :
awk ' { a = a FS $2 } END { for ( i in a) print i,a }' input
1 a b
2 k f
3 s p
Can... (1 Reply)
Hi,
I need an awk script (or whatever shell-construct) that would take data like below and get the max value of 3 column, when grouping by the 1st column.
clientname,day-of-month,max-users
-----------------------------------
client1,20120610,5
client2,20120610,2
client3,20120610,7... (3 Replies)
Hello
I am trying to figure out a script which could group a log file by user names. I worked with awk command and I could trim the log file to:
<USER: John Frisbie > /* Thu Aug 06 2009 15:11:45.7974 */ FLOAT GRANT WRITE John Frisbie (500 of 3005 write)
<USER: Shawn Sanders > /* Thu Aug 06... (2 Replies)
I run awk
cat $1|awk '{print $6}'
and get a lot of results and I want results to group them. For example my result is (o/p is unknown to user)
xyz
xyz
abc
pqr
xyz
pqr
etc
I wanna group them as
xyz=total found 7
abc=total ....
pqr=
Thank (3 Replies)
To start I have a table that has ticketholders. Each ticket holder has a unique number and each ticket holder is associated to a so called household number. You can have multiple guests w/i a household.
I would like to create 3 flags (form a, for a household that has 1-4 gst) form b 5-8 gsts... (3 Replies)