Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Completey new to scripting, question/help?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Completey new to scripting, question/help?
# 15  
Old 05-29-2015
What's driving people crazy is moving targets or creeping specs. Mayhap you can talk the CEO into valueing the time spent upfront?

Hint: use sub() (or split) to split the user and/or remove his ".com", and yes, change the sort keys. All in above script.
# 16  
Old 05-29-2015
Quote:
Originally Posted by RudiC
What's driving people crazy is moving targets or creeping specs. Mayhap you can talk the CEO into valueing the time spent upfront?
Agreed. Smilie

Quote:
Hint: use sub() (or split) to split the user and/or remove his ".com", and yes, change the sort keys. All in above script.
Thanks for the hint. So here's what I did below:

Code:
awk '
match($0, /"username":"[^"]*"/) {
        split($3, d, "@")
        user = substr($0, RSTART + 12, RLENGTH - 17)
        split(user, e, "@")
        c[e[2] "," d[1] "," e[1]]++
}
END {   for(i in c)
                printf("%d,%s\n", c[i], i)
}'      company.log | sort -t, -k2,2 -k3,3 -k4,4

This works, now I just need to get the first column in the output (which is the number assigned to the total number of entries per user), to become the fourth column in the output.
Last edited by xdawg; 05-29-2015 at 07:42 PM..
# 17  
Old 05-29-2015
Quote:
Originally Posted by xdawg
Agreed. Smilie



Thanks for the hint. So here's what I did below:

Code:
awk '
match($0, /"username":"[^"]*"/) {
        split($3, d, "@")
        user = substr($0, RSTART + 12, RLENGTH - 17)
        split(user, e, "@")
        c[e[2] "," d[1] "," e[1]]++
}
END {   for(i in c)
                printf("%d,%s\n", c[i], i)
}'      liquidgrids.log | sort -t, -k2,2 -k3,3 -k4,4

This works, now I just need to get the first column in the output (which is the number assigned to the total number of entries per user), to become the fourth column in the output.
I don't see anything in the above script that gets rid of the .com. And, I don't think I understand what:
  • you're trying to count,
  • the order of the output fields, nor
  • the desired output sort order (I think the sort keys are org, date, and user; but I'm not sure).
I assume that the even though joe@mycompany.com and joe@yourcompany.org both have user joe, you don't really want counts for joe do you? What about joe@abc.com, joe@abc.net, and joe@abc.org? Do you really believe that abc.com, abc.net, and abc.org are all addresses for the abc organization, or could these be three different organizations? Does your CEO really want to strip off the .com, .net, and .AnyOtherThing; or was that another knee jerk reaction without thinking about the consequences?

If you could give us about a 10 line sample log file and show us the exact output that your CEO wants from that sample, we might be willing to make one more attempt at a working solution.
# 18  
Old 05-29-2015
Quote:
Originally Posted by Don Cragun
If you could give us about a 10 line sample log file and show us the exact output that your CEO wants from that sample, we might be willing to make one more attempt at a working solution.
Ok sorry, lets see if I can clarify from the following log sample:


Code:
 
 qtp111659197-5776 - 05-26@09:37:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"alan@companyA.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},)
 qtp111659197-5776 - 05-26@09:38:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"alan@companyA.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},) 
 qtp111659197-5776 - 05-26@09:39:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"bryan@companyB.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},) 
 qtp111659197-5776 - 05-26@09:40:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"carl@companyA.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},) 
 qtp111659197-5776 - 05-26@09:41:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"dan@companyB.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},) 
 qtp111659197-5776 - 05-27@09:37:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"alan@companyA.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},) 
 qtp111659197-5776 - 05-27@09:38:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"bryan@companyB.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},) 
 qtp111659197-5776 - 05-27@09:39:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"carl@companyA.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},) 
 qtp111659197-5776 - 05-27@09:40:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"dan@companyB.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},)
 qtp111659197-5776 - 05-27@09:41:34:240 INFO  (TimingInfoProxy.java:41)     - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"dan@companyB.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},)

So basically I need to export his requirements into a spreadsheet. He wants to be able to see the data sorted first by organization, then by date, then user alphanumerically (within that organization), and finally the number of entries for that user for that specified date.

So for the first user with an entry in the sample log, alan@companyA.com, he had two log entries on 05-26 and one entry on 05-27. So output for just him would look like this:

Code:
 
 companyA,05-26,alan,2
 companyA,05-27,alan,1

And this is what it would look like for the entire 10 line sample log:

Code:
 
 companyA,05-26,alan,2
 companyA,05-26,carl,1
 companyA,05-27,alan,1
 companyA,05-27,carl,1
 companyB,05-26,bryan,1
 companyB,05-26,dan,1
 companyB,05-27,bryan,1
 companyB,05-27,dan,2

---------- Post updated at 03:32 PM ---------- Previous update was at 02:26 PM ----------

FWIW this seems to work:

Code:
 
 awk '
match($0, /"username":"[^"]*"/) {
        split($3, d, "@")
        user = substr($0, RSTART + 12, RLENGTH - 17)
        split(user, e, "@")
        c[e[2] "," d[1] "," e[1]]++
}
END {   for(i in c)
                printf("" "%s,%d\n", i, c[i])
}'      liquidgrids.log | sort -t, -k1,1 -k2,2 -k3,3

The way I got rid of the .com from the username was changing
Code:
RLENGTH - 13

into
Code:
RLENGTH - 17

Not sure if that's a proper way to go about this or if it would cause problems that I'm unaware of.
# 19  
Old 05-29-2015
This does what you asked for this time:
Code:
awk '
match($0, /"username":"[^"]*"/) {
	split($3, d, "@")
	email = substr($0, RSTART + 12, RLENGTH - 13)
	split(email, e, "[@.]")
	c[e[2] "," d[1] "," e[1]]++
}
END {	for(i in c)
		printf("%s,%d\n", i, c[i])
}' file | sort -t, -k1,3

# 20  
Old 05-29-2015
Quote:
Originally Posted by Don Cragun
This does what you asked for this time:
Code:
awk '
match($0, /"username":"[^"]*"/) {
    split($3, d, "@")
    email = substr($0, RSTART + 12, RLENGTH - 13)
    split(email, e, "[@.]")
    c[e[2] "," d[1] "," e[1]]++
}
END {    for(i in c)
        printf("%s,%d\n", i, c[i])
}' file | sort -t, -k1,3

Thanks again Don! Smilie
# 21  
Old 05-29-2015
Current common domains in the US include .co, tv, and .travel as well as .com, .net, and .org. And outside the US there are common domains like .co.uk and .com.au.

Using RLENGTH - 17 works for .com and other 3 letter domains; but it won't work for the others. Splitting on "@" and on "." works on all domains. However, if you have usernames that contain a period (e.g., first.last@company.domain, it won't work. If this is (or, in the future, could be) a problem for you, change:
Code:
	split(email, e, "[@.]")
	c[e[2] "," d[1] "," e[1]]++

to:
Code:
	split(email, e, "[@]")
	split(e[2], a, ".")
	c[a[1] "," d[1] "," e[1]]++
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Scripting question

hi all, I am writing a script and beginner in shell scripting. I have tried the below script. could you please check and let me know whether the below scirpt is correct. Unix details : HP Unix Input file. cat input.txt | tail -4 HTS40002.W1978.PROM HTS40003.W1978.PROM... (17 Replies)
Discussion started by: arun888
17 Replies

2. Shell Programming and Scripting

Scripting question

Preview of command prompt f ---> to start ferret q----> to stop ferret asp@nex:~$ f NOAA/PMEL TMAP FERRET v6.82 Linux 2.6.18-308.8.2.el5PAE 32-bit - 08/03/12 3-Dec-12 16:44 yes? go my.jnl yes?column=4/skip=1/type=num,text ............filename.txt ---... (4 Replies)
Discussion started by: nex_asp
4 Replies

3. UNIX for Dummies Questions & Answers

Scripting question

folks; I have a script to remove any files that older than 14 days then move any files that younger than 7 days to another directory. but for some reason it doesn't move the files, when i do it manually it works but not through the script. i tried 2 different ways in writing the move part but it... (6 Replies)
Discussion started by: Katkota
6 Replies

4. Shell Programming and Scripting

Scripting question

Folks; I'm writing a shell script to extract some fields out of a log file & it will run periodically, how can i make it runs starting from where it left of. for example; if the script will do the extract every 2 days, let's say the first run will extract fields until July 25, 2007 @ 11:15:22... (1 Reply)
Discussion started by: moe2266
1 Replies

5. Solaris

Scripting question

I'm writing a small script that will run an executable program (sort of like TOP). To exit the executable, you have to enter control C (^c). I'm trying to use a redirect input file to send the ^c but I'm not having any luck. My short script looks like this - /mydirectory/abc.script < abc.in >... (1 Reply)
Discussion started by: gonzotonka
1 Replies

6. Shell Programming and Scripting

scripting question?

I am writing a backup script for AIX 5 and running into a problem where the output isn't being shown in the output log that is being created. Any ideas on how this would be corrected? I have included the script below. The only thing showing up in the file is listed below. I was hoping to capture... (2 Replies)
Discussion started by: justinburbridge
2 Replies

7. Shell Programming and Scripting

scripting question

I'm new to shell scripting and am having a problem trying to do something in C shell. I want to write a script that will input something instead of a user doing it. For example, using the command 'write' the user is supposed to type something to be sent to another user. I want a script to be able... (3 Replies)
Discussion started by: batmike
3 Replies

8. Shell Programming and Scripting

another scripting question

Hello I am working on cleaning up permissions on Oracle mountpoints and datafiles in unix. I am looking for a script or a scripting idea to 1st. 1. grep for owner oracle 2. ensure its a directory owned for oracle 3. chmod 750 on the oracle owned directory. 4. grep for oracle files, etc... (3 Replies)
Discussion started by: jigarlakhani
3 Replies

9. Shell Programming and Scripting

Scripting Question

This script searches for core files and if it finds one, it emails me to let me know.I DONT want it to email me if it doesn't find one but I can't figure out what I need to change or add. Any thoughts? Script below: /bin/find / -name core -type f -ls -exec file {} \;|/usr/bin/mailx -s... (1 Reply)
Discussion started by: damielle
1 Replies

10. UNIX for Dummies Questions & Answers

another scripting question

I am writing a script that will identify the oldest file in a directory. Here's the syntax: #!/bin/ksh cd directory chmod 777 * ls -r -1t > file1 sed -n -e "1P" < file1 > file2 So my problem is, now I have file2, which contains the name of the oldest file in the directory. How do I use,... (1 Reply)
Discussion started by: kristy
1 Replies
Login or Register to Ask a Question