It returns the data I need, the only other issue is that it also returns counts for entries that aren't user related as the log files do contain entries that don't have to do with user activity.
Do the log entries that you want to exclude from the report contain "username":"email@address" strings? If they do, how do you identify the log entries you want to select or the log entries you want to exclude?
Basically, this is another request for a clear definition of your input file format.
Do the log entries that you want to exclude from the report contain "username":"email@address" strings? If they do, how do you identify the log entries you want to select or the log entries you want to exclude?
Basically, this is another request for a clear definition of your input file format.
The log entries that I would like excluded do not contain
Thanks RudiC, the output was mostly correct, however some lines of output were just blank space after the date (I added the commas so later I can output into csv easily):
Code:
3, 05-22 ,
Hmm, I'm learning a lot as we go along. I've been breaking down what DonC wrote (RudiC's will be next probably tonight when I get home), and I think I just need to be able to tell it not to output any line that returns a 0 (no match found) when this is run:
Code:
match($0, /"username":"[^"]*"/)
I just haven't figured out how to tell it that, maybe I am thinking about this the wrong way.
Thanks RudiC, the output was mostly correct, however some lines of output were just blank space after the date (I added the commas so later I can output into csv easily):
Code:
3, 05-22 ,
Hmm, I'm learning a lot as we go along. I've been breaking down what DonC wrote (RudiC's will be next probably tonight when I get home), and I think I just need to be able to tell it not to output any line that returns a 0 (no match found) when this is run:
Code:
match($0, /"username":"[^"]*"/)
I just haven't figured out how to tell it that, maybe I am thinking about this the wrong way.
You're learning quickly. I also submitted something that did it slightly differently, but RudiC posted his before I posted mine, so I deleted it. I am reposting mine here (with changes for CSV format output without the spaces around the commas) hoping that something in your log data contains the string username but does not match the pattern "username":"user@host" (which the code below would treat differently than RudiC's suggestion does):
Code:
awk '
match($0, /"username":"[^"]*"/) {
split($3, d, "@")
user = substr($0, RSTART + 12, RLENGTH - 13)
c[d[1] "," user]++
}
END { for(i in c)
printf("%d,%s\n", c[i], i)
}' logs/mycompany.log | sort -t, -k3,3 -k2,2
This can still produce output like you've shown above, if a log entry contains the literal string "username":"" (and maybe also if the second quoted string contains blanks). So, let us know if the above code still produces unwanted output and we can try to track down the log entries causing the problem and eliminate them from consideration.
---------- Post updated 05-29-15 at 11:24 AM ---------- Previous update was 05-28-15 at 08:06 PM ----------
Ok, so I'm back, the CEO has changed how he wants the output sorted/presented.
He wants it primarily sorted alphanumerically by organization which is the second part of the username after the "@".
Then the secondary sort key would be increasing date. And finally a third sort key would be alphanumeric using just the user without the org part.
So thinking it through, if I were to use Don your last piece of code:
Code:
awk '
match($0, /"username":"[^"]*"/) {
split($3, d, "@")
user = substr($0, RSTART + 12, RLENGTH - 13)
c[d[1] "," user]++
}
END { for(i in c)
printf("%d,%s\n", c[i], i)
}' liquidgrids.log | sort -t, -k3,3 -k2,2
And here is a sample entry again:
Code:
qtp111659197-5776 - 05-26@09:37:34:240 INFO (TimingInfoProxy.java:41) - com.mycompany.api.ApiHandler-0>getUniqueDataBySource(data,{"has_values":false,"last_event_triggered":"","user_info":{"username":"joe@mycompany.com","orgid":"69d467a7-9786-47e1-9c12-bb40f9bfc65d","ip":"127.0.0.1"},"date_range":{"min_date":"","start_date":"","end_date":"","trending_start_date":"","trending_end_date":""},"terms":{"and_filtering":[]}},)
We would have to split the user key into two pieces, user (in this example joe) and org (in this example mycompany without the .com)? And then change the sort order of the keys at the end of the script? Or would this require a second script?
hi all,
I am writing a script and beginner in shell scripting. I have tried the below script. could you please check and let me know whether the below scirpt is correct.
Unix details : HP Unix
Input file.
cat input.txt | tail -4
HTS40002.W1978.PROM
HTS40003.W1978.PROM... (17 Replies)
Preview of command prompt
f ---> to start ferret
q----> to stop ferret
asp@nex:~$ f
NOAA/PMEL TMAP
FERRET v6.82
Linux 2.6.18-308.8.2.el5PAE 32-bit - 08/03/12
3-Dec-12 16:44
yes? go my.jnl
yes?column=4/skip=1/type=num,text ............filename.txt
---... (4 Replies)
folks;
I have a script to remove any files that older than 14 days then move any files that younger than 7 days to another directory. but for some reason it doesn't move the files, when i do it manually it works but not through the script. i tried 2 different ways in writing the move part but it... (6 Replies)
Folks;
I'm writing a shell script to extract some fields out of a log file & it will run periodically, how can i make it runs starting from where it left of. for example;
if the script will do the extract every 2 days, let's say the first run will extract fields until July 25, 2007 @ 11:15:22... (1 Reply)
I'm writing a small script that will run an executable program (sort of like TOP). To exit the executable, you have to enter control C (^c). I'm trying to use a redirect input file to send the ^c but I'm not having any luck. My short script looks like this - /mydirectory/abc.script < abc.in >... (1 Reply)
I am writing a backup script for AIX 5 and running into a problem where the output isn't being shown in the output log that is being created. Any ideas on how this would be corrected? I have included the script below. The only thing showing up in the file is listed below. I was hoping to capture... (2 Replies)
I'm new to shell scripting and am having a problem trying to do something in C shell. I want to write a script that will input something instead of a user doing it. For example, using the command 'write' the user is supposed to type something to be sent to another user. I want a script to be able... (3 Replies)
Hello
I am working on cleaning up permissions on Oracle mountpoints and datafiles in unix. I am looking for a script or a scripting idea to 1st.
1. grep for owner oracle
2. ensure its a directory owned for oracle
3. chmod 750 on the oracle owned directory.
4. grep for oracle files, etc... (3 Replies)
This script searches for core files and if it finds one, it emails me to let me know.I DONT want it to email me if it doesn't find one but I can't figure out what I need to change or add. Any thoughts? Script below:
/bin/find / -name core -type f -ls -exec file {} \;|/usr/bin/mailx -s... (1 Reply)
I am writing a script that will identify the oldest file in a directory. Here's the syntax:
#!/bin/ksh
cd directory
chmod 777 *
ls -r -1t > file1
sed -n -e "1P" < file1 > file2
So my problem is, now I have file2, which contains the name of the oldest file in the directory. How do I use,... (1 Reply)
