I have a huge log file where user information such as name,address,point balance etc are stored.
I need to extract only point balance,first name,last name only.
How to achieve this, tried with awk and jq but could not get the result.
Log file
Code:
app2.hostname/log.2017-08-05.gz:[2017-08-05 11:43:42,508] app2.hostname 1501947827514 NA:NA:NA http-nio-8080-exec-1 INFO Rest Response: 200 [Headers: {X-Application-Context=[application:9015], Content-Type=[application/json;charset=UTF-8], Transfer-Encoding=[chunked], Date=[Sat, 05 Aug 2017 15:43:42 GMT]}] {"accountStatus":{"varStatusInfo":{},"accessType":"READ_ONLY"},"accountBalance":{"pointsBalance":111834},"userInformation":{"firstName":"xx","lastName":"xx","address":{"line1":"14 KETTLEWELL WAY","line2":"","city":"OTTAWA","stateCode":"ON","postalCode":"K2W1G3","countryCode":"CA"},"phoneNumbers":[{"type":"OTHER","number":"xx"}],"emailAddresses":[{"type":"OTHER","email":"xx@gmail.com"}],"additionalInfo":{"SourceCode":"2"}}} (LoggingRestRequestResponseInterceptor)
app10/log.2017-07-01.gz:[2017-07-01 00:20:56,836] app10.hostname 1498882864809 NA:NA:NA http-nio-8080-exec-23 INFO Got successful response for the url GET http://hostname:8080/sss/accounts/999999. Response: {"accountBalance":{"pointsBalance":29878},"accountStatus":{"accessType":"READ_ONLY","varStatusInfo":{}},"userInformation":{"additionalInfo":{"SourceCode":"2"},"address":{"line1":"14879 86 AVENUE","line2":"","city":"yy","stateCode":"BC","postalCode":"V3S7E6","countryCode":"CA"},"emailAddresses":[{"email":"yy@gmail.com","type":"OTHER"}],"firstName":"yy","lastName":"yy","phoneNumbers":[{"number":"77777","type":"OTHER"}]}} (HttpClientUtil)
Last edited by RudiC; 09-08-2017 at 04:12 PM..
Reason: Corrected code tags
The problem here is there is anther text in the log file named named [(LoggingRestRequestResponseInterceptor)] . I am not able to get the fields in that pattern using the step 1.
Step 1 scans and takes all the fields which is in between (HttpClientUtil).
If i can combine the second pattern [(LoggingRestRequestResponseInterceptor)] in the step 1 i believe it will be outputing all the required fields.
This is where struggling. ---------- Post updated at 03:38 AM ---------- Previous update was at 01:41 AM ----------
Tried the below code in getting the fields between the keyword (LoggingRestRequestResponseInterceptor)
Although this modifies and extends the specs in post#1, it doesn't make things much clearer. And, your code samples don't really help...
Wildly guessing / assuming that
- every record in in one single line
- "in between" means "in the same line"
- you want records with either keyword
- the keyword doesn't need to be listed in the output
, would this come close to what you need?
Sorry , if it confused. This is exactly the requirement.And it worked .Thank you.
Could you please tell what the below code does.
The period at the beginning and parameters inside the square bracket
Code:
grep -Eo ".((first|last)Name|pointsBalance)[^,}]*
Moderator's Comments:
Please use CODE tags as required by forum rules!
Last edited by RudiC; 09-11-2017 at 07:04 AM..
Reason: Added CODE tags.
A bracket expression is a list of characters enclosed in "[ ]". It normally matches any single character from the list ... If the list begins with '^', it matches any single character ... not from the rest of the list.
As the field values follow the field name and are terminated by either a comma or a right brace, that regex part allows for all chars EXCEPT those two and is terminated by either.
The period at the beginning allows for any char, esp. the double quote for the field names. It's a lazy way to do so; you could also specify an escaped double quote.
Last edited by rbatte1; 09-11-2017 at 08:13 AM..
Reason: Separated square brackets, else they appear to make a single square character.
Hi
I have a daily rsync that runs and i am trying to find a easy way of extracting the start time and end time of the sync and extract the details of how much data was copied.
I would like to use this information to see if i can increase the amount of rsyncs that run in a day.
so the log... (3 Replies)
I am running awk command to extract data from log file to calculate last 15 minutes log using below command and now i am getting bellow error:
awk '$0>=$from' from=$(`date -u +"####<%d-%b-%Y %H:%M:%S o'clock GMT>"-15min`) test.log
Error:
date: 0551-402 Invalid character in date/time... (8 Replies)
Hi
I am trying to extract data from within a log file and output format to a new file for further manipulation can someone provide script to do this?
For example I have a file as below and just want to extract all delimited variances of tag 32=* up to the delimiter "|" and output to a new file... (2 Replies)
Hi all,
Hope someone here will be able to help me. Our system has some scripts that are run from a cron job every ten mins and is used to see how many error there are in that time frame. Problem is that in the scripts grep is used to find the data, but as the day goes on these log file grow to a... (7 Replies)
I'm new to all this and I've been fiddling with this problem for HOURS and feel silly that I can't work it out!
I have a .log file that VERY long and looks like this:
2011-08-31 10:03:34 SUESTART AG Amndmnt Client WebRequest DNU SUEEND Sequence: 600,
2011-08-31 10:03:34 SUESTART... (11 Replies)
Hello,
I have a log file for the year, which contains lines starting with the data in the format of YYYY-MM-DD. I need to get all the lines that contain the DD being 04, how would I do this? I tried using grep "*-*04" but it didn't work.
Any quick one liners I should know about?
Thank you. (2 Replies)
You are given a 1 year logfile with each line starting with a date in the form “YYYY-MM-DD”. How would you extract logs from the 4th day of each month and put them into a new file (1 Reply)
hello:
i have a file and i am trying to extract only unique words from that file.
i used the command: cat messages.1 | tr " " "\n" | sort | uniq -c
but using this command outputs everything unique in the file be it words, numbers, like all the characters..i need a command which will only... (6 Replies)
It would be highly appreciable if any one helps me in this. I am trying to get it done through Java but I love unix and believe it can be done within minutes with couple of lines.
The input log file is a text file contains multiple entries seperated by a blank line.
Each seperated entry... (7 Replies)
09-11-2017
