Sponsored Content
Full Discussion: Extracting from log file
Top Forums UNIX for Beginners Questions & Answers Extracting from log file Post 303003175 by nextStep on Friday 8th of September 2017 01:23:29 PM
Old 09-08-2017
Extracting from log file

Hi All,



I have a huge log file where user information such as name,address,point balance etc are stored.
I need to extract only point balance,first name,last name only.
How to achieve this, tried with awk and jq but could not get the result.


Log file

Code:
app2.hostname/log.2017-08-05.gz:[2017-08-05 11:43:42,508] app2.hostname 1501947827514 NA:NA:NA http-nio-8080-exec-1 INFO  Rest Response: 200  [Headers: {X-Application-Context=[application:9015], Content-Type=[application/json;charset=UTF-8], Transfer-Encoding=[chunked], Date=[Sat, 05 Aug 2017 15:43:42 GMT]}] {"accountStatus":{"varStatusInfo":{},"accessType":"READ_ONLY"},"accountBalance":{"pointsBalance":111834},"userInformation":{"firstName":"xx","lastName":"xx","address":{"line1":"14 KETTLEWELL WAY","line2":"","city":"OTTAWA","stateCode":"ON","postalCode":"K2W1G3","countryCode":"CA"},"phoneNumbers":[{"type":"OTHER","number":"xx"}],"emailAddresses":[{"type":"OTHER","email":"xx@gmail.com"}],"additionalInfo":{"SourceCode":"2"}}} (LoggingRestRequestResponseInterceptor)

app10/log.2017-07-01.gz:[2017-07-01 00:20:56,836] app10.hostname 1498882864809 NA:NA:NA http-nio-8080-exec-23 INFO  Got successful response for the url GET http://hostname:8080/sss/accounts/999999. Response: {"accountBalance":{"pointsBalance":29878},"accountStatus":{"accessType":"READ_ONLY","varStatusInfo":{}},"userInformation":{"additionalInfo":{"SourceCode":"2"},"address":{"line1":"14879 86 AVENUE","line2":"","city":"yy","stateCode":"BC","postalCode":"V3S7E6","countryCode":"CA"},"emailAddresses":[{"email":"yy@gmail.com","type":"OTHER"}],"firstName":"yy","lastName":"yy","phoneNumbers":[{"number":"77777","type":"OTHER"}]}} (HttpClientUtil)


Last edited by RudiC; 09-08-2017 at 04:12 PM.. Reason: Corrected code tags
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Extracting the required text from log files

It would be highly appreciable if any one helps me in this. I am trying to get it done through Java but I love unix and believe it can be done within minutes with couple of lines. The input log file is a text file contains multiple entries seperated by a blank line. Each seperated entry... (7 Replies)
Discussion started by: hareeshram
7 Replies

2. UNIX for Dummies Questions & Answers

Extracting only words from a log file

hello: i have a file and i am trying to extract only unique words from that file. i used the command: cat messages.1 | tr " " "\n" | sort | uniq -c but using this command outputs everything unique in the file be it words, numbers, like all the characters..i need a command which will only... (6 Replies)
Discussion started by: vikbenq
6 Replies

3. Homework & Coursework Questions

extracting date from log file

You are given a 1 year logfile with each line starting with a date in the form “YYYY-MM-DD”. How would you extract logs from the 4th day of each month and put them into a new file (1 Reply)
Discussion started by: DOkuwa
1 Replies

4. Shell Programming and Scripting

Extracting data from a log file with date formats

Hello, I have a log file for the year, which contains lines starting with the data in the format of YYYY-MM-DD. I need to get all the lines that contain the DD being 04, how would I do this? I tried using grep "*-*04" but it didn't work. Any quick one liners I should know about? Thank you. (2 Replies)
Discussion started by: cpickering
2 Replies

5. Shell Programming and Scripting

Extracting log entries from a date onwards

One of the log file looks like entries as below. Wed Apr 6 14:51:18 2011 FAIL LOGIN: Client "9.191.21.54" Wed Apr 6 14:52:53 2011 CONNECT: Client "9.191.21.54" Wed Apr 6 14:52:54 2011 OK LOGIN: Client "9.191.21.54" Wed Apr 6 14:55:10 2011 CONNECT: Client "9.191.21.54" Wed Apr 6... (2 Replies)
Discussion started by: rijeshpp
2 Replies

6. Shell Programming and Scripting

Extracting strings from a log file.

I'm new to all this and I've been fiddling with this problem for HOURS and feel silly that I can't work it out! I have a .log file that VERY long and looks like this: 2011-08-31 10:03:34 SUESTART AG Amndmnt Client WebRequest DNU SUEEND Sequence: 600, 2011-08-31 10:03:34 SUESTART... (11 Replies)
Discussion started by: SusieSA
11 Replies

7. Shell Programming and Scripting

Extracting the last 10mins worth of data in a log file

Hi all, Hope someone here will be able to help me. Our system has some scripts that are run from a cron job every ten mins and is used to see how many error there are in that time frame. Problem is that in the scripts grep is used to find the data, but as the day goes on these log file grow to a... (7 Replies)
Discussion started by: Goldengreen
7 Replies

8. Shell Programming and Scripting

Extracting Delimiter 'TAG' Data From log files

Hi I am trying to extract data from within a log file and output format to a new file for further manipulation can someone provide script to do this? For example I have a file as below and just want to extract all delimited variances of tag 32=* up to the delimiter "|" and output to a new file... (2 Replies)
Discussion started by: Buddyluv
2 Replies

9. Shell Programming and Scripting

Error while extracting data from log file

I am running awk command to extract data from log file to calculate last 15 minutes log using below command and now i am getting bellow error: awk '$0>=$from' from=$(`date -u +"####<%d-%b-%Y %H:%M:%S o'clock GMT>"-15min`) test.log Error: date: 0551-402 Invalid character in date/time... (8 Replies)
Discussion started by: oberoi1403
8 Replies

10. Shell Programming and Scripting

Extracting data from rsync log

Hi I have a daily rsync that runs and i am trying to find a easy way of extracting the start time and end time of the sync and extract the details of how much data was copied. I would like to use this information to see if i can increase the amount of rsyncs that run in a day. so the log... (3 Replies)
Discussion started by: treds
3 Replies
HTTP::Response::Encoding(3pm)				User Contributed Perl Documentation			     HTTP::Response::Encoding(3pm)

NAME
HTTP::Response::Encoding - Adds encoding() to HTTP::Response VERSION
$Id: Encoding.pm,v 0.5 2007/05/12 09:24:15 dankogai Exp $ SYNOPSIS
use LWP::UserAgent; use HTTP::Response::Encoding; my $ua = LWP::UserAgent->new(); my $res = $ua->get("http://www.example.com/"); warn $res->encoding; EXPORT
Nothing. METHODS
This module adds the following methods to HTTP::Response objects. "$res->charset" Tells the charset exactly as appears in the "Content-Type:" header. Note that the presence of the charset does not guarantee if the response content is decodable via Encode. To normalize this, you should try $res->encoder->mime_name; # with Encode 2.21 or above or use I18N::Charset; # ... mime_charset_name($res->encoding); "$res->encoder" Returns the corresponding encoder object or undef if it can't. "$res->encoding" Tells the content encoding in the canonical name in Encode. Returns undef if it can't. For most cases, you are more likely to successfully find encoding after GET than HEAD. HTTP::Response is smart enough to parse <meta http-equiv="Content-Type" content="text/html; charset=whatever"/> But you need the content to let HTTP::Response parse it. If you don't want to retrieve the whole content but interested in its encoding, try something like below; my $req = HTTP::Request->new(GET => $uri); $req->headers->header(Range => "bytes=0-4095"); # just 1st 4k my $res = $ua->request($req); warn $res->encoding; "$res->decoded_content" Discontinued since HTTP::Message already has this method. See HTTP::Message for details. INSTALLATION
To install this module, run the following commands: perl Makefile.PL make make test make install AUTHOR
Dan Kogai, "<dankogai at dan.co.jp>" BUGS
Please report any bugs or feature requests to "bug-http-response-encoding at rt.cpan.org", or through the web interface at <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=HTTP-Response-Encoding>. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes. SUPPORT
You can find documentation for this module with the perldoc command. perldoc HTTP::Response::Encoding You can also look for information at: o AnnoCPAN: Annotated CPAN documentation <http://annocpan.org/dist/HTTP-Response-Encoding> o CPAN Ratings <http://cpanratings.perl.org/d/HTTP-Response-Encoding> o RT: CPAN's request tracker <http://rt.cpan.org/NoAuth/Bugs.html?Dist=HTTP-Response-Encoding> o Search CPAN <http://search.cpan.org/dist/HTTP-Response-Encoding> ACKNOWLEDGEMENTS
GAAS for LWP. MIYAGAWA for suggestions. COPYRIGHT &; LICENSE Copyright 2007 Dan Kogai, all rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.10.0 2009-07-12 HTTP::Response::Encoding(3pm)
All times are GMT -4. The time now is 05:55 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy