Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How the filter output?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How the filter output?
# 1  
Old 01-20-2014
How the filter output?
Hey,

I'm using some sensors that can be read by http.
If I run following command:

Code:
curl -v 'http://192.168.111.23:8080/sensor/52f5c63cc4221fbbedcb499908a0a823?version=1.0&interval=minute&unit=watt&callback=realtime'

I'm getting:
Quote:
* About to connect() to 192.168.111.23 port 8080 (#0)
* Trying 192.168.111.23... connected
* Connected to 192.168.111.23 (192.168.111.23) port 8080 (#0)
> GET /sensor/52f5c63cc4221fbbedcb499908a0a823?version=1.0&interval=minute&unit=watt&callback=realtime HTTP/1.1
> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 192.168.111.23:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/json
<
realtime([[1390215660,1110],[1390215661,1113],[1390215662,1109],[1390215663,1113],[1390215664,1114],[1390215665,1120],[1390215666,1120],[1390215667,1118],[1390215668,1121],[1390215669,1120],[1390215670,1117],[1390215671,1122],[1390215672,1114],[1390215673,1105],[1390215674,1111],[1390215675,1106],[1390215676,1111],[1390215677,1118],[1390215678,1117],[1390215679,1116],[1390215680,1108],[1390215681,1106],[1390215682,1112],[1390215683,1106],[1390215684,1114],[1390215685,1112],[1390215686,1109],[1390215687,1106],[1390215688,1109],[1390215689,1111],[1390215690,1105],[1390215691,1114],[1390215692,1111],[1390215693,1108],[1390215694,1113],[1390215695,1106],[1390215696,1102],[1390215697,1117],[1390215698,1107],[1390215699,1104],[1390215700,1108],[1390215701,1109],[1390215702,1111],[1390215703,1112],[1390215704,1110],[1390215705,1105],[1390215706,1109],[1390215707,1109],[1390215708,1107],[1390215709,1114],[1390215710,1110],[1390215711,1107],[1390215712,1110],[1390215713,1121],[1390215714,1111],[1390215715,1107],[13902* Closing connection #0
15716,1113],[1390215717,1111],[1390215718,1104],[1390215719,1111]]);

I would like to put this now in a sheet with only the 'unique' values of date and value.
So I would like to have something as:
Quote:
1390215660,1110
1390215661,1113
1390215662,1109
1390215663,1113
...

The endgame is that I can put this in a graph in my monitoring tool (cacti).

Any suggestions what the best way is?


I was playing with putting the results in a file (sensor1.data) and filter it with sed, but this won't deal with any double entries...
Code:
sed -n '/</!p' sensor1.data | sed -n '/>/!p' | sed 's/\(.\{10\}\)//' | sed 's/.$//' | sed 's/\[/\m/g' | sed 's/\]/\m/g' | sed s/m,m/\ /g | sed 's/m//g' | sed
 's/\"//g' | sed 's/nan/0/g'

# 2  
Old 01-20-2014
Hello,

Not sure if following will meet your requirement, but it doesn't please let us know the expected output.

Code:
awk -F"[ ]" '{print $1}' file-name | sed 's/\[//g;s/\]/ /g;s/[[:alpha:]]//g' | tr ' ' '\n' | sed 's/\(^,\)\(.*\)/\2/g' | awk 'gsub(/\(/,X) gsub(/\)/,Y) gsub(/\;/,Z) 1'

Output will be as follows.

Code:
1390215660,1110
1390215661,1113
1390215662,1109
1390215663,1113
1390215664,1114
1390215665,1120
1390215666,1120
1390215667,1118
1390215668,1121
1390215669,1120
1390215670,1117
1390215671,1122
1390215672,1114
1390215673,1105
1390215674,1111
1390215675,1106
1390215676,1111
1390215677,1118
1390215678,1117
1390215679,1116
1390215680,1108
1390215681,1106
1390215682,1112
1390215683,1106
1390215684,1114
1390215685,1112
1390215686,1109
1390215687,1106
1390215688,1109
1390215689,1111
1390215690,1105
1390215691,1114
1390215692,1111
1390215693,1108
1390215694,1113
1390215695,1106
1390215696,1102
1390215697,1117
1390215698,1107
1390215699,1104
1390215700,1108
1390215701,1109
1390215702,1111
1390215703,1112
1390215704,1110
1390215705,1105
1390215706,1109
1390215707,1109
1390215708,1107
1390215709,1114
1390215710,1110
1390215711,1107
1390215712,1110
1390215713,1121
1390215714,1111
1390215715,1107
13902*
15716,1113
1390215717,1111
1390215718,1104
1390215719,1111


Hope this may help.


Thanks,
R. Singh
# 3  
Old 01-20-2014
Quote:
Originally Posted by brononius
Hey,

I'm using some sensors that can be read by http.
If I run following command:
...
I would like to put this now in a sheet with only the 'unique' values of date and value.
So I would like to have something as:
....
Code:
$ cat file
* About to connect() to 192.168.111.23 port 8080 (#0)
* Trying 192.168.111.23... connected
* Connected to 192.168.111.23 (192.168.111.23) port 8080 (#0)
> GET /sensor/52f5c63cc4221fbbedcb499908a0a823?version=1.0&interval=minute&unit=watt&callback=realtime HTTP/1.1
> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 192.168.111.23:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/json
<
realtime([[1390215660,1110],[1390215661,1113],[1390215662,1109],[1390215663,1113],[1390215664,1114],[1390215665,1120],[1390215666,1120],[1390215667,1118],[1390215668,1121],[1390215669,1120],[1390215670,1117],[1390215671,1122],[1390215672,1114],[1390215673,1105],[1390215674,1111],[1390215675,1106],[1390215676,1111],[1390215677,1118],[1390215678,1117],[1390215679,1116],[1390215680,1108],[1390215681,1106],[1390215682,1112],[1390215683,1106],[1390215684,1114],[1390215685,1112],[1390215686,1109],[1390215687,1106],[1390215688,1109],[1390215689,1111],[1390215690,1105],[1390215691,1114],[1390215692,1111],[1390215693,1108],[1390215694,1113],[1390215695,1106],[1390215696,1102],[1390215697,1117],[1390215698,1107],[1390215699,1104],[1390215700,1108],[1390215701,1109],[1390215702,1111],[1390215703,1112],[1390215704,1110],[1390215705,1105],[1390215706,1109],[1390215707,1109],[1390215708,1107],[1390215709,1114],[1390215710,1110],[1390215711,1107],[1390215712,1110],[1390215713,1121],[1390215714,1111],[1390215715,1107],[13902* Closing connection #0
15716,1113],[1390215717,1111],[1390215718,1104],[1390215719,1111]]);

This will just print and does not check unique date
Code:
$ awk '/^realtime\(/,/\)/{gsub(/[\[\]\(\);]|[[:alpha:]]/,x);for(i=1;i<=NF;i+=2)print $i,$(i+1)}' FS=, file

following one checks unique date
Code:
$ awk '/^realtime\(/,/\)/{gsub(/[\[\]\(\);]|[[:alpha:]]/,x);for(i=1;i<=NF;i+=2)if(!A[$i+0]++)print $i,$(i+1)}' FS=,  file

Code:
1390215660 1110
1390215661 1113
1390215662 1109
1390215663 1113
1390215664 1114
1390215665 1120
1390215666 1120
1390215667 1118
1390215668 1121
1390215669 1120
1390215670 1117
....
....

# 4  
Old 01-20-2014
Quote:
Originally Posted by Akshay Hegde
following one checks unique date
Code:
$ awk '/^realtime\(/,/\)/{gsub(/[\[\]\(\);]|[[:alpha:]]/,x);for(i=1;i<=NF;i+=2)if(!A[$i+0]++)print $i,$(i+1)}' FS=,  file

This one does it all I believe. Smilie

I'm thinking now to combine both (the awk and the curl command) and put the result in a file with >>
But since the unique filter is done before the results are written into a file, it's won't give me the correct results. Since unique dates can be written into the file. Do I sound clear? Smilie

Code:
curl -v 'http://192.168.111.23:8080/sensor/52f5c63cc4221fbbedcb499908a0a823?version=1.0&interval=minute&unit=watt&callback=realtime'

# 5  
Old 01-20-2014
Quote:
Originally Posted by brononius
This one does it all I believe. Smilie

I'm thinking now to combine both (the awk and the curl command) and put the result in a file with >>
But since the unique filter is done before the results are written into a file, it's won't give me the correct results. Since unique dates can be written into the file. Do I sound clear? Smilie

Code:
curl -v 'http://192.168.111.23:8080/sensor/52f5c63cc4221fbbedcb499908a0a823?version=1.0&interval=minute&unit=watt&callback=realtime'

Try :

Code:
$ curl -v  'http://192.168.111.23:8080/sensor/52f5c63cc4221fbbedcb499908a0a823?version=1.0&interval=minute&unit=watt&callback=realtime' | \
awk '/^realtime\(/,/\)/{gsub(/[\[\]\(\);]|[[:alpha:]]/,x);for(i=1;i<=NF;i+=2)if(!A[$i+0]++)print $i,$(i+1)}' FS=, OFS=, >uniq_output_file.csv

OR

Code:
$ curl -v   'http://192.168.111.23:8080/sensor/52f5c63cc4221fbbedcb499908a0a823?version=1.0&interval=minute&unit=watt&callback=realtime' >file.tmp

$ awk '/^realtime\(/,/\)/{gsub(/[\[\]\(\);]|[[:alpha:]]/,x);for(i=1;i<=NF;i+=2)if(!A[$i+0]++)print $i,$(i+1)}' FS=, OFS=, file.tmp >uniq_output_file.csv

# 6  
Old 01-20-2014
First one seems to do the trick perfectly.

But the csv file is each time overwritten, correct?
Should be nice if only the new (=non existing values) are added to the csv.


I've just noticed that you can only read the values of the last minute (with second interval).
So I should run this script each minute.

Not sure how i can handle it with cacti to graph it, but I suppose that I'll will clear (echo n > ...?)the file once it's imported.

---------- Post updated at 03:36 PM ---------- Previous update was at 02:23 PM ----------

Okay, what i've done for now:

instead of >uniq_output_file.csv, i've used >>uniq_output_file.csv.
Code:
curl -v  'http://192.168.111.23:8080/sensor/52f5c63cc4221fbbedcb499908a0a823?version=1.0&interval=minute&unit=watt&callback=realtime' | \
awk '/^realtime\(/,/\)/{gsub(/[\[\]\(\);]|[[:alpha:]]/,x);for(i=1;i<=NF;i+=2)if(!A[$i+0]++)print $i,$(i+1)}' FS=, OFS=, >>uniq_output_file.csv

And after this, i run
Code:
sort -u uniq_output_file.csv -o uniq_output_file.csv


Not sure if you guys like this? Else let me know if there's a better way...
Last edited by brononius; 01-20-2014 at 10:17 AM..
# 7  
Old 01-20-2014
# Redirect output to file "filename" overwrite if file is existing
command >filename

# Redirect and append output to file "filename"
command >>filename


Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Filter output in curl

Hello guys, I'm writing a little script which sends me sms with my shell script via api of a sms provider. problem is I can't filter my curl output for this site: site url:... (1 Reply)
Discussion started by: genius90
1 Replies

2. Shell Programming and Scripting

Use of awk to filter out the command output

Hi All, I am trying to find out number of cores present for hp-ux server from the output of print_manifest (as shown below). i suppose awk will be best tool to use for filtering. output of print_manifest is : System Hardware Model: ia64 hp Integrity Virtual Partition ... (6 Replies)
Discussion started by: omkar.jadhav
6 Replies

3. UNIX for Advanced & Expert Users

Filter top -bc -n10 output

I have posted a thread in Unix for Dummies question...... https://www.unix.com/unix-dummies-questions-answers/223769-filter-top-bc-n10-output.html Can anybody help me out with this please? Thanks Avinash (3 Replies)
Discussion started by: avinthm
3 Replies

4. UNIX for Dummies Questions & Answers

Filter top -bc -n10 output

Hi "top -bc -n10" gives me the output of the "top" command 10 times. Now i dont want repeated results and hence want to return only the last output. And I would like to do it in one command and not in script. can you please help me out? Thanks (2 Replies)
Discussion started by: avinthm
2 Replies

5. Shell Programming and Scripting

Filter output as an array element

I am filtering the value of Server status from a file and am storing it in a temp file which I compare later to exit with appropriate status. I am wondering if I can directly output the value of Server status as an array element and then compare the value of elements to get the right exit status ... (2 Replies)
Discussion started by: paslas
2 Replies

6. Shell Programming and Scripting

Filter out output and then write to a file

I have a process which writes lots of output to the screen and am capturing it to a file. But the issue is that there are few lines which repeat every second which is causing the file size to be enormous. I am trying to filter out and then run the command but its not working. Can you confirm... (7 Replies)
Discussion started by: rajkumarme_1
7 Replies

7. Shell Programming and Scripting

How to filter only the last 'n' lines of a grep output?

I am running a grep query for searching a pattern, and the output is quite huge. I want only the last 200 lines to be displayed, and I am not sure if tail will do the trick (can tail read from std in/out instead of files?). Please help me out. (1 Reply)
Discussion started by: shell_newbie
1 Replies

8. Shell Programming and Scripting

Shell Script to Filter Output

Hi guys - I use this script below to search for files that are of todays date. It also informs if the file doesnt exist or is old. I am trying to make the script inform of the errors in the "begining" of the output email. Such as all errors are grouped together. I am using the following... (1 Reply)
Discussion started by: DallasT
1 Replies

9. IP Networking

Filter wireshark output

Hi I have a wireshark file saved (from my network) and I have to analyze the flows inside it. The problem is that i have to analyze not the complete file (60.000 pkts!) but just a subset of it. In other words i have to sample the wireshark.file.dump and for example from 60.000 pkts take... (2 Replies)
Discussion started by: Dedalus
2 Replies

10. UNIX for Dummies Questions & Answers

Read from a file then filter the output

Hi i need a script which reads a certain file and then display it on the screen. The thing is i need to filter the display output. this is an example of the file which is to be loaded in the script. asdfg1.1.1|98 hjkldfe4.0.3|123 asdzxzvdweradfsdafascv10.0.10|123456789... (1 Reply)
Discussion started by: khestoi
1 Replies
Login or Register to Ask a Question