Shell Programming and Scripting

Please I need to rearrange data acquired by USB port from a sensor network. The information is mixed and I need to convert it into a kind of table. This is my input file:

Node 4D5A joined
Temperature: 27,5
Humidity: 40
Dew Point: 23
No motion detected
LUX: 389
Temperature: 28
Humidity: 41
Dew Point: 21
Node 58DF joined
Temperature: 28,5
Humidity: 41
Dew Point: 24
LUX: 384
Node 4D5A joined
Temperature: 21,5
Humidity: 43
Dew Point: 20
No motion detected

Realize that every time a node send info, first is sent its ID and later different data without any specific pattern. My idea is to identify every line between Node and Node and arrange them into other file. Any pattern is posible... but I don't have any idea on how to implement it. I have been trying with awk, sed, grep... but with no result.

A possible solution could be like this:

Node:data:value
Node:data:value

or

Node
Data:value
Node
Data:value

please give some ideas or solutions

Thanks a lot!

You may need to change it a bit if you want to remove blanks for values or filter out unwanted lines (maybe you don't need the message "No motion detected", I don't know):

Code:

awk '
   BEGIN { OFS=":" }
   {
      if ($1 == "Node") {
         n=$2
      } else {
         split($0, x, ":");
         print n, x[1], x[2];
      }
   }
' input_file.txt

Great! Now I have easier line identifier, but actually my final target is to get something like this:

4D5A:Temperature: 27,5 / 28 / 21,5
4D5A:Humidity: 40 / 41 / 43
4D5Aew Point: 23 / 21 / 20
4D5A:Motion: 0 / 0
4D5A:LUX: 389
58DF:Temperature: 28,5
58DF:Humidity: 41
58DFew Point: 24
58DF:Motion: 1
58DF:LUX: 384

Is it posible by awk as well?

Thanks again

This should work. By the way, I didn't understand the sorting "algorithm" you've used for your expected output... Is it a random sort?

Code:

awk '
   BEGIN { OFS=":" }
   {
      if ($1 == "Node") {
         n=$2
      } else {
         split($0, x, ":");
         key=n ":" x[1];
         val=x[2]?x[2]:" 0";
         y[key]=y[key] (y[key]?" /":"") val;
      }
   }
   END {
      for (node_data in y) print(node_data, y[node_data]);
   }
' input_file.txt | sort

Ahhh... you mean fot Temperature and everything, hehe... sorting alphabetically is right. Thank you very much.

Can you explain me every step of the program?

Basically, we first need to know which node every information in the input file is related to and then, given the node name and the measurement name, populate an associative array (hash) which keys are the concatenation between "node_name:measurement_name". For each element, the associated value will be the union of all the consecutive values found, separated by "/".

We once tell awk that the Output Field Separator for print statements will be a colon:

Code:

   BEGIN { OFS=":" }

For each row of the file, we execute the code between curly braces. Here, when we notice that the first space-separated field of the file corresponds to the word "Node", we assign to the "n" variable the node name (=2nd field, $2):

Code:

   {
      if ($1 == "Node") {
         n=$2

If that's not the case, we elaborate the measurement information. By the means of split function, we create an array called "x" which will contain the two colon-separated fields: x[1] is the measurement name (eg. "Temperature"), x[2] is the measurement value (eg. " 27,5"):

Code:

      } else {
         split($0, x, ":");

Now we compose a key for our hash, concatenating "n" (eg. "4D5A"), a colon ":" and the measurement name.
For example, in the second loop, our key will be "4D5A:Temperature", in the 3rd "4D5A:Humidity" and so on.

Code:

         key=n ":" x[1];

The corresponding value for the key just found will be the measurement value, x[2]. Here I've used a ternary operator because you want to display "0" even if the measurement name hasn't an associated value, for example in the case of "No motion detected": if x[2] is not null, val=x[2], else val=" 0".

Code:

         val=x[2]?x[2]:" 0";

Now that we have defined the key and the value, we populate an associative array called "y".
There's another ternary operator: if an element associated with the key is already present, the value will be the actual non-null value (y[key]) followed by a slash " /" and the measurement value (val); if it is the first time we insert a value for the key "key", we don't put the slash.

Code:

         y[key]=y[key] (y[key]?" /":"") val;
      }
   }

Finally, when all the hash is populated with the values got from every record of the input file, we simply print all the key/value pairs. The print output is OFS delimited.

Code:

   END {
      for (node_data in y) print(node_data, y[node_data]);
   }

The awk result is piped through the "sort" command to beautify and make readable the output

Hope it's more clear now