Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Grabbing data between 2 points in text file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Grabbing data between 2 points in text file
# 1  
Old 08-03-2013
Linux Grabbing data between 2 points in text file
I have a text file that shows the output of my solar inverters. I want to separate this into sections. overview , device 1 , device 2 , device 3. Each device has different number of lines. but they all have unique starting points. Overview starts with 6 #'s, Devices have 4#'s and their data starts with 2 #'s. I know the answer is in front of me. How to separate them so I can import them to a box at home that is running appache.


Code:
/var/www/tmp $ cat sunnywebbox-summary.txt
###### Overview:
         GriPwr (         GriPwr): 192 W
      GriEgyTdy (      GriEgyTdy): 17.55920136064 kWh
      GriEgyTot (      GriEgyTot): 27014.58717195136 kWh
          OpStt (          OpStt):
            Msg (            Msg):

  #### Device WR18UW8E:1573704651 (WR18UW8E:1573704651):

    ## Process data:
                Error (          Error):
              E-Total (        E-Total):  kWh
                  Fac (            Fac):  Hz
              h-Total (        h-Total):  h
                  Iac (            Iac):  mA
                  Ipv (            Ipv):  mA
                 Mode (           Mode):
                  Pac (            Pac):  W
             Power On (       Power On):
        Serial Number (  Serial Number):
          Temperature (    Temperature):  grdC
                  Vac (            Vac):  V
                  Vpv (            Vpv):  V

  #### Device WR25UW8E:1374125280 (WR25UW8E:1374125280):

    ## Process data:
                Error (          Error): -------
              E-Total (        E-Total): 20193.09417195136 kWh
                  Fac (            Fac): 60.000 Hz
              h-Total (        h-Total): 24283.4258315628 h
                  Iac (            Iac): 218 mA
                  Ipv (            Ipv): 363 mA
                 Mode (           Mode): Mpp-Search
                  Pac (            Pac): 53 W
             Power On (       Power On): 3179
        Serial Number (  Serial Number): 1374125280
          Temperature (    Temperature): 43.4 grdC
                  Vac (            Vac): 245 V
                  Vpv (            Vpv): 333 V

  #### Device WRHU0U5A:2120048843 (WRHU0U5A:2120048843):

    ## Process data:
             A.Ms.Amp (       A.Ms.Amp): 0.414 A
             A.Ms.Vol (       A.Ms.Vol): 359.200 V
            A.Ms.Watt (      A.Ms.Watt): 148 W
                Error (          Error): -------
              E-Total (        E-Total): 6821.493 kWh
            GridMs.Hz (      GridMs.Hz): 60.010 Hz
      GridMs.PhV.phsA (GridMs.PhV.phsA): 247.890 V
      GridMs.PhV.phsB (GridMs.PhV.phsB): 0.000 V
      GridMs.PhV.phsC (GridMs.PhV.phsC): 0.000 V
        Inv.TmpLimStt (  Inv.TmpLimStt): NoneDrt
            MainModel (      MainModel): Solar-WR
                 Mode (           Mode): MPP
          Mt.TotOpTmh (    Mt.TotOpTmh): 5546.575025102 h
            Mt.TotTmh (      Mt.TotTmh): 5806.537352761 h
         Op.EvtCntUsr (   Op.EvtCntUsr): 54
             Op.EvtNo (       Op.EvtNo): 0
          Op.GriSwStt (    Op.GriSwStt): Cls
            Op.TmsRmg (      Op.TmsRmg): 0.000 s
                  Pac (            Pac): 139 W
        Serial Number (  Serial Number): 2120048843

>>>>> UPDATE <<<<<<
I have been working with SED all evening.. I just can't grab one data set from the file without grabbing a part of another. see below what I have been doing.

Code:
sed -n '/Overview/,/\           /p' sunnywebbox-summary.txt

sed -n '/#### Device WR25/,/\                     /p' sunnywebbox-summary.txt

 sed -n '/#### Device WRH/,/\                     /p' sunnywebbox-summary.txt

Code:
/var/www/tmp $ sed -n '/#### Device WR25/,/\####/p' sunnywebbox-summary.txt


  #### Device WR25UW8E:1374125280 (WR25UW8E:1374125280):

    ## Process data:
                Error (          Error): -------
              E-Total (        E-Total): 20193.09417195136 kWh
                  Fac (            Fac): 60.000 Hz
              h-Total (        h-Total): 24283.4258315628 h
                  Iac (            Iac): 218 mA
                  Ipv (            Ipv): 363 mA
                 Mode (           Mode): Mpp-Search
                  Pac (            Pac): 53 W
             Power On (       Power On): 3179
        Serial Number (  Serial Number): 1374125280
          Temperature (    Temperature): 43.4 grdC
                  Vac (            Vac): 245 V
                  Vpv (            Vpv): 333 V

  #### Device WRHU0U5A:2120048843 (WRHU0U5A:2120048843):

Moderator's Comments:
Mod Comment It looks like you've been posting for years without using CODE tags. In many of your posts, it didn't make a lot of difference, but please get into the habit of using CODE tags when you show sample code, input, and output. Leaving out the tags lets the HTML processing coalesce multiple spaces into single spaces which can significantly alter code that is needed to process input files, make code hard to read, and cause us to make suggestions that will produce output that is not formatted the way you want it. Please review the PM you should have received by now for simple directions showing how to use CODE tags.
Last edited by Don Cragun; 08-04-2013 at 01:28 AM.. Reason: Add CODE tags.
# 2  
Old 08-04-2013
try awk instead using the device names as reference points ...
Code:
awk "/WR25/,/WRHU/" sunnywebbox-summary.txt

# 3  
Old 08-04-2013
Grabbing data between 2 points in text file
Quote:
Originally Posted by Just Ice
try awk instead using the device names as reference points ...
Code:
awk "/WR25/,/WRHU/" sunnywebbox-summary.txt


gives me what I have been looking at all evening

#### Device WR25UW8E:1374125280 (WR25UW8E:1374125280):

## Process data:
Error ( Error):
E-Total ( E-Total): kWh
Fac ( Fac): Hz
h-Total ( h-Total): h
Iac ( Iac): mA
Ipv ( Ipv): mA
Mode ( Mode):
Pac ( Pac): W
Power On ( Power On):
Serial Number ( Serial Number):
Temperature ( Temperature): grdC
Vac ( Vac): V
Vpv ( Vpv): V

#### Device WRHU0U5A:2120048843 (WRHU0U5A:2120048843):
# 4  
Old 08-04-2013
the problem here is that your input does not have easily delimited paragraphs where sed or awk can quickly work ... it would have been quiet easy to awk "/WR25/,/^$/" sunnywebbox-summary.txt but the empty line immediately after the Device line kills that ... doing the original awk command line i suggested would work if you could just remove the second Device line like in awk "/WR25/,/WRHU/" sunnywebbox-summary.txt | grep -v WRHU but that would only work if you know which device follows in the report and the listed device is not the last in the list ...

anyways, the script below should work with or without an argument ... it really only uses sed to get the desired output but needed all the other lines to determine the sed reference addresses ... somebody here could probably script this in perl or awk better but at least you have something to start with ...
Code:
#! /bin/ksh
PATH=/usr/bin:/bin:/usr/local/bin:/usr/sbin:/sbin
device=$1
testfile=testfile1

grep -n -i device $testfile > /tmp/$$
if [ ! $device ]
then
    print "\n--- Devices ---"
    awk '{print $4}' /tmp/$$ | awk -F: '{print $1}'
    print "\nWhich device to report? \c"
    read device
fi

linecnt1=$(wc -l < $testfile)
linecnt2=$(wc -l < /tmp/$$)
startcnt=$(grep -n $device /tmp/$$ | awk -F":" '{print $1}')
if [ $startcnt -eq linecnt2 ]
then
    endline=$linecnt1
else
    nextcnt=$(expr $startcnt + 1)
    tempcnt=$(sed -n "${nextcnt}p" /tmp/$$ | awk -F":" '{print $1}')
    endcnt=$(expr $tempcnt - 1)
    endline=$endcnt
fi
startline=$(grep -n $device $testfile | awk -F":" '{print $1}')

echo
sed -n "${startline},${endline}p" $testfile

rm -f /tmp/$$ 2> /dev/null

exit 0

# 5  
Old 08-04-2013
This one exits before it prints
Code:
awk '($1=="####" && p) {p=0} $0~s {p=1} p' s=WR25 sunnywebbox-summary.txt

Last edited by MadeInGermany; 08-05-2013 at 08:05 AM.. Reason: p=0 rather than exit; permits multi-match e.g. s=UW8E
This User Gave Thanks to MadeInGermany For This Post:
Just Ice
# 6  
Old 08-05-2013
Thank everyone, I devised my own fix. I converted the txt file to xml and that made extraction much easier.

Mike
# 7  
Old 08-05-2013
You could try this to separate the overview and the devices into their own .txt files:
Code:
awk '/^..#### / {fn = substr ($NF, 1, 9)".txt"; gsub (/[)(:]/, "", fn)} {print >fn}'

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grabbing text between two lines with shell variables.

I would like to grab complex html text between lines using variables. I am running Debian and using mksh shell. Here is the part of the html that I want to extract from. I would like to extract the words 'to love,' and I would like to use the above and below lines as reference points. ... (3 Replies)
Discussion started by: bedtime
3 Replies

2. Shell Programming and Scripting

Reducing the decimal points of numbers (3d coordinates) in a file; how to input data to e.g. Python

I have a file full of coordinates of the form: 37.68899917602539 58.07500076293945 57.79100036621094 The numbers don't always have the same number of decimal points. I need to reduce the decimal points of all the numbers (there are 128 rows of 3 numbers) to 2. I have tried to do this... (2 Replies)
Discussion started by: crunchgargoyle
2 Replies

3. Shell Programming and Scripting

Grabbing data from a secured website

Hello there, I am a beginner in Perl, and I have a challenging project: I have to create a program that checks regularly on an online bank account for new operations, it should then feed a database keeping track of all the money going in and out. Of course the login details of this online... (4 Replies)
Discussion started by: freddie50
4 Replies

4. Shell Programming and Scripting

Grabbing a chunk of text from a file

Hi, I have a Report.txt file. Say the contents of this file are : 1 2 3 4 5 7 df v g gf e r dfkf lsdk dslsdklsdk Report Start: xxxxxxdad asdffsdfsdfsdfasfasdffasdf sadfasdfsadffsfsdf Report End. sdfasdfasdf sdfasfdasdfasdfasdfasdf sadfasdfsdf I need to grab from Report Start... (3 Replies)
Discussion started by: mrskittles99
3 Replies

5. Shell Programming and Scripting

Grabbing top line of text in a file and setting it to a variable

If I have a txt file with test.txt somelineoftext and I want to set that line of text to variable in a script: so #!/bin/bash var='' becomes #!/bin/bash var='somelineoftext' (3 Replies)
Discussion started by: digitalviking
3 Replies

6. Shell Programming and Scripting

Grabbing text and using that text in a newly created line

Hello, I am really stuck and I'm hoping somone can help. I have a text file that is similar to this: <--First User--> <function>account='uid=user1,....... <--Second User--> <function>account='uid=user2,.......What I want is to grab the usernames after "uid=" and before the following... (9 Replies)
Discussion started by: mafia910
9 Replies

7. Shell Programming and Scripting

Calculate difference between consecutive data points in a column from a file

Hi, I have a file with one column data (sample below) and I am trying to write a shell script to calculate the difference between consecutive data valuse i.e Var = Ni -N(i-1) 0.3141 -3.6595 0.9171 5.2001 3.5331 3.7022 -6.1087 -5.1039 -9.8144 1.6516 -2.725 3.982 7.769 8.88 (5 Replies)
Discussion started by: malandisa
5 Replies

8. UNIX for Dummies Questions & Answers

Help grabbing data of a link

Hi folks; I have a temperature sensor that configured with an IP address so we can open a browser to see the temperatures in our 3 labs all showing in one screen. I'm trying to write a script to call this link then email me the results/output so i don't have to open a browser manually every... (3 Replies)
Discussion started by: Katkota
3 Replies

9. UNIX for Dummies Questions & Answers

How to get data only inside polygon created by points which is part of whole data from file?

hiii, Help me out..i have a huge set of data stored in a file.This file has has 2 columns which is latitude & longitude of a region. Now i have a program which asks for the number of points & based on this number it asks the user to enter that latitude & longitude values which are in the same... (7 Replies)
Discussion started by: reva
7 Replies

10. Shell Programming and Scripting

grabbing filename from text file....should be easy!

Quick question...I'm trying to grab the .tif file name from this output from our fax server. What is the best way i can do this in a bash script? I have been looking at regular expressions with bash or using awk but having some trouble. thanks! The only output i want is... (5 Replies)
Discussion started by: kuliksco
5 Replies
Login or Register to Ask a Question