Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Merge lines together in unix

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Merge lines together in unix
# 1  
Old 08-25-2010
Merge lines together in unix
I have a file like this. Pls help me to solve this in ksh
(I should look for only Message : 111 and need to print the start time to end time
Need to ignore other type of messages. Ex: if first message is 111 and second message is 000 or anything else then ignore the 2nd one and print start time of the first message. )

Example:
Code:
08-24-10 10:22:34,xxxMessage : 111
08-24-10 10:24:38,xxbMessage : 111
08-24-10 10:24:39,xxcMessage : 111
08-24-10 10:24:40,xxdMessage : 000 
08-24-10 11:28:11,xxeMessage : 111
08-24-10 11:32:35,xxfMessage : 111

The output should be
Code:
08-24-10 10:22:34 to 08-24-10 10:24:39
08-24-10 11:28:11 to 08-24-10 11:32:35



Moderator's Comments:
Mod Comment Please use code tags, thank you!
Last edited by Franklin52; 08-25-2010 at 12:48 PM..
# 2  
Old 08-25-2010
Code:
# cat infile
08-24-10 10:22:34,xxxMessage : 111
08-24-10 10:24:38,xxbMessage : 111
08-24-10 10:24:39,xxcMessage : 111
08-24-10 10:24:40,xxcMessage : 000
08-24-10 11:28:11,xxeMessage : 111
08-24-10 11:32:35,xxfMessage : 111

Code:
# ./justdoit
08-24-10 10:22:34 to 08-24-10 10:24:39
08-24-10 11:28:11 to 08-24-10 11:32:35

Code:
## justdoit ##
#!/bin/bash
notdel=111
rm -f tmpX 2>/dev/null
while read -r l
  do
    if [[ $(echo "$l" | grep $notdel) ]] ; then
     echo "$l" >> tmpX
    else
     cnt=$(cat tmpX | wc -l)
     y="";   x="N;"
      while [ $(( cnt -= 1 )) -gt 1 ]
       do
        y="$y$x";
       done
       sed -e '{/111/N;}' -e "{/111/!d;$y;bx}"  -e :x -e '{s/^\([^,]*\).*\n\([^,]*\).*$/\1 to \2/}'  tmpX
       rm -f tmpX
    fi
  done<infile
cnt=$(cat tmpX | wc -l)
y="";x="N;"
   while [ $(( cnt -= 1 )) -gt 1 ]
    do
     y="$y$x";
    done
      sed -e '{/111/N;}' -e '{s/^\([^,]*\).*\n\([^,]*\).*$/\1 to \2/}'  tmpX

Last edited by ygemici; 08-26-2010 at 02:33 AM..
This User Gave Thanks to ygemici For This Post:
mnjx
# 3  
Old 08-25-2010
I think this is a bit easier to understand:

Code:
#!/usr/bin/env ksh
while read line
do
        if [[ $line == *" : 111" ]]              # line with your flag
        then
                if [[ -z $start_time ]]          # dont have start; save start
                then
                        start_time="${line%%,*}"   # ditch all from first comma to end of line
                else                                   # have start, just update end
                        end_time="${line%%,*}"
                fi
        else                                   # not our line, print out last start/end pair
                if [[ -n $end_time ]]             # but only if we have a pair and not printed
                then
                        printf "%s  to  %s\n" "$start_time" "$end_time"    # print times
                        start_time=""                # next time we see is a start time
                        end_time=""
                fi
        fi
done <input-file
if [[ -n $end_time ]]             # handle last case, but only if both seen
then
        printf "%s  to  %s\n" "$start_time" "$end_time"    # print times
fi

Last edited by agama; 08-25-2010 at 08:42 PM..
This User Gave Thanks to agama For This Post:
mnjx
# 4  
Old 08-26-2010
agama,

Thanks a lot !
I am having some problem with the above code. The above code works only for few cases. Please see the below example.
Code:
2010-08-26 11:22:34,611 INFO   [Thread-xxxx]- [(xxxx,2723)Service Message : 111
2010-08-26 11:28:11,573 INFO   [Thread-abcd]- [(xxxxx,2723)Service Message : 000
2010-08-26 11:32:35,187 INFO   [Thread-zs]- [(xxxxx,2581)Service Message :   |LS
2010-08-26 11:33:35,189 INFO   [Thread-zs]- [(xxxxx,2581)Service Message : 111
2010-08-26 11:33:35,189 INFO   [Thread-zs]- [(xxxxx,2581)Service Message : 111
2010-08-26 11:34:36,189 INFO   [Thread-zs]- [(xxxxx,2581)Service Message : 111
2010-08-26 11:35:37,573 INFO   [Thread-abcd]- [(xxxxx,2724)Service Message : |LS
2010-08-26 11:36:38,189 INFO   [Thread-zs]- [(xxxxx,2585)Service Message : 111

Output should be :
Code:
2010-08-26 11:22:34
2010-08-26 11:33:35 to 2010-08-26 11:34:36
2010-08-26 11:36:38

Last edited by Scott; 08-27-2010 at 03:06 PM.. Reason: Code tags, please...
# 5  
Old 08-26-2010
MySQL
Code:
# cat infile
2010-08-26 11:22:34,611 INFO [Thread-xxxx]- [(xxxx,2723)Service Message : 111
2010-08-26 11:28:11,573 INFO [Thread-abcd]- [(xxxxx,2723)Service Message : 000
2010-08-26 11:32:35,187 INFO [Thread-zs]- [(xxxxx,2581)Service Message : |LS
2010-08-26 11:33:35,189 INFO [Thread-zs]- [(xxxxx,2581)Service Message : 111
2010-08-26 11:33:35,189 INFO [Thread-zs]- [(xxxxx,2581)Service Message : 111
2010-08-26 11:34:36,189 INFO [Thread-zs]- [(xxxxx,2581)Service Message : 111
2010-08-26 11:35:37,573 INFO [Thread-abcd]- [(xxxxx,2724)Service Message : |LS
2010-08-26 11:36:38,189 INFO [Thread-zs]- [(xxxxx,2585)Service Message : 111

Code:
# ./justdoit
2010-08-26 11:22:34
2010-08-26 11:33:35 to 2010-08-26 11:34:36
2010-08-26 11:36:38

Code:
 
## justdoit ##
#!/bin/bash
notdel=111
rm -f tmpX 2>/dev/null
while read -r l
  do
    if [[ $(echo "$l" | grep $notdel) ]] ; then
     echo "$l" >> tmpX
    else
     cnt=$(cat tmpX 2>/dev/null | wc -l )
     y="";   x="N;"
      while [ $(( cnt -= 1 )) -gt 1 ]
       do
        y="$y$x";
       done
       sed -e '{/111/N;}' -e "{/111/!d;$y;bx}"  -e :x -e '{s/^\([^,]*\).*\n\([^,]*\).*$/\1 to \2/}' tmpX 2>/dev/null | sed 's/,.*//'
       rm -f tmpX
    fi
  done<infile
cnt=$(cat tmpX 2>/dev/null | wc -l )
y="";x="N;"
   while [ $(( cnt -= 1 )) -gt 1 ]
    do
     y="$y$x";
    done
      sed -e '{/111/N;}' -e '{s/^\([^,]*\).*\n\([^,]*\).*$/\1 to \2/}'  tmpX 2>/dev/null | sed 's/,.*//'

This User Gave Thanks to ygemici For This Post:
mnjx
# 6  
Old 08-26-2010
Quote:
Originally Posted by mnjx
agama,

Thanks a lot !
I am having some problem with the above code. The above code works only for few cases.
Yes, the code sample I gave assumed that there was always a start/end pair. You didn't mention how to handle the case that you've pointed out.

Small change below handles this case now.

Code:
#!/usr/bin/env ksh
while read line
do
        if [[ $line == *" : 111" ]]              # line with your flag
        then
                if [[ -z $start_time ]]          # dont have start; save start
                then
                        start_time="${line%%,*}"   # ditch all from first comma to end of line
                else                                   # have start, just update end
                        end_time="${line%%,*}"
                fi
        else                                   # not our line, print out last start/end pair
                if [[ -n $end_time ]]             # print pair if we have both
                then
                        printf "%s  to  %s\n" "$start_time" "$end_time"    # print times
                else
                      if [[ -n $start_time ]]
                      then
                        printf "%s\n" "$start_time"
                      fi
                fi
                start_time=""                # next time we see is a start time
                end_time=""
        fi
done <input-file
if [[ -n $end_time ]]             # handle last case, but only if both seen
then
        printf "%s  to  %s\n" "$start_time" "$end_time"    # print times
else
        if [[ -n $start_time ]]
        then
           printf "%s\n" "$start_time"
        end
fi

This User Gave Thanks to agama For This Post:
mnjx
# 7  
Old 08-26-2010
Code:
awk 'BEGIN{c=0;start="";end=""}
{
   if ($NF=="111")
      { split($2,a,",")
        if (c==0) start=$1 FS a[1]
        else end=$1 FS a[1]
        c++;next;
      }
   else {
          if (start=="") next
          if (end=="") print start
          else print start, "to", end
          c=0;start="";end="";next
       }
}
END { if (c>0)           
          {if (end=="") print start
          else print start, "to", end}
    }
' infile

This User Gave Thanks to rdcwayx For This Post:
mnjx
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merge Lines

Hello I have an input as below this is test we are( ) one end of description I am looking for output this is test we are () one end of description (2 Replies)
Discussion started by: Tomlight
2 Replies

2. Shell Programming and Scripting

Merge lines

Hello I have a file with CAR 23 COLOR 12 CAR 44 COLOR 12 CAR 55 COLOR 20 SIZE BIG CAR 56 CAR 57 COLOR 11 How can merge the CAR and the COLOR + SIZE (if there are COLOR,SIZE) CAR 23 COLOR 12 CAR 44 COLOR 12 CAR 55 COLOR 20 SIZE BIG CAR 56 CAR 57 COLOR 11 Every line begin in... (4 Replies)
Discussion started by: sharong
4 Replies

3. Shell Programming and Scripting

Merge two non-consecutive lines

Hello - First post here. I need help combining two lines that are non-consecutive in a file. Using sed, awk or perl preferably. So the file looks as follows. Please note, the "Line#:" is there only for reference. The lines can only be distinguished by whether they have "start" or "done" in... (2 Replies)
Discussion started by: munkee
2 Replies

4. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

5. UNIX for Dummies Questions & Answers

To merge a few lines to 1 line

Hi Experts, This is my input file. input.txt 0 /dev/fd 25 /var 1 /tmp 1 /var/run 1. If this file has single line, then leave it, print the single line else merge the 4 lines above into 1 line as below e.g (6 Replies)
Discussion started by: streddy
6 Replies

6. Shell Programming and Scripting

Merge two lines using sed

Hi, I am trying to merge two lines, first line starts with a particular pattern and second line ends with a particular pattern in a file. Something like: First line starts with say ABC Second line ends with say XYZ After a merge, the line should become ABC.......XYZ I tried... (14 Replies)
Discussion started by: Sunny Arora
14 Replies

7. Shell Programming and Scripting

merge lines

Hi guys in input every 1st line 1st ID value located in 2nd line 1st occurrence . I need to print them down accordingly.. Thanx in advance input rs1040480_XXXXX.value rs1040481_YYYYY.value rs1040482_TXXXX.value 0.7408157 0.3410044 0.7408157 ... (7 Replies)
Discussion started by: stateperl
7 Replies

8. Shell Programming and Scripting

Merge 2 lines in file

Hi All, I have a data in flat file like below. Some of the information are in second row. 111_ABCProcess ----- ----- IN 0/0 111_PQRTrimPRocess ----- ----- OI 0/0 111_ZigZagTrimProcess ----- ----- ... (1 Reply)
Discussion started by: Amit.Sagpariya
1 Replies

9. Shell Programming and Scripting

Merge two lines

Hi I have two lines of data formatted as displayed below shop, price, remark, date a,#N/A,order to -fd, 20091011 and would like it to be shop:a price:#N/A remark:order to -fd date:20091011 How can I do it? Many thanks (2 Replies)
Discussion started by: lalelle
2 Replies

10. Shell Programming and Scripting

Merge lines into one

Source data file from oracle, terminated by ",". 'Cause some of fields have \r\n, then those lines were splitted into multiple lines in the expoted data file. Just vi this file, and found ^M. How to concatenate these line into one if it has a ^M at then end. thanks, (7 Replies)
Discussion started by: anypager
7 Replies
Login or Register to Ask a Question