×
UNIX.COM Login
Username:
Password:  
Show Password






👤


Shell Programming and Scripting

BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Text processing in UNIX

👤 Login to reply

 
Thread Tools Search this Thread Display Modes
    #1  
Old 06-04-2018
bikerboy bikerboy is offline
Registered User
 
Join Date: Feb 2015
Last Activity: 4 June 2018, 10:19 AM EDT
Posts: 11
Thanks: 1
Thanked 0 Times in 0 Posts
Text processing in UNIX

Greetings!

I have a text file that I am trying to process to get the desired output but looks like I will need the community help.

Input File:




Code:
a|x|london|consumer|consumer1|country||D|consumer|consumer1|country||1
a|x|paris|consumer|consumer2|country||D|consumer1|consumer2|country||2
a|x|paris|consumer3|consumer2|country||D|consumer1|consumer2|country||2
a|x|spain|id|sys|id|country|U|consumer|sys|id|country|3
b|x|spain|hash|sys|id|country|U|consumer|sys|id|country|3
b|x|spain|tkn|sys|id|country|U|consumer|sys|id|country|3
b|x|spain|txt|sys|id|country|U|consumer|sys|id|country|3


I need to read a file similar to the above example and get the following output. the output needs to be broken down by the number we see at the end of each line. Same numbered lines needs to be grouped.

Output:



Code:
a|x|london|consumer=consumer|consumer1=consumer1 and country=country|1
a|x|paris|consumer=consumer1, consumer3=consumer1|consumer2 = consumer2 and country=country|2
a|x|spain|id=consumer, hash=consumer,tkn=consumer,txt=consumer|sys=sys and id=id and country=country|3

In other terms.

for each number grouping in the file we do this



Code:
$1|$2|$3|$4=$9[,$4=$9 if we have more than one line]|$5=$10[and $6=$11 and $7=$12 if they have some value in their fields]| $13

Here is what I tried..




Code:
 key=$(head -1 filename.out | awk -F'|' '{ print $1"|"$2"|"$3 }')
        value1=""
        value2=""

        while read p; do
          anon1=$( echo $p | awk -F'|' '{ print $4"="$9 }')
            if [ "$value1" != "" ]; then value1="${value1}, ${anon1}"; else value1="${anon1}"; fi
          anon2=$( echo $p | awk -F'|' '{ print $5"="$10" and "$6"="$11" and "$7"="$12 }')
        done < filename.out

        if [ "$value2" != "" ]; then value2="${value2} and ${anon2}"; else value2="${anon2}"; fi
        echo $key"|"$value1"|"$value2


the above code works but i want to loop this through the rest of the file where we have lines broken down by numbered values like the input shown above in the example. if i run the above code to the above input it combines all the lines which is not the desired output. it should only combine the lines that share the same numbered value at the end of each line.

Thank you very much.
Sponsored Links
    #2  
Old 3 Weeks Ago
MadeInGermany MadeInGermany is offline Forum Staff  
Moderator
 
Join Date: May 2012
Last Activity: 16 July 2018, 3:34 PM EDT
Location: Simplicity
Posts: 4,153
Thanks: 365
Thanked 1,418 Times in 1,274 Posts
You must delay the printout until the last field changes, and then print the stored values. At the end of the file you must do it once more, therefore a function is appropriate.


Code:
prtout(){
  [[ -n $lastf13 ]] && echo "$key|$out1|$out2|$lastf13"
}

out1=""
out2=""
lastf13=""
while IFS="|" read f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 f14
do
  if [[ $f13 != $lastf13 ]]
  then
    prtout
    key="$f1|$f2|$f3"
    out1="$f4=$f9"
    out2="$f5=$f10"
    [[ -n $f6 ]] && out2="$out2 and $f6=$f11"
    [[ -n $f7 ]] && out2="$out2 and $f7=$f12"
    lastf13=$f13
  else
    out1="$out1,$f4=$f9"
  fi
done < filename.out
prtout

Sponsored Links
    #3  
Old 3 Weeks Ago
MadeInGermany MadeInGermany is offline Forum Staff  
Moderator
 
Join Date: May 2012
Last Activity: 16 July 2018, 3:34 PM EDT
Location: Simplicity
Posts: 4,153
Thanks: 365
Thanked 1,418 Times in 1,274 Posts
The shell has the advantage that you can easily boost it with external commands.
If this is not intended, it is ideal for awk (using its loop over lines with automatic field splitting)


Code:
# shell code with embeded awk 'script'
awk '
  BEGIN { FS=OFS="|" }
# the FS is the field delimiter for the input
# the OSF is for the print command
  function prtout(){
    if (lastf13!="") print key,out1,out2,lastf13
  }
# main loop
  {
    if ($13!=lastf13) {
      prtout()
      key=($1 FS $2 FS $3)
      out1=($4 "=" $9)
      out2=($5 "=" $10)
      if ($6!="") out2=(out2 " and " $6 "=" $11)
      if ($7!="") out2=(out2 " and " $7 "=" $12)
      lastf13=$13
    } else {
      out1=(out1 "," $4 "=" $9)
    }
  }
  END { prtout() }
' < filename.out

    #4  
Old 3 Weeks Ago
Chubler_XL's Unix or Linux Image
Chubler_XL Chubler_XL is online now Forum Staff  
Moderator
 
Join Date: Oct 2010
Last Activity: 16 July 2018, 4:15 PM EDT
Posts: 3,551
Thanks: 159
Thanked 1,267 Times in 1,161 Posts
If your input file is not sorted on the last field, awk can build up arrays for the fields and output at the end:


Code:
awk -F'|' '{
    if ($13 in k)
        d1[$13] = d1[$13] "," $4 "=" $9
    else {
        d1[$13] = $4 "=" $9
        k[$13] = $1 FS $2 FS $3
        d2[$13] = $5 "=" $10
        if ($6!="") d2[$13] = d2[$13] " and " $6 "=" $11
        if ($7!="") d2[$13] = d2[$13] " and " $7 "=" $12
    }
}
END {
   for(id in k)
      print k[id] FS d1[id] FS d2[id] FS id
}' filename.out

Sponsored Links
👤 Login to reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Text processing getmilo Shell Programming and Scripting 4 07-31-2017 11:01 AM
Help with text processing bikerboy Shell Programming and Scripting 7 02-25-2015 09:39 PM
Text processing using awk dovah Shell Programming and Scripting 3 07-17-2014 01:23 AM
awk processing / Shell Script Processing to remove columns text file ajayram Programming 5 03-23-2013 01:51 PM
Awk text processing zam Shell Programming and Scripting 8 01-12-2010 07:11 PM



All times are GMT -4. The time now is 04:16 PM.

Unix & Linux Forums Content Copyright©1993-2018. All Rights Reserved.