Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Awk: group multiple fields from different records Post 303019779 by beca123456 on Friday 6th of July 2018 06:51:13 AM
Old 07-06-2018
Awk: group multiple fields from different records

Hi,

My input looks like that:
Code:
A|123|qwer
A|456|tyui
A|456|wsxe
B|789|dfgh

Using awk, I am trying to get:
Code:
A|123;456|qwer;tyui;wsxe
B|789|dfgh

For records with same $1, group all the $2 in a field (without replicates), and all the $3 in a field (without replicates).

What I have tried:
Code:
echo -e "A|123|qwer\nA|456|tyui\nA|456|wsxe\nB|789|dfgh" | gawk 'BEGIN{FS=OFS="|"}{a[$1]=sprintf("%s%s", a[$1], a[$1] ~ /$2/ ? "":";"$2); b[$1]=sprintf("%s%s", b[$1], b[$1] ~ /$3/ ? "":";"$3)}END{for(i in a){print i FS a[i] FS b[i]}}'

(Wrong) output:
Code:
A|;123;456;456|;qwer;tyui;wsxe
B|;789|;dfgh

However, I still cannot manage to remove the duplicated strings inside fields $2 and $3.

Last edited by beca123456; 07-06-2018 at 09:12 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

AWK ??-print for fields within records in a file

Hello all, Would appreciate if someone can help me out on the following requirement. INPUT FILE: -------------------------- TPS REPORT abc def ghi jkl mon pqr stu vrs lll END OF TPS REPORT TPS REPORT field1 field2 field3 field4 field5 field6 (8 Replies)
Discussion started by: hyennah
8 Replies

2. Infrastructure Monitoring

Processing records as group - awk

I have a file has following records policy glb id 1233 name Permit ping from "One" to "Second" "Address1" "Any" "ICMP-ANY" permit policy id 999251 service "snmp-udp" exit policy glb id 1234 name Permit telnet from "One" to "Second" "Address2" "Any" "TCP-ANY" permit policy id 1234... (3 Replies)
Discussion started by: baskar
3 Replies

3. Shell Programming and Scripting

how to parse with awk (using different fields), then group by a field?

When parsing multiple fields in a file using AWK, how do you group by one of the fields and parse by delimiters? to clarify If a file had tom | 223-2222-4444 , randofield ivan | 123-2422-4444 , random filed ... | and , are the delimiters ... How would you group by the social security... (4 Replies)
Discussion started by: Josef_Stalin
4 Replies

4. UNIX for Dummies Questions & Answers

keeping last record among group of records with common fields (awk)

input: ref.1;rack.1;1 #group1 ref.1;rack.1;2 #group1 ref.1;rack.2;1 #group2 ref.2;rack.3;1 #group3 ref.2;rack.3;2 #group3 ref.2;rack.3;3 #group3 Among records from same group (i.e. with same 1st and 2nd field - separated by ";"), I would need to keep the last record... (5 Replies)
Discussion started by: beca123456
5 Replies

5. Shell Programming and Scripting

awk gsub multiple fields

Hi, I am trying to execute this line awk -F ";" -v OFS=";" '{gsub(/\./,",",$6); print}' FILE but for multiple fields $6 $7 $8 Do you have a suggstion? Tried: awk -F ";" -v OFS="";"" "function GSUB( F ) {gsub(/\./,\",\",$F); print} { GSUB( 6 ); GSUB( 7 ); GSUB( 8 ) } 1"... (2 Replies)
Discussion started by: nakaedu
2 Replies

6. Shell Programming and Scripting

awk multiple fields separators

Can you please help me with this .... Input File share "FTPTransfer" "/v31_fs01/root/FTP-Transfer" umask=022 maxusr=4294967295 netbios=NJ09FIL530 share "Test" "/v31_fs01/root/Test" umask=022 maxusr=4294967295 netbios=NJ09FIL530 share "ENR California" "/v31_fs01/root/ENR California"... (14 Replies)
Discussion started by: greycells
14 Replies

7. UNIX for Dummies Questions & Answers

Make all records with the same number of fields (awk)

Hi, input: AA|BB|CC DD|EE FF what I am trying to get: AA|BB|CC DD|EE| FF|| I tried to create first an UDF for printing repeats, but I think I have an issue with my END section or my array: function repeat(str, n, rep, i) { for(i=1 ;i<n;i++) rep=rep str return rep } ... (6 Replies)
Discussion started by: beca123456
6 Replies

8. Shell Programming and Scripting

Shell Script to Group by Based on Multiple Fields in a file

Hi, I want to know if there is any simple approach to SUM a field based on group by of different fields for e.g. file1.txt contains below data 20160622|XXX1||50.00||50.00|MONEY|Plan1| 20160622|XXX1||100.00||100.00|MONEY|Plan1| 20160623|XXX1||25.00||25.00|MONEY|Plan1|... (3 Replies)
Discussion started by: cnu_theprince
3 Replies

9. Shell Programming and Scripting

Print multiple fields with awk

so its common knowledge one can print multiple fields with simple commands like this: echo 12 44 45 552 24 | awk '{print $1,$4,$3}' but suppose i want to avoid specifying the "$" symbol. is that possible? can something like this be done: echo 12 44 45 552 24 | awk '{print $(1,4,3)}' ... (9 Replies)
Discussion started by: SkySmart
9 Replies

10. UNIX for Beginners Questions & Answers

awk for matching fields between files with repeated records

Hello all, I am having trouble with what should be an easy task, but seem to be missing something fundamental. I have two files, with File 1 consisting of a single field of many thousands of records. I also have File 2 with two fields and many thousands of records. My goal is that when $1 of... (2 Replies)
Discussion started by: jvoot
2 Replies
CROSSPOST(8)						      System Manager's Manual						      CROSSPOST(8)

NAME
crosspost - create the links for cross posted articles SYNOPSIS
crosspost [ -D dir ] [ -s ] [ file... ] DESCRIPTION
Crosspost reads group and article number data from files or standard input if none are specified. (A single dash in the file list means to read standard input.) It uses this information to create the hard, or symbolic, links for cross posted articles. Crosspost is designed to be used by InterNetNews to create the links as the articles come in. Normally innd creates the links but by having crosspost create the links innd spends less time waiting for disk IO. In this mode one would start innd(8) using the ``-L'' flag. Crosspost expects input in the form: group.name/123 group2.name/456 group3.name/789 with one line per article. Any dots in the input are translated into "/" to translate the news group into a pathname. The first field is assumed to be the name of an existing copy of the article. Crosspost will attempt to link all the subsequent entries to the first using hard links if possible or symbolic links if that fails. By default, crosspost processes its input as an INN channel feed written as a ``WR'' entry in the newsfeeds(5) file, for example: crosspost:*:Tc,Ap,WR:/usr/lib/news/bin/crosspost To process the history file and re-create all the links for all articles use: awk <history -F' ' '(NF > 2){print $3}' | crosspost (where the -F is followed by a tab character.) The ``-D'' flag can be used to specify where the article spool is stored. The default directory is /var/spool/news. By default crosspost will fsync(2) each article after updating the links. The ``-s'' flag can be used to prevent this. HISTORY
Written by Jerry Aguirre <jerry@ATC.Olivetti.Com>. SEE ALSO
newsfeeds(5), innd(8). CROSSPOST(8)
All times are GMT -4. The time now is 08:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy