Sponsored Content
Top Forums Shell Programming and Scripting Awk multiple lines with 3rd column onto a single line? Post 302264764 by SoMoney on Thursday 4th of December 2008 05:33:13 PM
Old 12-04-2008
Awk multiple lines with 3rd column onto a single line?

I have a H U G E file with over 1million entries in it.
Looks something like this:

USER0001|DEVICE001|VAR1
USER0001|DEVICE001|VAR2
USER0001|DEVICE001|VAR3
USER0001|DEVICE001|VAR4
USER0001|DEVICE001|VAR5
USER0001|DEVICE001|VAR6
USER0001|DEVICE002|VAR1
USER0001|DEVICE002|VAR2
USER0001|DEVICE002|VAR3
USER0001|DEVICE002|VAR4
USER0001|DEVICE002|VAR5
USER0002|DEVICE003|VAR1
USER0002|DEVICE003|VAR2
USER0002|DEVICE003|VAR3
USER0002|DEVICE003|VAR4
USER0002|DEVICE003|VAR5
USER0002|DEVICE004|VAR1
USER0002|DEVICE004|VAR2
USER0002|DEVICE004|VAR3
USER0002|DEVICE004|VAR4
USER0002|DEVICE004|VAR5
USER0003|DEVICE005|VAR1
USER0003|DEVICE005|VAR2
USER0003|DEVICE005|VAR3
USER0003|DEVICE005|VAR4
USER0003|DEVICE005|VAR5
USER0003|DEVICE005|VAR6
USER0003|DEVICE006|VAR1
USER0003|DEVICE006|VAR2
USER0003|DEVICE006|VAR6


I would like a way to output only the uniq veriables onto a single line like so:

USER0001|DEVICE001|VAR1|VAR2|VAR3|VAR4|VAR5|VAR6
USER0001|DEVICE002|VAR1|VAR2|VAR3|VAR4|VAR5
USER0002|DEVICE003|VAR1|VAR2|VAR3|VAR4|VAR5
USER0002|DEVICE004|VAR1|VAR2|VAR3|VAR4|VAR5
USER0003|DEVICE005|VAR1|VAR2|VAR3|VAR4|VAR5|VAR6
USER0003|DEVICE006|VAR1|VAR2|VAR6


Is this doable with awk or do I need to use perl array?

Last edited by SoMoney; 12-04-2008 at 06:39 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting a single line into multiple lines

I have a case where, I need to look into a file. Go to each line of the file, find the length of the line, if the length of the line is more than 75 chars, I need to split the line into multiple lines of 75chars max. If the length of the line is less than 75, we need not do anything. So at the... (4 Replies)
Discussion started by: thanuman
4 Replies

2. Shell Programming and Scripting

Multiple lines into a single line

Hi, I've some files with the following data and i need to convert the lines between the separator ---, into a single line. I've tried with the paste cmd but my main problem is that the number of lines between the separator is not fix, it can very between 1-4 lines. Input --- 2010-02-22... (4 Replies)
Discussion started by: RickyC9999
4 Replies

3. Shell Programming and Scripting

Combine multiple lines in single line

This is related to one of my previous post but now with a slight difference: I need the "Updated:" to be in one line as well as the "Information:" on one line as well. These are in multiple lines right now as seen below. These can have 2 or more lines that needs to be in one line. System name:... (8 Replies)
Discussion started by: The One
8 Replies

4. Shell Programming and Scripting

Multiple lines in a single column to be merged as a single line for a record

Hi, I have a requirement with, No~Dt~Notes 1~2011/08/1~"aaa bbb ccc ddd eee fff ggg hhh" Single column alone got splitted into multiple lines. I require the output as No~Dt~Notes 1~2011/08/1~"aaa<>bbb<>ccc<>ddd<>eee<>fff<>ggg<>hhh" mean to say those new lines to be... (1 Reply)
Discussion started by: Bhuvaneswari
1 Replies

5. Shell Programming and Scripting

Awk multiple lines with 4th column on to a single line

This is related to one of my previous post.. I have huge file currently I am using loop to read file and checking each line to build this single record, its taking much much time to parse those records.. I thought there should be a way to do this in awk or sed. I found this code in this forum... (7 Replies)
Discussion started by: Vasan
7 Replies

6. Shell Programming and Scripting

Awk match multiple columns in multiple lines in single file

Hi, Input 7488 7389 chr1.fa chr1.fa 3546 9887 chr5.fa chr9.fa 7387 7898 chrX.fa chr3.fa 7488 7389 chr21.fa chr3.fa 7488 7389 chr1.fa chr1.fa 3546 9887 chr9.fa chr5.fa 7898 7387 chrX.fa chr3.fa Desired Output 7488 7389 chr1.fa chr1.fa 2 3546 9887 chr5.fa chr9.fa 2... (2 Replies)
Discussion started by: jacobs.smith
2 Replies

7. Shell Programming and Scripting

Combine multiple lines into single line

Hi All , I have a file with below data # User@Host: xyz @ # Query_time: t1 Lock_time: t2 Rows_sent: n1 Rows_examined: n2 SET timestamp=1396852200; select count(1) from table; # Time: 140406 23:30:01 # User@Host: abc @ # Query_time: t1 Lock_time: t2 Rows_sent: n1 Rows_examined:... (6 Replies)
Discussion started by: rakesh_411
6 Replies

8. Shell Programming and Scripting

Coverting multiple lines to a single line

Hi all, I have a requirement to covert multiple lines in a comma delimited file to a single line through shell scripting. We should compare the data in the first column in each line. If it is same, then the other data should be put in the same line.Below is the sample input and expected output:... (4 Replies)
Discussion started by: Bobby_2000
4 Replies

9. UNIX for Beginners Questions & Answers

Merging multiple lines into single line based on one column

I Want to merge multiple lines based on the 1st field and keep into single record. SRC File: AAA_POC_DB.TAB1 AAA_POC_DB.TAB2 AAA_POC_DB.TAB3 AAA_POC_DB.TAB4 BBB_POC_DB.TAB1 BBB_POC_DB.TAB2 CCC_POC_DB.TAB6 OUTPUT ----------------- 'AAA_POC_DB','TAB1','TAB2','TAB3','TAB4'... (10 Replies)
Discussion started by: raju2016
10 Replies

10. UNIX for Beginners Questions & Answers

Multiple lines to single line

I have code as below # create temporary table `temp4277`(key(waybill_no)) select waybill_no,concat_ws('',card_type,card_series_no) cardinfo from rfid_temp_ticket where waybill_no='4277' group by... (4 Replies)
Discussion started by: kaushik02018
4 Replies
Data::Dumper::Simple(3pm)				User Contributed Perl Documentation				 Data::Dumper::Simple(3pm)

NAME
Data::Dumper::Simple - Easily dump variables with names SYNOPSIS
use Data::Dumper::Simple; warn Dumper($scalar, @array, %hash); warn Dumper($scalar, @array, \%hash); warn Dumper $scalar, @array, %hash; ABSTRACT
This module allow the user to dump variables in a Data::Dumper format. Unlike the default behavior of Data::Dumper, the variables are named (instead of $VAR1, $VAR2, etc.) Data::Dumper provides an extended interface that allows the programmer to name the variables, but this interface requires a lot of typing and is prone to tyops (sic). This module fixes that. DESCRIPTION
"Data::Dumper::Simple" is actually a source filter that replaces all instances of "Dumper($some, @args)" in your code with a call to "Data::Dumper->Dump()". You can use the one function provided to make dumping variables for debugging a trivial task. Note that this is primarily a debugging tool. "Data::Dumper" offers a bit more than that, so don't expect this module to be more than it is. Note that if you strongly object to source filters, I've also released Data::Dumper::Names. It does what this module does by it uses Pad- Walker instead of a source filter. Unfortunately, it has a few limitations and is not as powerful as this module. Think of Data::Dumper::Names as a "proof of concept". The Problem Frequently, we use "Data::Dumper" to dump out some variables while debugging. When this happens, we often do this: use Data::Dumper; warn Dumper($foo, $bar, $baz); And we get simple output like: $VAR1 = 3; $VAR2 = 2; $VAR3 = 1; While this is usually what we want, this can be confusing if we forget which variable corresponds to which variable printed. To get around this, there is an extended interface to "Data::Dumper": warn Data::Dumper->Dump( [$foo, $bar, $baz], [qw/*foo *bar *baz/] ); This provides much more useful output. $foo = 3; $bar = 2; $baz = 1; (There's more control over the output than what I've shown.) You can even use this to output more complex data structures: warn Data::Dumper->Dump( [$foo, @array], [qw/*foo *array/] ); And get something like this: $foo = 3; @array = ( 8, 'Ovid' ); Unfortunately, this can involve a lot of annoying typing. warn Data::Dumper->Dump( [$foo, \%this, @array, \%that], [qw/*foo *that *array *this/] ); You'll also notice a typo in the second array ref which can cause great confusion while debugging. The Solution With "Data::Dumper::Simple" you can do this instead: use Data::Dumper::Simple. warn Dumper($scalar, @array, %hash); Note that there's no need to even take a reference to the variables. The output of the above resembles this (sample data, of course): $scalar = 'Ovid'; @array = ( 'Data', 'Dumper', 'Simple', 'Rocks!' ); %hash = ( 'it' => 'does', 'I' => 'hope', 'at' => 'least' ); Taking a reference to an array or hash works as expected, but taking a reference to a scalar is effectively a no-op (because it can turn into a confusing reference to a reference); my $foo = { hash => 'ref' }; my @foo = qw/foo bar baz/; warn Dumper ($foo, @foo); Produces: $foo = { 'hash' => 'ref' }; $foo = [ 'foo', 'bar', 'baz' ]; Note that this means similarly named variables can get quite confusing, as in the example above. If you already have a &Dumper function, you can specify a different function name with the "as" key in the import list: use Data::Dumper::Simple as => 'display'; warn display( $scalar, @array, %hash ); Also, if you really, really can't stand typing "warn" or "print", you can turn on "autowarn": use Data::Dumper::Simple as => 'display', autowarn => 1; display($scalar, @array, $some->{ data }); Or you can send the output (as a list) to a different function: use Data::Dumper::Simple as => 'debug', autowarn => 'to_log'; sub to_log { my @data = @_; # some logging function } debug( $customer => @order_nums ); # yeah, we support the fat comma "=>" and newlines EXPORT
The only thing exported is the Dumper() function. Well, actually that's not really true. Nothing is exported. However, a source filter is used to automatically rewrite any apparent calls to "Dumper()" so that it just Does The Right Thing. SEE ALSO
* Data::Dumper - Stringified perl data structures * Filter::Simple - Simplified source filtering BUGS
This module uses a source filter. If you don't like that, don't use this. There are no known bugs but there probably are some as this is Alpha Code. LIMITATIONS
* Calling with a sub Do not try to call "Dumper()" with a subroutine in the argument list: Dumper($foo, some_sub()); # Bad! The filter gets confused by the parentheses. Your author was going to fix this but it became apparent that there was no way that "Dumper()" could figure out how to name the return values from the subroutines, thus ensuring further breakage. So don't do that. * Multiple enreferencing Getting really crazy by using multiple enreferencing will confuse things (e.g., "\\\$foo"), don't do that, either. I might use "Text::Balanced" at some point to fix this if it's an issue. * Slices List and hash slices are not supported at this time. * String interpolation "Dumper($foo)" can potentially interpolate if it's in a string. This is because of a weird edge case with "FILTER_ONLY code" which caused a failure on some items being dumped. I've fixed that, but made the module a wee bit less robust. This will hopefully be fixed in the next release of Text::Balanced. * Line numbers may be wrong Because this module uses a source filter, line numbers reported from syntax or other errors may be thrown off a little. This is probably a bug in the source filter implementation, which should use "#line" directives. As a workaround until this is fixed, put a directive (such as "#line 10000") a few lines ahead of the suspected bug. If the error is reported as happening in line 10007, you know to look about eight lines below your directive for the bug. Be sure to remove the bogus directive once you find the bug! * The parentheses are optional, but the syntax isn't bulletproof If you try, it's not hard to confuse the parser. Patches welcome. Note that this is not a drop-in replacement for "Data::Dumper". If you need the power of that module, use it. AUTHOR
Curtis "Ovid" Poe, <eop_divo_sitruc@yahoo.com> Reverse the name to email me. COPYRIGHT AND LICENSE
Copyright 2004 by Curtis "Ovid" Poe This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.8.8 2008-03-08 Data::Dumper::Simple(3pm)
All times are GMT -4. The time now is 11:25 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy