Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: formatting data file with awk or sed
Top Forums Shell Programming and Scripting formatting data file with awk or sed Post 302407590 by durden_tyler on Thursday 25th of March 2010 09:55:01 PM
Old 03-25-2010
One way to do it with Perl -

Code:
perl -ne 'chomp; if (/^\d+/ && $.<=80){$i++<=9 ? push @x,split/ /,$_ : push @y,split/ /,$_} END{foreach $i(@x){foreach $j(@y){print "$i\t$j\n"}}}' yourfile

tyler_durden

---------- Post updated at 09:55 PM ---------- Previous update was at 09:45 PM ----------

Here's the test on a dummy file with similar structure, on my system:

Code:
$ 
$ cat -n data.txt
     1  header line 1
     2  header line 2
     3  header line 3
     4  123 456
     5  901 234
     6  000 111
     7  666 777
     8  334
     9  real data line 1
    10  real data line 2
$ 
$ perl -ne 'chomp; if (/^\d+/ && $.<=8){$i++<=1 ? push @x,split/ /,$_ : push @y,split/ /,$_}
            END {foreach $i(@x){foreach $j(@y){print "$i\t$j\n"}}}' data.txt
123     000
123     111
123     666
123     777
123     334
456     000
456     111
456     666
456     777
456     334
901     000
901     111
901     666
901     777
901     334
234     000
234     111
234     666
234     777
234     334
$ 
$

Line nos. 4 and 5 consist of x values (123, 456, 901, 234).
Line nos. 6, 7 and 8 consist of y values (000, 111, 666, 777, 334).
Line no. 8 in my file corresponds to line no. 80 in yours.
I test for $i++ <= 1 because only the first two matching lines contain x values. You'd test for $i++ <= 9 because the first 10 matching lines contain x values in your file.

HTH,
tyler_durden
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Clense Junk Data File - Using Shell or awk or sed

Hello Shell Gurus i need help in solving this puzzle. We have a junk data file that needs to be fed into the database. Need to clense the data file thru shell script. I am not a expert and so need help with Here is what i need to do on the input file -Step -1 Replace all pipes ‘|' within... (1 Reply)
Discussion started by: rimss
1 Replies

2. Shell Programming and Scripting

Big data file - sed/grep/awk?

Morning guys. Another day another question. :rolleyes: I am knocking up a script to pull some data from a file. The problem is the file is very big (up to 1 gig in size), so this solution: for results in `grep "^\ ... works, but takes ages (we're talking minutes) to run. The data is held... (8 Replies)
Discussion started by: dlam
8 Replies

3. Shell Programming and Scripting

Awk formatting of a data file - nested for loops?

Hello - is there any way in awk I can do... 4861 x(1) y(1) z(1) 4959 x(1) y(1) z(1) 5007 x(1) y(1) z(1) 4861 x(2) y(2) z(2) 4959 x(2) y(2) z(2) 5007 x(2) y(2) z(2) 4861 x(3) y(3) z(3) 4959 x(3) y(3) z(3) 5007 x(3) y(3) z(3) to become... 4861 x(1) y(1) z(1) 4861 x(2) y(2) z(2)... (3 Replies)
Discussion started by: catwoman
3 Replies

4. Shell Programming and Scripting

sed or awk to extract data from Xml file

Hi, I want to get data from Xml file by using sed or awk command. I want to get the following result : mon titre 1;Createur1;Dossier1 mon titre 1;Createur1;Dossier1 and save it in cvs file (fichier.cvs). FROM this Xml file (test.xml): <playlist version="1"> <trackList> <track>... (1 Reply)
Discussion started by: yeclota
1 Replies

5. Shell Programming and Scripting

Formatting help needed awk or sed maybe

I am executing the following command: sort file1.txt | uniq -c | sort -n > file2.txt The problem is that in file 2, I get leading spaces, Like so: 1 N/A|A8MW11 8 N/A|ufwo1 9 N/A|a8mw11 10 900003|smoketest297688 10 N/A|a9dg4 10 danny|danni 12... (5 Replies)
Discussion started by: ddurden7
5 Replies

6. Shell Programming and Scripting

how to get data from hex file using SED or AWK based on pattern sign

I have a binary (hex) file I need to parse to get some data which are encoded this way: .* b4 . . . 01 12 .* af .* 83 L1 x1 x2 xL 84 L2 y1 y2 yL By another words there is a stream of hexadecimal bytes (in my example separated by space for better readability). I need to get value stored in... (3 Replies)
Discussion started by: sameucho
3 Replies

7. Shell Programming and Scripting

AWK/Shell script for formatting data in a file

Hi All, Need an urgent help to convert a unix file in to a particular format: **source file:** 1111111 2d2f2h2 3dfgsd3 ........... 1111111 <-- repeats in every nth line. remaining all lines will be different 123ss41 432ff45 ........... 1111111 <-- repetition qwe1234 123weq3... (1 Reply)
Discussion started by: rajivnairfis
1 Replies

8. Shell Programming and Scripting

awk - sed / reading from a data file and doing algebraic operations

Hi everyone, I am trying to write a bash script which reads a data file and does some algebraic operations. here is the structure of data.xml file that I have; 1 <data> 2 . 3 . 4 . 5 </data> 6 <data> 7 . 8 . 9 . 10</data> etc. Each data block contains same number of lines (say... (4 Replies)
Discussion started by: hayreter
4 Replies

9. Shell Programming and Scripting

Data formatting using awk

Need assistance on the data extraction using awk Below is the format and would like to extract the data in another format ------------------------------------------------------------------------------------------------- Minimum Temperature (deg F ) DAY 1 2 3 4 5 6 7 8 9 10 11... (4 Replies)
Discussion started by: ajayram_arya
4 Replies

10. Shell Programming and Scripting

Problem in formatting output in sed / awk

I have a file like this : ! 1 ! 542255 ! 50,140.00 ! ! 2 ! 551717 ! 5,805.00 ! ! 3 ! 551763 ! 8,130.00 ! ! 4 ! 551779 ! 750.00 ! ! 5 ! 551810 ! 56,580.00 ! ! 6 ! 551816 ! 1,350.00 ! ! 7 ! 551876 ! 360.00 ! ! 8 ! 551898 ! ... (10 Replies)
Discussion started by: adam1969in
10 Replies

LEARN ABOUT DEBIAN

moose::meta::attribute::native::trait::string

Moose::Meta::Attribute::Native::Trait::String(3pm)	User Contributed Perl Documentation	Moose::Meta::Attribute::Native::Trait::String(3pm)

NAME

       Moose::Meta::Attribute::Native::Trait::String - Helper trait for Str attributes

VERSION

       version 2.0603

SYNOPSIS

	 package MyHomePage;
	 use Moose;

	 has 'text' => (
	     traits  => ['String'],
	     is      => 'rw',
	     isa     => 'Str',
	     default => q{},
	     handles => {
		 add_text     => 'append',
		 replace_text => 'replace',
	     },
	 );

	 my $page = MyHomePage->new();
	 $page->add_text("foo");    # same as $page->text($page->text . "foo");

DESCRIPTION

       This trait provides native delegation methods for strings.

DEFAULT TYPE

       If you don't provide an "isa" value for your attribute, it will default to "Str".

PROVIDED METHODS

       o   inc

	   Increments the value stored in this slot using the magical string autoincrement operator. Note that Perl doesn't provide analogous
	   behavior in "--", so "dec" is not available. This method returns the new value.

	   This method does not accept any arguments.

       o   append($string)

	   Appends to the string, like ".=", and returns the new value.

	   This method requires a single argument.

       o   prepend($string)

	   Prepends to the string and returns the new value.

	   This method requires a single argument.

       o   replace($pattern, $replacement)

	   Performs a regexp substitution ("s" in perlop). There is no way to provide the "g" flag, but code references will be accepted for the
	   replacement, causing the regex to be modified with a single "e". "/smxi" can be applied using the "qr" operator. This method returns
	   the new value.

	   This method requires two arguments.

       o   match($pattern)

	   Runs the regex against the string and returns the matching value(s).

	   This method requires a single argument.

       o   chop

	   Just like "chop" in perlfunc. This method returns the chopped character.

	   This method does not accept any arguments.

       o   chomp

	   Just like "chomp" in perlfunc. This method returns the number of characters removed.

	   This method does not accept any arguments.

       o   clear

	   Sets the string to the empty string (not the value passed to "default").

	   This method does not have a defined return value.

	   This method does not accept any arguments.

       o   length

	   Just like "length" in perlfunc, returns the length of the string.

       o   substr

	   This acts just like "substr" in perlfunc. When called as a writer, it returns the substring that was replaced, just like the Perl
	   builtin.

	   This method requires at least one argument, and accepts no more than three.

BUGS

       See "BUGS" in Moose for details on reporting bugs.

AUTHOR

       Moose is maintained by the Moose Cabal, along with the help of many contributors. See "CABAL" in Moose and "CONTRIBUTORS" in Moose for
       details.

COPYRIGHT AND LICENSE

       This software is copyright (c) 2012 by Infinity Interactive, Inc..

       This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

perl v5.14.2							    2012-06-28			Moose::Meta::Attribute::Native::Trait::String(3pm)
All times are GMT -4. The time now is 02:51 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy