Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: formatting data file with awk or sed
Top Forums Shell Programming and Scripting formatting data file with awk or sed Post 302407590 by durden_tyler on Thursday 25th of March 2010 09:55:01 PM
Old 03-25-2010
One way to do it with Perl -

Code:
perl -ne 'chomp; if (/^\d+/ && $.<=80){$i++<=9 ? push @x,split/ /,$_ : push @y,split/ /,$_} END{foreach $i(@x){foreach $j(@y){print "$i\t$j\n"}}}' yourfile

tyler_durden

---------- Post updated at 09:55 PM ---------- Previous update was at 09:45 PM ----------

Here's the test on a dummy file with similar structure, on my system:

Code:
$ 
$ cat -n data.txt
     1  header line 1
     2  header line 2
     3  header line 3
     4  123 456
     5  901 234
     6  000 111
     7  666 777
     8  334
     9  real data line 1
    10  real data line 2
$ 
$ perl -ne 'chomp; if (/^\d+/ && $.<=8){$i++<=1 ? push @x,split/ /,$_ : push @y,split/ /,$_}
            END {foreach $i(@x){foreach $j(@y){print "$i\t$j\n"}}}' data.txt
123     000
123     111
123     666
123     777
123     334
456     000
456     111
456     666
456     777
456     334
901     000
901     111
901     666
901     777
901     334
234     000
234     111
234     666
234     777
234     334
$ 
$

Line nos. 4 and 5 consist of x values (123, 456, 901, 234).
Line nos. 6, 7 and 8 consist of y values (000, 111, 666, 777, 334).
Line no. 8 in my file corresponds to line no. 80 in yours.
I test for $i++ <= 1 because only the first two matching lines contain x values. You'd test for $i++ <= 9 because the first 10 matching lines contain x values in your file.

HTH,
tyler_durden
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Clense Junk Data File - Using Shell or awk or sed

Hello Shell Gurus i need help in solving this puzzle. We have a junk data file that needs to be fed into the database. Need to clense the data file thru shell script. I am not a expert and so need help with Here is what i need to do on the input file -Step -1 Replace all pipes ‘|' within... (1 Reply)
Discussion started by: rimss
1 Replies

2. Shell Programming and Scripting

Big data file - sed/grep/awk?

Morning guys. Another day another question. :rolleyes: I am knocking up a script to pull some data from a file. The problem is the file is very big (up to 1 gig in size), so this solution: for results in `grep "^\ ... works, but takes ages (we're talking minutes) to run. The data is held... (8 Replies)
Discussion started by: dlam
8 Replies

3. Shell Programming and Scripting

Awk formatting of a data file - nested for loops?

Hello - is there any way in awk I can do... 4861 x(1) y(1) z(1) 4959 x(1) y(1) z(1) 5007 x(1) y(1) z(1) 4861 x(2) y(2) z(2) 4959 x(2) y(2) z(2) 5007 x(2) y(2) z(2) 4861 x(3) y(3) z(3) 4959 x(3) y(3) z(3) 5007 x(3) y(3) z(3) to become... 4861 x(1) y(1) z(1) 4861 x(2) y(2) z(2)... (3 Replies)
Discussion started by: catwoman
3 Replies

4. Shell Programming and Scripting

sed or awk to extract data from Xml file

Hi, I want to get data from Xml file by using sed or awk command. I want to get the following result : mon titre 1;Createur1;Dossier1 mon titre 1;Createur1;Dossier1 and save it in cvs file (fichier.cvs). FROM this Xml file (test.xml): <playlist version="1"> <trackList> <track>... (1 Reply)
Discussion started by: yeclota
1 Replies

5. Shell Programming and Scripting

Formatting help needed awk or sed maybe

I am executing the following command: sort file1.txt | uniq -c | sort -n > file2.txt The problem is that in file 2, I get leading spaces, Like so: 1 N/A|A8MW11 8 N/A|ufwo1 9 N/A|a8mw11 10 900003|smoketest297688 10 N/A|a9dg4 10 danny|danni 12... (5 Replies)
Discussion started by: ddurden7
5 Replies

6. Shell Programming and Scripting

how to get data from hex file using SED or AWK based on pattern sign

I have a binary (hex) file I need to parse to get some data which are encoded this way: .* b4 . . . 01 12 .* af .* 83 L1 x1 x2 xL 84 L2 y1 y2 yL By another words there is a stream of hexadecimal bytes (in my example separated by space for better readability). I need to get value stored in... (3 Replies)
Discussion started by: sameucho
3 Replies

7. Shell Programming and Scripting

AWK/Shell script for formatting data in a file

Hi All, Need an urgent help to convert a unix file in to a particular format: **source file:** 1111111 2d2f2h2 3dfgsd3 ........... 1111111 <-- repeats in every nth line. remaining all lines will be different 123ss41 432ff45 ........... 1111111 <-- repetition qwe1234 123weq3... (1 Reply)
Discussion started by: rajivnairfis
1 Replies

8. Shell Programming and Scripting

awk - sed / reading from a data file and doing algebraic operations

Hi everyone, I am trying to write a bash script which reads a data file and does some algebraic operations. here is the structure of data.xml file that I have; 1 <data> 2 . 3 . 4 . 5 </data> 6 <data> 7 . 8 . 9 . 10</data> etc. Each data block contains same number of lines (say... (4 Replies)
Discussion started by: hayreter
4 Replies

9. Shell Programming and Scripting

Data formatting using awk

Need assistance on the data extraction using awk Below is the format and would like to extract the data in another format ------------------------------------------------------------------------------------------------- Minimum Temperature (deg F ) DAY 1 2 3 4 5 6 7 8 9 10 11... (4 Replies)
Discussion started by: ajayram_arya
4 Replies

10. Shell Programming and Scripting

Problem in formatting output in sed / awk

I have a file like this : ! 1 ! 542255 ! 50,140.00 ! ! 2 ! 551717 ! 5,805.00 ! ! 3 ! 551763 ! 8,130.00 ! ! 4 ! 551779 ! 750.00 ! ! 5 ! 551810 ! 56,580.00 ! ! 6 ! 551816 ! 1,350.00 ! ! 7 ! 551876 ! 360.00 ! ! 8 ! 551898 ! ... (10 Replies)
Discussion started by: adam1969in
10 Replies

LEARN ABOUT DEBIAN

perl::critic::policy::regularexpressions::prohibitfixedstringmat

Perl::Critic::Policy::RegularExpressions::ProhibitFixedSUsergContributed)Perl::Critic::Policy::RegularExpressions::ProhibitFixedStringMatches(3pm)

NAME

       Perl::Critic::Policy::RegularExpressions::ProhibitFixedStringMatches - Use "eq" or hash instead of fixed-pattern regexps.

AFFILIATION

       This Policy is part of the core Perl::Critic distribution.

DESCRIPTION

       A regular expression that matches just a fixed set of constant strings is wasteful of performance and is hard on maintainers.  It is much
       more readable and often faster to use "eq" or a hash to match such strings.

	   # Bad
	   my $is_file_function = $token =~ m/A (?: open | close | read ) z/xms;

	   # Faster and more readable
	   my $is_file_function = $token eq 'open' ||
				  $token eq 'close' ||
				  $token eq 'read';

       For larger numbers of strings, a hash is superior:

	   # Bad
	   my $is_perl_keyword =
	       $token =~ m/A (?: chomp | chop | chr | crypt | hex | index
				  lc | lcfirst | length | oct | ord | ... ) z/xms;

	   # Better
	   Readonly::Hash my %PERL_KEYWORDS => map {$_ => 1} qw(
	       chomp chop chr crypt hex index lc lcfirst length oct ord ...
	   );
	   my $is_perl_keyword = $PERL_KEYWORD{$token};

       Conway also suggests using "lc()" instead of a case-insensitive match.

   VARIANTS
       This policy detects both grouped and non-grouped strings.  The grouping may or may not be capturing.  The grouped body may or may not be
       alternating.  "A" and "z" are always considered anchoring which "^" and "$" are considered anchoring is the "m" regexp option is not in
       use.  Thus, all of these are violations:

	   m/^foo$/;
	   m/A foo z/x;
	   m/A foo z/xm;
	   m/A(foo)z/;
	   m/A(?:foo)z/;
	   m/A(foo|bar)z/;
	   m/A(?:foo|bar)z/;

       Furthermore, this policy detects violations in "m//", "s///" and "qr//" constructs, as you would expect.

CONFIGURATION

       This Policy is not configurable except for the standard options.

CREDITS

       Initial development of this policy was supported by a grant from the Perl Foundation.

AUTHOR

       Chris Dolan <cdolan@cpan.org>

COPYRIGHT

       Copyright (c) 2007-2011 Chris Dolan.  Many rights reserved.

       This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.  The full text of this license
       can be found in the LICENSE file included with this module

perl v5.14.2							    2012-Perl::Critic::Policy::RegularExpressions::ProhibitFixedStringMatches(3pm)
All times are GMT -4. The time now is 06:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy