Sponsored Content
Top Forums UNIX for Advanced & Expert Users Need optimized awk/perl/shell to give the statistics for the Large delimited file Post 303023382 by neutronscott on Friday 14th of September 2018 11:54:42 PM
Old 09-15-2018
something like:

Code:
awk -F\| '!a[$1]++ { print $1 > "dis_col1.txt"; } !b[$2]++ { print $2 > "dis_col2.txt"; } END { print NR; }' file

This User Gave Thanks to neutronscott For This Post:
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Could someone give me an example of awk accessing array defined in Korn Shell?

As per title and much apprecieated! (2 Replies)
Discussion started by: biglau
2 Replies

2. UNIX for Dummies Questions & Answers

Trim String in 3rd Column in Tab Delimited File...SED/PERL/AWK?

Hey Everybody, I am having much trouble figuring this out, as I am not really a programmer..:mad: Datafile.txt Column0 Column1 Column2 ABC DEF xxxGHI I am running using WGET on a cronjob to grab a datafile, but I need to cut the first three characters from... (6 Replies)
Discussion started by: rickdini
6 Replies

3. Shell Programming and Scripting

Large pipe delimited file that I need to add CR/LF every n fields

I have a large flat file with variable length fields that are pipe delimited. The file has no new line or CR/LF characters to indicate a new record. I need to parse the file and after some number of fields, I need to insert a CR/LF to start the next record. Input file ... (2 Replies)
Discussion started by: clintrpeterson
2 Replies

4. Shell Programming and Scripting

Extracting a portion of data from a very large tab delimited text file

Hi All I wanted to know how to effectively delete some columns in a large tab delimited file. I have a file that contains 5 columns and almost 100,000 rows 3456 f g t t 3456 g h 456 f h 4567 f g h z 345 f g 567 h j k lThis is a very large data file and tab delimited. I need... (2 Replies)
Discussion started by: Lucky Ali
2 Replies

5. Shell Programming and Scripting

Script Optimization - large delimited file, for loop with many greps

Since there are approximately 75K gsfiles and hundreds of stfiles per gsfile, this script can take hours. How can I rewrite this script, so that it's much faster? I'm not as familiar with perl but I'm open to all suggestions. ls file.list>$split for gsfile in `cat $split`; do csplit... (17 Replies)
Discussion started by: verge
17 Replies

6. Shell Programming and Scripting

Awk getting statistics of a grid file,

Hi , I have the following file which is basically a grid (has more than 100000 rows) LLL1 PPP1 LLL1 PPP2 LLL1 PPP3 ............... LLL1 5500 ..... LLL2 PPP1 LLL2 PPP2 LLL2 PPP3 ............... LLL1 5500 ..... L100 PPP1 L100 PPP2 L100 PPP3 ............... 2100 5500... (6 Replies)
Discussion started by: alex2005
6 Replies

7. Shell Programming and Scripting

awk read one delimited file, search another delimited file

Hello folks, I have another doozy. I have two files. The first file has four fields in it. These four fields map to different locations in my second file. What I want to do is read the master file (file 2 - 23 fields) and compare each line against each record in file 1. If I get a match in all four... (4 Replies)
Discussion started by: dagamier
4 Replies

8. Shell Programming and Scripting

Removing dupes within 2 delimited areas in a large dictionary file

Hello, I have a very large dictionary file which is in text format and which contains a large number of sub-sections. Each sub-section starts with the following header : #DATA #VALID 1 and ends with a footer as shown below #END The data between the Header and the Footer consists of... (6 Replies)
Discussion started by: gimley
6 Replies

9. Shell Programming and Scripting

Perl script give answers by file

Hi, I am new in perl. I am running a perl installation script, its asking for paths and so many inputs. Can we provide that info by any file. so i can avoid the interactive installation. (2 Replies)
Discussion started by: Priy
2 Replies
Path::Class(3)						User Contributed Perl Documentation					    Path::Class(3)

NAME
Path::Class - Cross-platform path specification manipulation VERSION
version 0.26 SYNOPSIS
use Path::Class; my $dir = dir('foo', 'bar'); # Path::Class::Dir object my $file = file('bob', 'file.txt'); # Path::Class::File object # Stringifies to 'foo/bar' on Unix, 'fooar' on Windows, etc. print "dir: $dir "; # Stringifies to 'bob/file.txt' on Unix, 'bobfile.txt' on Windows print "file: $file "; my $subdir = $dir->subdir('baz'); # foo/bar/baz my $parent = $subdir->parent; # foo/bar my $parent2 = $parent->parent; # foo my $dir2 = $file->dir; # bob # Work with foreign paths use Path::Class qw(foreign_file foreign_dir); my $file = foreign_file('Mac', ':foo:file.txt'); print $file->dir; # :foo: print $file->as_foreign('Win32'); # foofile.txt # Interact with the underlying filesystem: # $dir_handle is an IO::Dir object my $dir_handle = $dir->open or die "Can't read $dir: $!"; # $file_handle is an IO::File object my $file_handle = $file->open($mode) or die "Can't read $file: $!"; DESCRIPTION
"Path::Class" is a module for manipulation of file and directory specifications (strings describing their locations, like '/home/ken/foo.txt' or 'C:WindowsFoo.txt') in a cross-platform manner. It supports pretty much every platform Perl runs on, including Unix, Windows, Mac, VMS, Epoc, Cygwin, OS/2, and NetWare. The well-known module "File::Spec" also provides this service, but it's sort of awkward to use well, so people sometimes avoid it, or use it in a way that won't actually work properly on platforms significantly different than the ones they've tested their code on. In fact, "Path::Class" uses "File::Spec" internally, wrapping all the unsightly details so you can concentrate on your application code. Whereas "File::Spec" provides functions for some common path manipulations, "Path::Class" provides an object-oriented model of the world of path specifications and their underlying semantics. "File::Spec" doesn't create any objects, and its classes represent the different ways in which paths must be manipulated on various platforms (not a very intuitive concept). "Path::Class" creates objects representing files and directories, and provides methods that relate them to each other. For instance, the following "File::Spec" code: my $absolute = File::Spec->file_name_is_absolute( File::Spec->catfile( @dirs, $file ) ); can be written using "Path::Class" as my $absolute = Path::Class::File->new( @dirs, $file )->is_absolute; or even as my $absolute = file( @dirs, $file )->is_absolute; Similar readability improvements should happen all over the place when using "Path::Class". Using "Path::Class" can help solve real problems in your code too - for instance, how many people actually take the "volume" (like "C:" on Windows) into account when writing "File::Spec"-using code? I thought not. But if you use "Path::Class", your file and directory objects will know what volumes they refer to and do the right thing. The guts of the "Path::Class" code live in the "Path::Class::File" and "Path::Class::Dir" modules, so please see those modules' documentation for more details about how to use them. EXPORT The following functions are exported by default. file A synonym for "Path::Class::File->new". dir A synonym for "Path::Class::Dir->new". If you would like to prevent their export, you may explicitly pass an empty list to perl's "use", i.e. "use Path::Class ()". The following are exported only on demand. foreign_file A synonym for "Path::Class::File->new_foreign". foreign_dir A synonym for "Path::Class::Dir->new_foreign". Notes on Cross-Platform Compatibility Although it is much easier to write cross-platform-friendly code with this module than with "File::Spec", there are still some issues to be aware of. o On some platforms, notably VMS and some older versions of DOS (I think), all filenames must have an extension. Thus if you create a file called foo/bar and then ask for a list of files in the directory foo, you may find a file called bar. instead of the bar you were expecting. Thus it might be a good idea to use an extension in the first place. AUTHOR
Ken Williams, KWILLIAMS@cpan.org COPYRIGHT
Copyright (c) Ken Williams. All rights reserved. This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. SEE ALSO
Path::Class::Dir, Path::Class::File, File::Spec perl v5.16.2 2013-08-25 Path::Class(3)
All times are GMT -4. The time now is 02:19 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy