Sponsored Content
Full Discussion: Data File Processing Help
Top Forums UNIX for Dummies Questions & Answers Data File Processing Help Post 302179413 by mavsman on Thursday 27th of March 2008 04:15:55 PM
Old 03-27-2008
Data File Processing Help

I need to read contents of directory and create a list of data files that match a certain pattern and process by renaming it and calling a existing .ksh script then archiving off to file another directory. Any suggestions or samples u could point me to on using .ksh perl or other to process through the list of data files? What would you recommend to strip off a suffix from a data file?. tia
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

data processing

hi i am having a file of following kind: 20015#67143645#143123#4214 62014#67143148#67143159#456 15432#67143568#00143862#4632 54112#67143752#0067143657#143 54623#67143357#167215#34531 65446#67143785#143598#7456 75642#67143546#156146#845 24464#67143465#172532#6544... (5 Replies)
Discussion started by: rochitsharma
5 Replies

2. Shell Programming and Scripting

Checking for a control file before processing a data file

Hi All, I am very new to Shell scripting... I got a requirement. I will have few text files(data files) in a particular directory. they will be with .txt extension. With same name, but with a different extension control files also will be there. For example, Sample_20081001.txt is the data... (4 Replies)
Discussion started by: purna.cherukuri
4 Replies

3. Shell Programming and Scripting

How should i know that the process is still processing data

I have some process . How should i know that the process is still processing data or got hanged even though it is showing that it is running in background I know of a command called truss. how should i use this command and determine 1) process is still processing data 2) process got hanged... (7 Replies)
Discussion started by: ali560045
7 Replies

4. Shell Programming and Scripting

Help with data processing, maybe awk

I have a file, first 5 columns are very normal, like "1107",106027,71400,"Y","BIOLOGY",, however, the 6th columns, the user can put comments, anything, just any characters, like new line, double quote, single quote, whatever from the keyboard, like"Please load my previous SOM597G course content in... (3 Replies)
Discussion started by: freelong
3 Replies

5. Programming

Data processing

Hello guys! I have some issue in how to processing some data. I have some files with 3 columns. The 1st column is a name of my sample. The 2nd column is a numerical sequence (very big sequence) starting from "1". And the 3rd column is a feature of each line, represented for a number (completely... (2 Replies)
Discussion started by: bfantinatti
2 Replies

6. UNIX for Dummies Questions & Answers

Genomic data processing

Dear fellow members, I've just joined the forum and am a newbie to shell scripting and programming. I'm stuck on the following problem. I'm working with large scale genomic data and need to do some analyses on it. Essentially it is text processing problem, so please don't mind the scientific... (0 Replies)
Discussion started by: mvaishnav
0 Replies

7. Shell Programming and Scripting

Data processing using awk

Hello, I have some bitrate data in a csv which is in an odd format and is difficult to process in Excel when I have thousands of rows. Therefore, I was thinking of doing this in bash and using awk as the primary application except that due to its complication, I'm a little stuck. ... (24 Replies)
Discussion started by: shadyuk
24 Replies

8. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

9. Shell Programming and Scripting

awk processing of variable number of fields data file

Hy! I need to post-process some data files which have variable (and periodic) number of fields. For example, I need to square (data -> data*data) the folowing data file: -5.34281E-28 -3.69822E-29 8.19128E-29 9.55444E-29 8.16494E-29 6.23125E-29 4.42106E-29 2.94592E-29 1.84841E-29 ... (5 Replies)
Discussion started by: radudownload
5 Replies

10. Shell Programming and Scripting

Data Processing

I have below Data *************************************************** ********************BEGINNING-1******************** directive url is : https://coursera-eu.mokar.com/directives/96df29ff-176a-35f7-8b1b-4ce483d15762 Src urls are :... (8 Replies)
Discussion started by: nikhil jain
8 Replies
File::GlobMapper(3)					User Contributed Perl Documentation				       File::GlobMapper(3)

NAME
File::GlobMapper - Extend File Glob to Allow Input and Output Files SYNOPSIS
use File::GlobMapper qw( globmap ); my $aref = globmap $input => $output or die $File::GlobMapper::Error ; my $gm = new File::GlobMapper $input => $output or die $File::GlobMapper::Error ; DESCRIPTION
This module needs Perl5.005 or better. This module takes the existing "File::Glob" module as a starting point and extends it to allow new filenames to be derived from the files matched by "File::Glob". This can be useful when carrying out batch operations on multiple files that have both an input filename and output filename and the output file can be derived from the input filename. Examples of operations where this can be useful include, file renaming, file copying and file compression. Behind The Scenes To help explain what "File::GlobMapper" does, consider what code you would write if you wanted to rename all files in the current directory that ended in ".tar.gz" to ".tgz". So say these files are in the current directory alpha.tar.gz beta.tar.gz gamma.tar.gz and they need renamed to this alpha.tgz beta.tgz gamma.tgz Below is a possible implementation of a script to carry out the rename (error cases have been omitted) foreach my $old ( glob "*.tar.gz" ) { my $new = $old; $new =~ s#(.*).tar.gz$#$1.tgz# ; rename $old => $new or die "Cannot rename '$old' to '$new': $! ; } Notice that a file glob pattern "*.tar.gz" was used to match the ".tar.gz" files, then a fairly similar regular expression was used in the substitute to allow the new filename to be created. Given that the file glob is just a cut-down regular expression and that it has already done a lot of the hard work in pattern matching the filenames, wouldn't it be handy to be able to use the patterns in the fileglob to drive the new filename? Well, that's exactly what "File::GlobMapper" does. Here is same snippet of code rewritten using "globmap" for my $pair (globmap '<*.tar.gz>' => '<#1.tgz>' ) { my ($from, $to) = @$pair; rename $from => $to or die "Cannot rename '$old' to '$new': $! ; } So how does it work? Behind the scenes the "globmap" function does a combination of a file glob to match existing filenames followed by a substitute to create the new filenames. Notice how both parameters to "globmap" are strings that are delimited by <>. This is done to make them look more like file globs - it is just syntactic sugar, but it can be handy when you want the strings to be visually distinctive. The enclosing <> are optional, so you don't have to use them - in fact the first thing globmap will do is remove these delimiters if they are present. The first parameter to "globmap", "*.tar.gz", is an Input File Glob. Once the enclosing "< ... >" is removed, this is passed (more or less) unchanged to "File::Glob" to carry out a file match. Next the fileglob "*.tar.gz" is transformed behind the scenes into a full Perl regular expression, with the additional step of wrapping each transformed wildcard metacharacter sequence in parenthesis. In this case the input fileglob "*.tar.gz" will be transformed into this Perl regular expression ([^/]*).tar.gz Wrapping with parenthesis allows the wildcard parts of the Input File Glob to be referenced by the second parameter to "globmap", "#1.tgz", the Output File Glob. This parameter operates just like the replacement part of a substitute command. The difference is that the "#1" syntax is used to reference sub-patterns matched in the input fileglob, rather than the $1 syntax that is used with perl regular expressions. In this case "#1" is used to refer to the text matched by the "*" in the Input File Glob. This makes it easier to use this module where the parameters to "globmap" are typed at the command line. The final step involves passing each filename matched by the "*.tar.gz" file glob through the derived Perl regular expression in turn and expanding the output fileglob using it. The end result of all this is a list of pairs of filenames. By default that is what is returned by "globmap". In this example the data structure returned will look like this ( ['alpha.tar.gz' => 'alpha.tgz'], ['beta.tar.gz' => 'beta.tgz' ], ['gamma.tar.gz' => 'gamma.tgz'] ) Each pair is an array reference with two elements - namely the from filename, that "File::Glob" has matched, and a to filename that is derived from the from filename. Limitations "File::GlobMapper" has been kept simple deliberately, so it isn't intended to solve all filename mapping operations. Under the hood "File::Glob" (or for older versions of Perl, "File::BSDGlob") is used to match the files, so you will never have the flexibility of full Perl regular expression. Input File Glob The syntax for an Input FileGlob is identical to "File::Glob", except for the following 1. No nested {} 2. Whitespace does not delimit fileglobs. 3. The use of parenthesis can be used to capture parts of the input filename. 4. If an Input glob matches the same file more than once, only the first will be used. The syntax ~ ~user . Matches a literal '.'. Equivalent to the Perl regular expression . * Matches zero or more characters, except '/'. Equivalent to the Perl regular expression [^/]* ? Matches zero or one character, except '/'. Equivalent to the Perl regular expression [^/]? Backslash is used, as usual, to escape the next character. [] Character class. {,} Alternation () Capturing parenthesis that work just like perl Any other character it taken literally. Output File Glob The Output File Glob is a normal string, with 2 glob-like features. The first is the '*' metacharacter. This will be replaced by the complete filename matched by the input file glob. So *.c *.Z The second is Output FileGlobs take the "*" The "*" character will be replaced with the complete input filename. #1 Patterns of the form /#d/ will be replaced with the Returned Data EXAMPLES
A Rename script Below is a simple "rename" script that uses "globmap" to determine the source and destination filenames. use File::GlobMapper qw(globmap) ; use File::Copy; die "rename: Usage rename 'from' 'to' " unless @ARGV == 2 ; my $fromGlob = shift @ARGV; my $toGlob = shift @ARGV; my $pairs = globmap($fromGlob, $toGlob) or die $File::GlobMapper::Error; for my $pair (@$pairs) { my ($from, $to) = @$pair; move $from => $to ; } Here is an example that renames all c files to cpp. $ rename '*.c' '#1.cpp' A few example globmaps Below are a few examples of globmaps To copy all your .c file to a backup directory '</my/home/*.c>' '</my/backup/#1.c>' If you want to compress all '</my/home/*.[ch]>' '<*.gz>' To uncompress '</my/home/*.[ch].gz>' '</my/home/#1.#2>' SEE ALSO
File::Glob AUTHOR
The File::GlobMapper module was written by Paul Marquess, pmqs@cpan.org. COPYRIGHT AND LICENSE
Copyright (c) 2005 Paul Marquess. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.16.3 2011-01-07 File::GlobMapper(3)
All times are GMT -4. The time now is 12:58 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy