10-28-2009
File transformation - what is most efficient method
I've done quite a bit of searching on this but cannot seem to find exactly what I'm looking for. Say I have a | delimited input file with 6 columns and I need to change the value of a few columns and create an output file. With my limited knowledge I can do this with many lines of code but want some expert opinions on what would be the most efficient as I'll be working with some large files.
Example input row:
12345|employee1|customer2|d. gibbins|20091028|10000
column1 - leave as is
column2 - if employee is in lookup1.txt, set to Y else N
column3 - if customer is in lookup2.txt, set to Y else N
column4 - upcase
column5 - change date format to MM/DD/YYYY
column6 - add explicit decimal
Example output row:
12345|Y|N|D. GIBBINS|10/28/2009|100.00
I'm not looking for exact syntax, but a general idea of commands you would use and/or workflow.
Thanks!
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I'm writing a script that at one point needs to check the contents of another script's log file to determine how to proceed. An example record from the log file is:
"mcref04152006","060417","ANTH0415","282","272","476,983.37","465,268.44","loaded"
I want my script to return this record if:
... (3 Replies)
Discussion started by: Glenn Arndt
3 Replies
2. Shell Programming and Scripting
I've got a multitude of text data files that carry exactly the same kind of data. Unfortunately some of them have a different filename format
some are: 'category'_'month'-'year'_act.txt
an example being: daf_Apr-1961_act.txt
and some are: 'category'_ 'year'-'month'_act.txt
an... (16 Replies)
Discussion started by: vrms
16 Replies
3. UNIX for Dummies Questions & Answers
I need to put together a RAID1 file server for use by Windoze systems. I've built zillions of windows systems from components. I was a HPUX SE for a long time at HP, but have been out of the game for years.
I've got an old workhorse mobo FIC PA-2013 with a 450 MHz K6 III+ I could use, but I'd... (2 Replies)
Discussion started by: pcmacd
2 Replies
4. Shell Programming and Scripting
Hi all, I have to transform a XML file like this:
<?xml version="1.0"?>
<vocabulary>
<voc_id>102</voc_id>
<name>Vocabulary Name</name>
<description>Voc description</description>
<relations>3</relations>
<hierarchy>5</hierarchy>
<word>
<word_id>1</word_id>
... (1 Reply)
Discussion started by: aLittleBeat
1 Replies
5. UNIX for Dummies Questions & Answers
I am having a file, around 500 lines. which contains one letter words, two letters words,...and so on(up to 15 letter words and words are not seprated by line). I need to compare all 1 letter words with 3,4,5 and 6 letters word, all 2 letters words with 2,3,4 and 5 letters words and all 3 letters... (3 Replies)
Discussion started by: akhay_ms
3 Replies
6. Homework & Coursework Questions
Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted!
1. The problem statement, all variables and given/known data:
Write a template main.c file via shell script to make it easier for yourself later.
The issue here isn't writing... (2 Replies)
Discussion started by: george3isme
2 Replies
7. UNIX for Dummies Questions & Answers
Hi Gurus!
I need to make some file transformations. Please help.
This is my input file. It has four columns with fixed width.
1 aaa bbbb cccc
2 eee dddd jjjj
3 fff gggg jjjj
4 hhh iiii cccc
5 kkk llll cccc
6 mmm nnnn oooo
7 ppp qqqq xxxx
8 rrr ... (1 Reply)
Discussion started by: kokoro
1 Replies
8. Shell Programming and Scripting
Hi,
I am trying to populate an array with data from a text file. I have a working method using awk but it is too slow and inefficent. See below.
The text file has 70,000 lines. As awk is a line editor it reads each line of the file until it gets to the required line and then processes it.... (3 Replies)
Discussion started by: carlr
3 Replies
9. Shell Programming and Scripting
Hi,
I was hoping someone could suggest an alternative to code I currently have as mine takes up far too much processor time and it to slow.
The situation:
I have a programme that runs on some files just before they are zipped up and archived, the program appends a one line summary of the... (4 Replies)
Discussion started by: RECrerar
4 Replies
10. Shell Programming and Scripting
I have a list of files defined in a single file , one on each line.(No.of files may wary each time)
eg. content of ETL_LOOKUP.dat
/data/project/randomname
/data/project/ramname
/data/project/raname
/data/project/radomname
/data/project/raame
/data/project/andomname
size of these... (5 Replies)
Discussion started by: h0x0r21
5 Replies
LEARN ABOUT REDHAT
dos2unix
dos2unix(1) General Commands Manual dos2unix(1)
NAME
dos2unix - DOS/MAC to UNIX text file format converter
SYNOPSYS
dos2unix [options] [-c convmode] [-o file ...] [-n infile outfile ...]
Options:
[-hkqV] [--help] [--keepdate] [--quiet] [--version]
DESCRIPTION
This manual page documents dos2unix, the program that converts plain text files in DOS/MAC format to UNIX format.
OPTIONS
The following options are available:
-h --help
Print online help.
-k --keepdate
Keep the date stamp of output file same as input file.
-q --quiet
Quiet mode. Suppress all warning and messages.
-V --version
Prints version information.
-c --convmode convmode
Sets conversion mode. Simulates dos2unix under SunOS.
-o --oldfile file ...
Old file mode. Convert the file and write output to it. The program default to run in this mode. Wildcard names may be used.
-n --newfile infile outfile ...
New file mode. Convert the infile and write output to outfile. File names must be given in pairs and wildcard names should NOT be
used or you WILL lost your files.
EXAMPLES
Get input from stdin and write output to stdout.
dos2unix
Convert and replace a.txt. Convert and replace b.txt.
dos2unix a.txt b.txt
dos2unix -o a.txt b.txt
Convert and replace a.txt in ASCII conversion mode. Convert and replace b.txt in ISO conversion mode. Convert c.txt from Mac to Unix
ascii format.
dos2unix a.txt -c iso b.txt
dos2unix -c ascii a.txt -c iso b.txt
dos2unix -c mac a.txt b.txt
Convert and replace a.txt while keeping original date stamp.
dos2unix -k a.txt
dos2unix -k -o a.txt
Convert a.txt and write to e.txt.
dos2unix -n a.txt e.txt
Convert a.txt and write to e.txt, keep date stamp of e.txt same as a.txt.
dos2unix -k -n a.txt e.txt
Convert and replace a.txt. Convert b.txt and write to e.txt.
dos2unix a.txt -n b.txt e.txt
dos2unix -o a.txt -n b.txt e.txt
Convert c.txt and write to e.txt. Convert and replace a.txt. Convert and replace b.txt. Convert d.txt and write to f.txt.
dos2unix -n c.txt e.txt -o a.txt b.txt -n d.txt f.txt
DIAGNOSTICS
BUGS
The program does not work properly under MSDOS in stdio processing mode. If you know why is that so, please tell me.
AUTHORS
Benjamin Lin - <blin@socs.uts.edu.au>
Bernd Johannes Wuebben (mac2unix mode) <wuebben@kde.org>
MISCELLANY
Tested environment:
Linux 1.2.0 with GNU C 2.5.8
SunOS 4.1.3 with GNU C 2.6.3
MS-DOS 6.20 with Borland C++ 4.02
Suggestions and bug reports are welcome.
SEE ALSO
unix2dos(1) mac2unix(1)
1995.03.31 dos2unix v3.0 dos2unix(1)