01-14-2014
Standardization of input source data files using shell script
Hi there,
I'm a newbie in unix and am fishing for options related to how raw input data files are handled. The scenario, as I'm sure y'all must be very familiar with, is this : we receive upwards of 50 data files in ASCII format from various source systems - now each file has its own structure (columns, datatypes etc) as well as certain "impurities" e.g. leading/trailing whitespaces, junk characters (produced during conversion from mainframe data to ASCII) etc...there is a need to 'sanitize' these files i.e. strip them of whitespaces, junk characters etc - how do we do this.....
Ideally, we would like to have a common shell script that parses each input file and produces a clean version (is this possible? will I need to have multiple shell scripts , one for each file?)
Can you please provide feedback based on your experience...
Thanks
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi Team,
please write a shell script.
It is for veritas netbackup logs. The result we currently have is a single file for each day's backups. we will keep the files in the directory and the file names are like below mentioned.
example :/opt/openv/netbackup/reports/Daily/NB_success*. The No... (6 Replies)
Discussion started by: rvrao77
6 Replies
2. Shell Programming and Scripting
I need to write a shell script to transfer the files every hour from source - target server.
The cron job should be running every hour and shouldn't copy already copied files to the remote server ?
I was able to write intial script but not able to get the logic for (in the next run it should... (12 Replies)
Discussion started by: radhirk
12 Replies
3. Shell Programming and Scripting
hello members,
I have some data ( seperated by semicolon ) with close to 240 rows in a text file temp1.
temp2.txt stores 204 rows of data ( seperated by semicolon ).
I want to :
Sort the data in both files by field1.i.e first data field in every row.
compare the data in both files and print... (6 Replies)
Discussion started by: novice82
6 Replies
4. Shell Programming and Scripting
Hi
I am having a question where I have to
1) Identify the number of files in a directory with a specific format
and if the count is >1 we need to concatenate those two files into one file and remember that in the second file the header should not be copied. it should be form first file.... (4 Replies)
Discussion started by: pradkumar
4 Replies
5. Shell Programming and Scripting
Hi,
I want to process all input files available into folder (C:\ShellPrg\InputFile\) Input files are abc.CSV , XYZ.zip (zip of CSV file), PQR.gz (zip of CSV file).
I want to check the extension of file, If its .zip/.gz then need to unzip the file as .CSV
I want to parse line by line of... (2 Replies)
Discussion started by: Poonamol
2 Replies
6. Shell Programming and Scripting
HI all,
I want to script where all the server names will be in a text file like
server1
server2
server3 . and the script should take servernames from a text file and perform copy of files if the files are not present on those servers.after which it should take next servername till the end of... (0 Replies)
Discussion started by: joseph.dmello
0 Replies
7. UNIX for Dummies Questions & Answers
I have multiple input files that I want to manipulate using a shell script. The files are called 250.1 through 250.1000 but I only want the script to manipulate 250.300 through 250.1000. Before I was using the following script to manipulate the text files:
for i in 250.*; do
|| awk... (4 Replies)
Discussion started by: evelibertine
4 Replies
8. Shell Programming and Scripting
Hi,
I am new to shell scripting.Please help me on this.I am using solaris 10 OS and shell i am using is
# echo $0
-sh
My requirement is i have source file say makefile.I need to extract files with extensions (.c |.cxx |.h |.hxx |.sc) from the makefile.after doing so i need to check whether... (13 Replies)
Discussion started by: muraliinfy04
13 Replies
9. UNIX for Dummies Questions & Answers
Dear experts,
I am new to linux programming. I have a shell script which i should run it on all my samples.
I only define input and out put for this script. The inputs are 3 numbers(coordination numbers) which are available in a series of text file.
Since i have a lots of samples, it takes a... (5 Replies)
Discussion started by: mohamadreza
5 Replies
10. Shell Programming and Scripting
i have written my shell script in notepad however i am struggling to pass the data file to be read to the script the data file is of .txt format. My target is to run the shell script from the terminal and pass 3 arguments e.g. polg@DESKTOP-BVPDC5C:~/CS1420/coursework$ bash valsplit.sh input.txt... (11 Replies)
Discussion started by: Gurdza32
11 Replies
LEARN ABOUT DEBIAN
tmx2tmx
TMX2TMX(1p) User Contributed Perl Documentation TMX2TMX(1p)
NAME
tmx2tmx - utility to convert and filter TMX files
SYNOPSYS
tmx2tmx -cat file1.tmx ... filen.tmx > file.tmx
tmx2tmx -toTrados file1.tmx > file2.tmx
tmx2tmx -clean file1.tmx > file2.tmx
tmx2tmx -select=PT,EN multilingual.tmx > pt-en.tmx
DESCRIPTION
This utility processes TMX documents and return TMX documents. Tasks done with this utility include conversion between TMX versions and TMX
cleaning.
TRADOS conversion
As you maybe know, TRADOS is a company producing computer software for translators. It includes WorkBench which imports TMX files.
Unfortunately, the version I used do not import TMX version 1.4.
This process is done using the switch "-toTrados":
tmx2tmx -toTrados file.tmx > trados.tmx
TMX Cleaning
Specially when translation memories are created from anotated text, or extracted directly from the Internet using any automatic method.
This switch is used to remove junk in translation units.
This option tries to remove junk from TMX files, like empty pairs where one of the sides is empty, or removing other junk type.
Use it this way:
tmx2tmx -clean file.tmx > file2.tmx
Concatenating TMX
tmx2tmx -cat file1.tmx ... filen.tmx > file.tmx
ls | grep '.tmx$' | tmx2tmx -cat > file.tmx
Select languages
Select a bunch of languages:
tmx2tmx -select=PT,EN,FR huge.tmx > pt-en-fr.tmx
SEE ALSO
tmx2html, po2tmx, XML::TMX
AUTHOR
Alberto Simo~es, <albie@alfarrabio.di.uminho.pt>
COPYRIGHT AND LICENSE
Copyright 2004 by Projecto Natura
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
perl v5.14.2 2012-06-05 TMX2TMX(1p)