perform 3 awk commands to multiple files in multiple directories
Hi,
I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them.
Each of these text files contains records (where this record is unique and there are no duplicates) and i want to initially separate each of these records into its own file and name it based on the second field (where the $2 field is an identifier and have this form : cust_xxx_yyy of the record...I'm currently doing this (example for file hour_1/hour_1.txt) :
(1)
which results to multiple .txt files starting with cust_
then i want to have all these files as a single column file, therefore i do this:
(2)
and finally i want to remove the first 3 records of the newly created files thus i do the following:
(3)
I know that there might be an easier way of doing this even for a single directory, but is there a way to write a universal script and perform these 3 commands in all the directories?
Thanks in advance!
Last edited by Franklin52; 10-27-2011 at 03:11 AM..
Reason: Please use code tags, thank you
How do I perform a command to multiple files? For example, I want to look at all files in a directory and print the ones that do not contain a certain string. How do I go about doing this? (4 Replies)
I have this Korn script that I wrote (with some help) that is run by cron. I basically watches a file system for a specific filename to be uploaded (via FTP), checks to make sure that the file is no longer being uploaded (by checking the files size), then runs a series of other scripts. The... (2 Replies)
Hi,
filenames:
contains name of list of files to search in.
placelist
contains the names of places to be searched in all files in "filenames"
for i in $(<filenames)
do
egrep -f placelist $i
if ]
then
echo $i
fi
done >> outputfile
Output i am getting: (0 Replies)
Hello,
I am trying to write a bash shell script that does the following:
1.Finds all *.txt files within my directory of interest
2. reads each of the files (25 files) one by one (tab-delimited format and have the same data format)
3. skips the first 10 rows of the file
4. extracts and... (4 Replies)
Hi guys,
say I have a few files in a directory (58 text files or somthing)
each one contains mulitple strings that I wish to replace with other strings
so in these 58 files I'm looking for say the following strings:
JAM (replace with BUTTER)
BREAD (replace with CRACKER)
SCOOP (replace... (19 Replies)
Hi,
I'd like to process multiple files. For example:
file1.txt
file2.txt
file3.txt
Each file contains several lines of data. I want to extract a piece of data and output it to a new file.
file1.txt ----> newfile1.txt
file2.txt ----> newfile2.txt
file3.txt ----> newfile3.txt
Here is... (3 Replies)
I have multiple files that starts as TRADE_LOG spread across multiple folders in the given structure..
./dir1/1/TRADE_LOG*.gz
./dir2/10/TRADE_LOG*.gz
./dir11/12/TRADE_LOG*.gz
./dir12/13/TRADE_LOG*.gz
when I do ftp uisng mput from the "." dir I am getting the below given error
mput... (1 Reply)
Hi,
Very unfamiliar with unix/linux stuff. Our admin is on vacation so, need help very quickly.
I have directories (eg 40001, 40002, etc) that each have one subdirectory (01).
Each subdir 01 has multiple subdirs (001, 002, 003, etc). They are same in each dir.
I need to keep the top and... (7 Replies)
Hi!
I'm new in awk and I need some help.
I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files.
Thanks in advance for help!
:-)
ciao (5 Replies)
Hello all. Newbie here.
In a directory, I have 50 files and one additional file that is a list of the names of the 50 files. I would like to create a directory for each of the 50 files, and I need the 50 directory names to correspond to the 50 file names. I know this can be done by running... (6 Replies)
Discussion started by: Zeckendorff
6 Replies
LEARN ABOUT REDHAT
dos2unix
dos2unix(1) General Commands Manual dos2unix(1)NAME
dos2unix - DOS/MAC to UNIX text file format converter
SYNOPSYS
dos2unix [options] [-c convmode] [-o file ...] [-n infile outfile ...]
Options:
[-hkqV] [--help] [--keepdate] [--quiet] [--version]
DESCRIPTION
This manual page documents dos2unix, the program that converts plain text files in DOS/MAC format to UNIX format.
OPTIONS
The following options are available:
-h --help
Print online help.
-k --keepdate
Keep the date stamp of output file same as input file.
-q --quiet
Quiet mode. Suppress all warning and messages.
-V --version
Prints version information.
-c --convmode convmode
Sets conversion mode. Simulates dos2unix under SunOS.
-o --oldfile file ...
Old file mode. Convert the file and write output to it. The program default to run in this mode. Wildcard names may be used.
-n --newfile infile outfile ...
New file mode. Convert the infile and write output to outfile. File names must be given in pairs and wildcard names should NOT be
used or you WILL lost your files.
EXAMPLES
Get input from stdin and write output to stdout.
dos2unix
Convert and replace a.txt. Convert and replace b.txt.
dos2unix a.txt b.txt
dos2unix -o a.txt b.txt
Convert and replace a.txt in ASCII conversion mode. Convert and replace b.txt in ISO conversion mode. Convert c.txt from Mac to Unix
ascii format.
dos2unix a.txt -c iso b.txt
dos2unix -c ascii a.txt -c iso b.txt
dos2unix -c mac a.txt b.txt
Convert and replace a.txt while keeping original date stamp.
dos2unix -k a.txt
dos2unix -k -o a.txt
Convert a.txt and write to e.txt.
dos2unix -n a.txt e.txt
Convert a.txt and write to e.txt, keep date stamp of e.txt same as a.txt.
dos2unix -k -n a.txt e.txt
Convert and replace a.txt. Convert b.txt and write to e.txt.
dos2unix a.txt -n b.txt e.txt
dos2unix -o a.txt -n b.txt e.txt
Convert c.txt and write to e.txt. Convert and replace a.txt. Convert and replace b.txt. Convert d.txt and write to f.txt.
dos2unix -n c.txt e.txt -o a.txt b.txt -n d.txt f.txt
DIAGNOSTICS BUGS
The program does not work properly under MSDOS in stdio processing mode. If you know why is that so, please tell me.
AUTHORS
Benjamin Lin - <blin@socs.uts.edu.au>
Bernd Johannes Wuebben (mac2unix mode) <wuebben@kde.org>
MISCELLANY
Tested environment:
Linux 1.2.0 with GNU C 2.5.8
SunOS 4.1.3 with GNU C 2.6.3
MS-DOS 6.20 with Borland C++ 4.02
Suggestions and bug reports are welcome.
SEE ALSO unix2dos(1)mac2unix(1)1995.03.31 dos2unix v3.0 dos2unix(1)