08-28-2017
Are all of your .csv files sorted on field 38? If not, your code won't work. (You will get artificially high counts for the number of distinct values in a file because uniq produces a line of output for each case where a field 38 value changes from the value found on the previous line.)
Does each distinct value in field 38 of all of your files appear in only one or your input files? If not, your code won't work. (You don't have any way to determine which distinct values in a single file appear in one or more of the other files.)
Are you always processing 3 files?
Having 3 files of a megabyte each each should not cause any problem producing a single merged or sorted combined file. Why are you unable to merge them?
Why not just use a single awk script to read all of your files once and produce the output you want for each input file and for the combined input from all of the input files?
Do you really want the number of distinct field 38 values in each input file? Or, do you really just want the number of distinct field 38 values in the merged input files?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern.
Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies
2. UNIX for Dummies Questions & Answers
Ok, another fun hiccup in my UNIX learning curve. I am trying to count the number of occurrences of an IP address across multiple files named example.hits. I can extract the number of occurrences from the files individually but when you use grep -c with multiple files you get the output similar to... (5 Replies)
Discussion started by: MrAd
5 Replies
3. Shell Programming and Scripting
Seems like can use awk and perl command. But I don't have the idea to write the command line. Thanks for all of your advise.
For example, if I have the file whose content are:
Sample 1. ATAGCAGAGGGAGTGAAGAGGTGGTGGGAGGGAGCT
Sample 2. ACTTTTATTTGAATGTAATATTTGGGACAATTATTC
Sample 3.... (1 Reply)
Discussion started by: patrick chia
1 Replies
4. Shell Programming and Scripting
how to count the total number of lines of all the files under a directory using perl script..
I mean if I have 10 files under a directory then I want to count the total number of lines of all the 10 files contain. Please help me in writing a perl script on this. (5 Replies)
Discussion started by: adityam
5 Replies
5. Shell Programming and Scripting
Please advice how can we search for a string say (abc) in multiple files and to get total occurrence of that searched string. (Need number of records that exits in period of time).
File look like this (read as filename.yyyymmdd)
a.20100101
b.20100108
c.20100115
d.20100122
e.20100129... (2 Replies)
Discussion started by: zooby
2 Replies
6. Shell Programming and Scripting
i want to find the no:of occurrences of a word in a file
cat 1.txt
unix script unix script
unix script unix script unix script unix script
unix script unix script unix
unix
script
unix script unix script now i want to find , how many times 'unix' was occurred
please help me
thanks... (6 Replies)
Discussion started by: mahesh1987
6 Replies
7. Shell Programming and Scripting
Hi Guys,
I need to write a script to compare the count of two csv files each having 5 columns.
Everyday a csv file is recived.
Now we need to compare the count of todays csv file with yesterday's csv file and if the total count of records is same in todays csv file and yesterday csv file out... (3 Replies)
Discussion started by: Vivekit82
3 Replies
8. Shell Programming and Scripting
Hi,
Very good wishes to all!
Please help to provide the shell script for generating the record counts in filed wise from the .csv file
My question:
Source file:
Field1 Field2 Field3
abc 12f sLm
1234 hjd 12d
Hyd 34
Chn
My target file should generate the .csv file with the... (14 Replies)
Discussion started by: Kirands
14 Replies
9. Shell Programming and Scripting
Hi,
I have a .dat file with contents like the below:
Input file
============SEQ NO-1: COLUMN1==========
9835619
7152815
============SEQ NO-2: COLUMN2 ==========
7615348
7015548
9373086
============SEQ NO-3: COLUMN3===========
9373086
Expected Output: (I just... (1 Reply)
Discussion started by: MS06
1 Replies
10. UNIX for Beginners Questions & Answers
Hello All,
just wanted to export multiple tables from oracle sql using unix shell script to csv file and the below code is exporting only the first table.
Can you please suggest why? or any better idea?
export FILE="/abc/autom/file/geo_JOB.csv"
Export= `sqlplus -s dev01/password@dEV3... (16 Replies)
Discussion started by: Hope
16 Replies
LAM(1) BSD General Commands Manual LAM(1)
NAME
lam -- laminate files
SYNOPSIS
lam [-f min.max] [-p min.max] [-s sepstring] [-t c] file ...
DESCRIPTION
lam copies the named files side by side onto the standard output. The n-th input lines from the input files are considered fragments of the
single long n-th output line into which they are assembled. The name ``-'' means the standard input, and may be repeated.
Normally, each option affects only the file after it. If the option letter is capitalized it affects all subsequent files until it appears
again uncapitalized. The options are described below.
-f min.max Print line fragments according to the format string min.max, where min is the minimum field width and max the maximum field
width. If min begins with a zero, zeros will be added to make up the field width, and if it begins with a '-', the fragment
will be left-adjusted within the field.
-p min.max Like -f, but pad this file's field when end-of-file is reached and other files are still active.
-s sepstring Print sepstring before printing line fragments from the next file. This option may appear after the last file.
-t c The input line terminator is c instead of a newline. The newline normally appended to each output line is omitted.
To print files simultaneously for easy viewing use pr(1).
EXAMPLES
The command
lam file1 file2 file3 file4
joins 4 files together along each line. To merge the lines from four different files use
lam file1 -S "
" file2 file3 file4
Every 2 lines of a file may be joined on one line with
lam - - < file
and a form letter with substitutions keyed by '@' can be done with
lam -t @ letter changes
SEE ALSO
join(1), pr(1), printf(3)
BSD
December 1, 2001 BSD