Sponsored Content
Top Forums Shell Programming and Scripting Finding total distinct count from multiple csv files through UNIX script Post 303002576 by Don Cragun on Monday 28th of August 2017 06:08:53 AM
Old 08-28-2017
As RudiC suggested (and as you well know from being a member of this forum for over 2.5 years), if you don't provide sample input and corresponding desired output from a couple of sample input files, you don't tell us what operating system and shell you're using, and you don't provide a clear specification of what you are trying to do, there is a good chance that any suggestions provided might not work with your data in your environment. The following is completely untested and makes several assumptions that might be wrong...
Code:
awk -F'|' '
!(($38 + 0) in a) {
	a[$38 + 0]
	c++
}
END {	print c
}' *.csv

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

finding duplicate files by size and finding pattern matching and its count

Hi, I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern. Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies

2. UNIX for Dummies Questions & Answers

grep running total/ final total across multiple files

Ok, another fun hiccup in my UNIX learning curve. I am trying to count the number of occurrences of an IP address across multiple files named example.hits. I can extract the number of occurrences from the files individually but when you use grep -c with multiple files you get the output similar to... (5 Replies)
Discussion started by: MrAd
5 Replies

3. Shell Programming and Scripting

How to use the programming in UNIX to count the total G+C and the GC%?What command li

Seems like can use awk and perl command. But I don't have the idea to write the command line. Thanks for all of your advise. For example, if I have the file whose content are: Sample 1. ATAGCAGAGGGAGTGAAGAGGTGGTGGGAGGGAGCT Sample 2. ACTTTTATTTGAATGTAATATTTGGGACAATTATTC Sample 3.... (1 Reply)
Discussion started by: patrick chia
1 Replies

4. Shell Programming and Scripting

perl script on how to count the total number of lines of all the files under a directory

how to count the total number of lines of all the files under a directory using perl script.. I mean if I have 10 files under a directory then I want to count the total number of lines of all the 10 files contain. Please help me in writing a perl script on this. (5 Replies)
Discussion started by: adityam
5 Replies

5. Shell Programming and Scripting

Search and find total count from multiple files

Please advice how can we search for a string say (abc) in multiple files and to get total occurrence of that searched string. (Need number of records that exits in period of time). File look like this (read as filename.yyyymmdd) a.20100101 b.20100108 c.20100115 d.20100122 e.20100129... (2 Replies)
Discussion started by: zooby
2 Replies

6. Shell Programming and Scripting

Finding total count of a word.

i want to find the no:of occurrences of a word in a file cat 1.txt unix script unix script unix script unix script unix script unix script unix script unix script unix unix script unix script unix script now i want to find , how many times 'unix' was occurred please help me thanks... (6 Replies)
Discussion started by: mahesh1987
6 Replies

7. Shell Programming and Scripting

Script to compare count of two csv files

Hi Guys, I need to write a script to compare the count of two csv files each having 5 columns. Everyday a csv file is recived. Now we need to compare the count of todays csv file with yesterday's csv file and if the total count of records is same in todays csv file and yesterday csv file out... (3 Replies)
Discussion started by: Vivekit82
3 Replies

8. Shell Programming and Scripting

Shell script for field wise record count for different Files .csv files

Hi, Very good wishes to all! Please help to provide the shell script for generating the record counts in filed wise from the .csv file My question: Source file: Field1 Field2 Field3 abc 12f sLm 1234 hjd 12d Hyd 34 Chn My target file should generate the .csv file with the... (14 Replies)
Discussion started by: Kirands
14 Replies

9. Shell Programming and Scripting

Help with Getting distinct record count from a .dat file using UNIX command

Hi, I have a .dat file with contents like the below: Input file ============SEQ NO-1: COLUMN1========== 9835619 7152815 ============SEQ NO-2: COLUMN2 ========== 7615348 7015548 9373086 ============SEQ NO-3: COLUMN3=========== 9373086 Expected Output: (I just... (1 Reply)
Discussion started by: MS06
1 Replies

10. UNIX for Beginners Questions & Answers

Export Oracle multiple tables to multiple csv files using UNIX shell scripting

Hello All, just wanted to export multiple tables from oracle sql using unix shell script to csv file and the below code is exporting only the first table. Can you please suggest why? or any better idea? export FILE="/abc/autom/file/geo_JOB.csv" Export= `sqlplus -s dev01/password@dEV3... (16 Replies)
Discussion started by: Hope
16 Replies
rawtopgm(1)						      General Commands Manual						       rawtopgm(1)

NAME
rawtopgm - convert raw grayscale bytes into a portable graymap SYNOPSIS
rawtopgm [-bpp [1|2]] [-littleendian] [-maxval N] [-headerskip N] [-rowskip N] [-tb|-topbottom] [width height] [imagefile] DESCRIPTION
Reads raw grayscale values as input. Produces a PGM file as output. The input file is just a sequence of pure binary numbers, either one or two bytes each, either bigendian or littleendian, representing gray values. They may be arranged either top to bottom, left to right or bottom to top, left to right. There may be arbitrary header information at the start of the file (to which rawtopgm pays no attention at all other than the header's size). Arguments to rawtopgm tell how to interpret the pixels (a function that is served by a header in a regular graphics format). The width and height parameters tell the dimensions of the image. If you omit these parameters, rawtopgm assumes it is a quadratic image and bases the dimensions on the size of the input stream. If this size is not a perfect square, rawtopgm fails. When you don't specify width and height, rawtopgm reads the entire input stream into storage at once, which may take a lot of storage. Otherwise, rawtopgm ordinarily stores only one row at a time. If you don't specify imagefile, or specify -, the input is from Standard Input. The PGM output is to Standard Output. OPTIONS
-maxval N N is the maxval for the gray values in the input, and is also the maxval of the PGM output image. The default is the maximum value that can be represented in the number of bytes used for each sample (i.e. 255 or 65535). -bpp [1|2] tells the number of bytes that represent each sample in the input. If the value is 2, The most significant byte is first in the stream. The default is 1 byte per sample. -littleendian says that the bytes of each input sample are ordered with the least significant byte first. Without this option, rawtopgm assumes MSB first. This obviously has no effect when there is only one byte per sample. -headerskip N rawtopgm skips over N bytes at the beginning of the stream and reads the image immediately after. The default is 0. This is useful when the input is actually some graphics format that has a descriptive header followed by an ordinary raster, and you don't have a program that understands the header or you want to ignore the header. -rowskip N If there is padding at the ends of the rows, you can skip it with this option. Note that rowskip need not be an integer. Amaz- ingly, I once had an image with 0.376 bytes of padding per row. This turned out to be due to a file-transfer problem, but I was still able to read the image. Skipping a fractional byte per row means skipping one byte per multiple rows. -bt -bottomfirst By default, rawtopgm assumes the pixels in the input go top to bottom, left to right. If you specify -bt or -bottomfirst, rawtopgm assumes the pixels go bottom to top, left to right. The Molecular Dynamics and Leica confocal format, for example, use the latter arrangement. If you don't specify -bt when you should or vice versa, the resulting image is upside down, which you can correct with pnmflip . This option causes rawtopgm to read the entire input stream into storage at once, which may take a lot of storage. Ordinarly, raw- topgm stores only one row at a time. For backwards compatibility, rawtopgm also accepts -tb and -topbottom to mean exactly the same thing. The reasons these are named backwards is that the original author thought of it as specifying that the wrong results of assuming the data is top to bottom should be corrected by flipping the result top for bottom. Today, we think of it as simply specifying the format of the input data so that there are no wrong results. SEE ALSO
pgm(5), rawtoppm(1), pnmflip(1) AUTHORS
Copyright (C) 1989 by Jef Poskanzer. Modified June 1993 by Oliver Trepte, oliver@fysik4.kth.se 14 September 2000 rawtopgm(1)
All times are GMT -4. The time now is 04:53 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy