|
|
Sponsored Content
Top Forums
Programming
awk to count occurrence of strings and loop for multiple columns
Post 302919042 by Don Cragun on Saturday 27th of September 2014 04:20:00 AM
09-27-2014
Quote:
Hi Don,
Thanks for your response. I'm currently using redhat. The line max is about 100k lines, while the number of fields are 100.
Yup, my desired output shall be what you have corrected. I manage to do the count for column by column, but wonder how could i perform a for loop one shot for 100 columns.
Thanks for your response. I'm currently using redhat. The line max is about 100k lines, while the number of fields are 100.
Yup, my desired output shall be what you have corrected. I manage to do the count for column by column, but wonder how could i perform a for loop one shot for 100 columns.
I'm still trying to get a feel for what the input and output data is going to look like. I'm assuming that the 100k number you gave is the number of lines in your input file (not the number of bytes in the longest line in your input file). Assuming that the numbers in the 1st column of your input are unique and that there are 6 distinct values in the other columns (A, G, C, T, D, and E) that means that we are converting 100k input rows with a maximum line length of about 210 bytes each into six output data lines (plus one header line) with a maximum line length approaching (3 * 99 spaces between fields +1 * 99 single letters for the col2 through col100 data + 99 * (6 digits + 1 comma) * 100k col1 values + 99 * 5 digits for the count values) 69.3 million bytes and an average line length approaching 11.5 million bytes.
Am I in the right ballpark here, or are my assumptions off? If my assumptions are off, where am I guessing wrong? Are there values other than A, G, C, T, D, and E that will appear in col2 through col100 in the input file?
Once you have created this file, do you have something that is going to be able to use this data?
|
|
10 More Discussions You Might Find Interesting
1. Linux
I need to find the line count of multiple strings in a particular file. The strings are as follows:
bmgcc
bmgccftp
bsmsftp
bulkftp
cctuneftp
crbtftp
crmpos
cso
gujhr
I am doing manual grep for each of the string to find the line count. The command i am using right now is:
grep mark... (3 Replies)
Discussion started by: salaathi
3 Replies
2. Shell Programming and Scripting
How to grep multiple string occurance in input file using single grep command? I have below input file with many IDP, RRBE messages. Out put should have count of each messages.
I have used below command but it is not working
grep -cH "(sent IDP Request)(Recv RRBCSM)" *.txt ... (5 Replies)
Discussion started by: sushmab82
5 Replies
3. Shell Programming and Scripting
Hi,
I would like to copy some columns from a particular file by mapping with the string names. i am using the .csv file format.
my one file consist of 100 of columns but i want only particular 4 columns such as ( First_name, Middle_name,Last_name & Stlc). but they are listed in many files... (15 Replies)
Discussion started by: dsh007
15 Replies
4. Shell Programming and Scripting
Can anyone help me to count number of occurrence of the strings based on column value. Say i have 300 files with 1000 record length from which i need to count the number of occurrence string which is existing from 213 to 219. Some may be unique and some may be repeated. (8 Replies)
Discussion started by: zooby
8 Replies
5. Shell Programming and Scripting
File 1
aaa
bbb
ccc
File 2
aaa
xxx
zzz
bbb
File 3
aaa
bbb
xxx
Output: (4 Replies)
Discussion started by: Misa-Misa
4 Replies
6. UNIX for Dummies Questions & Answers
Hey everyone,
I have an issue with a client that is passing me a list of values in one column, and occasionally the combination of all the values results in more than an 255 character string. My DB has a 255 character limit, so I am looking to take the column (comma delimited file), and if it... (1 Reply)
Discussion started by: perekl
1 Replies
7. Shell Programming and Scripting
Hi,
I want to count the occurrences of strings in a column and display as in example below:
Input:
get1 345 789 098
get2 567 982 090
fet4 777 610 632
get1 800 544 230
get1 600 788 451
get2 892 321 243
get1 673 111 235
fet3 789 220 278
fet4 768 222 341
output:
4 get1 345 789... (7 Replies)
Discussion started by: aydj
7 Replies
8. UNIX for Dummies Questions & Answers
Hi,
let's say an input looks like:
A|C|C|D
A|C|I|E
A|B|I|C
A|T|I|B
as the title of the thread explains, I am trying to get something like:
1|A=4
2|C=2|B=1|T=1
3|I=3|C=1
4|D=1|E=1|C=1|B=1
i.e. a count of every character in each field (first column of output) independently, sorted... (4 Replies)
Discussion started by: beca123456
4 Replies
9. UNIX for Dummies Questions & Answers
Hello,
I have a table that looks like what is shown below:
AA
BB
CC
XY
PQ
RS
AA
BB
CC
XY
RS
I would like the total counts depending on the set they belong to:
if search pattern is in {AA, BB, CC} --> count them as Type1 | wc -l (3 Replies)
Discussion started by: Gussifinknottle
3 Replies
10. UNIX for Beginners Questions & Answers
Hello, I have two tab files with headers
File1: with 4 columns
header1 header2 header3 header4
44 a bb 1
57 c ab 4
64 d d 5
File2: with 26 columns
header1.. header5 header6 header7 ... header 22...header26
id1 44 a bb
id2 57 ... (6 Replies)
Discussion started by: nans
6 Replies
LEARN ABOUT CENTOS
psc
PSC(1) General Commands Manual PSC(1) NAME
psc - prepare sc files SYNOPSIS
psc [-fLkrSPv] [-s cell] [-R n] [-C n] [-n n] [-d c] DESCRIPTION
Psc is used to prepare data for input to the spreadsheet calculator sc(1). It accepts normal ascii data on standard input. Standard out- put is a sc file. With no options, psc starts the spreadsheet in cell A0. Strings are right justified. All data on a line is entered on the same row; new input lines cause the output row number to increment by one. The default delimiters are tab and space. The column for- mats are set to one larger than the number of columns required to hold the largest value in the column. OPTIONS
-f Omit column width calculations. This option is for preparing data to be merged with an existing spreadsheet. If the option is not specified, the column widths calculated for the data read by psc will override those already set in the existing spreadsheet. -L Left justify strings. -k Keep all delimiters. This option causes the output cell to change on each new delimiter encountered in the input stream. The default action is to condense multiple delimiters to one, so that the cell only changes once per input data item. -r Output the data by row first then column. For input consisting of a single column, this option will result in output of one row with multiple columns instead of a single column spreadsheet. -s cell Start the top left corner of the spreadsheet in cell. For example, -s B33 will arrange the output data so that the spreadsheet starts in column B, row 33. -R n Increment by n on each new output row. -C n Increment by n on each new output column. -n n Output n rows before advancing to the next column. This option is used when the input is arranged in a single column and the spreadsheet is to have multiple columns, each of which is to be length n. -d c Use the single character c as the delimiter between input fields. -P Plain numbers only. A field is a number only when there is no imbedded [-+eE]. -S All numbers are strings. -v Print the version of psc SEE ALSO
sc(1) AUTHOR
Robert Bond PSC 7.16 19 September 2002 PSC(1)