Attach filename to wc results on massive number of files Post: 303032276

Sponsored Content

Top Forums Shell Programming and Scripting Attach filename to wc results on massive number of files Post 303032276 by yifangt on Thursday 14th of March 2019 01:22:04 PM

03-14-2019

Registered User

Attach filename to wc results on massive number of files

Hello,
I have massive number of big files that needed to be counted for the total number of lines (> 100x millions) each. I want the file name attached to the count results so that they are aligned nicely matching name and counts.
I could do each file at a time, which will take hours to finish, so that the jobs were sent to background as I have multiple cores available to get the job done quickly. The problem with my script is the "echo -n $f" "; always accomplishes first, and the ${f}_R1.fq.gz | wc -l part is behind too much and the result was not aligned as expected.

Here is my code:

Code:

for f in $(cat ${LIST1}); do 
echo -n $f" "  >> raw_reads_count.table1; 
zcat ${f}_R1.fq.gz | wc -l >> raw_reads_count.table1 &      #This is the part
 done
------------------------------------------------------------------------------------------------------
messed-up output:
a      
bb    
ccc   
xyz 
267234214
777234211
937214233
1027254258
------------------------------------------------------------------------------------------------------
 Expected output:
a    267234214
bb   937214233
ccc  777234211
xyz 1027254258

How should I improve my script to get what is expected? Thanks a lot!

yifangt

View Public Profile for yifangt

Find all posts by yifangt

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

awk | stop after specified number of results

I am searching some rather large text files using grep and or awk. What I would like to know is if there is a way (either with grep, awk, or realy any other unix tool) to stop the search when a predifined number of results are returned. I would like to do this for speed purpuses. When i get...

2. UNIX for Advanced & Expert Users

pine does'nt attach files

Hello All, I am maintaining a server and I use pine as MUA and sendmail as MTA. Suddenly many users in the network face the problem of not being able to attach files using pine. I checked the sendmail.cf file and found a variable "MaxMessageSize = 1000000". Eventhough the message size...

3. Shell Programming and Scripting

attach 2 files using mailx

if test.dat is the file cat test.dat|uuencode test.dat|mailx -s "subject" mailid can be used for attaching test.dat how can i attach more than one file to a mail using mailx

4. Shell Programming and Scripting

attach multiple files in email

I am trying to send multiple files as attachment in one email, I tried to search but couldn't find. Please let me know if any solutions.

5. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ...

6. Shell Programming and Scripting

How to attach two files in unix script

Hi, My script has to send 2 files as a separate attachment(Note : files to be sent without zipping) to the specified email id. Below code was used but it is not attaching the file as expected instead the file contents are displayed in the body of the email. Kindly,help with your...

7. UNIX for Dummies Questions & Answers

massive tarred grib files totally unacceptable

Hi, I have 7 terabytes of tar files, one for every single day since 1980. Inside these tar files are GRIB files, each with 100+ variables. There's 8 GRIBs in each tar, corresponding to different times of the day. I need 6 friggin variables..., and it takes TWO WEEKS TO EXTRACT ALL THE TAR FILES...

8. Shell Programming and Scripting

counting the number of characters in the filename of all files in a directory?

I am trying to display the output of ls and also print the number of characters in EVERY file name. This is what I have so far: #!/bin/sh for x in `ls`; do echo The number of characters in x | wc -m done Any help appreciated

9. Shell Programming and Scripting

How to count number of results found?

Hi guys, I'm struggling with this one, any help is appreciated. I have File1 with hundreds of unique words, like this: word1 word2 word3 I want to count each word from file1 in file2 and return how many times each word is found. I tried something like this: for i in $(cat file1); do...

10. Shell Programming and Scripting

Adding filename and line number from multiple files to final file

Hi all, I have 20 files (file001.txt upto file020.txt) and I want to read them from 3rd line upto end of file (line 1002). But in the final file they should appear to start from line 1. I need following kind of output in a single file: Filename Line number 2ndcolumn 4thcolumn I...

LEARN ABOUT DEBIAN

vcd2fst

VCD2FST(1)							Filetype Conversion							VCD2FST(1)

NAME

       vcd2fst - Converts VCD files to FST files

SYNTAX

       vcd2fst [option]... [VCDFILE] [FSTFILE]

DESCRIPTION

       Converts VCD files to FST files.

OPTIONS

       -v,--vcdname <filename>
	      Specify VCD input filename.

       -f,--fstname <filename>
	      Specify FST output filename.

       -F,--fastpack
	      Indicates that fastlz should be used instead of gzip for block data.

       -c,--compress
	      Indicates that the entire file should be run through gzip on close.  This results in much smaller files at the expense of a one-time
	      decompression penalty on file open during reads.

       -p,--parallel
	      Indicates that parallel mode should be enabled.  This spawns a worker thread to continue with FST block processing while	conversion
	      continues on the main thread for new FST block data.

       -h,--help
	      Show help screen.

       EXAMPLES

       Note that you should specify dumpfile.vcd directly or use "-" for stdin.

       vcd2fst dumpfile.vcd dumpfile.fst --compress
	      This indicates that the FST file should be post-compressed on close.

AUTHORS

       Anthony Bybell <bybell@rocketmail.com>

SEE ALSO

       fst2vcd(1) vcd2lxt(1) vcd2lxt2(1) lxt2vcd(1) vcd2vzt(1) vzt2vcd(1) gtkwave(1)

Anthony Bybell							      3.3.35								VCD2FST(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

awk | stop after specified number of results

Discussion started by: evan108

2. UNIX for Advanced & Expert Users

pine does'nt attach files

Discussion started by: maybemedic

3. Shell Programming and Scripting

attach 2 files using mailx

Discussion started by: anumkoshy

4. Shell Programming and Scripting

attach multiple files in email

Discussion started by: mgirinath