Sponsored Content
Top Forums Shell Programming and Scripting Attach filename to wc results on massive number of files Post 303032349 by yifangt on Friday 15th of March 2019 12:46:01 PM
Old 03-15-2019
parallel to restrict the process number

@bakunin @all
Your comments are exactly what I wanted to catch. Here, I reformed my script with GNU parallel to control the process limits, but I hit another wall:
Code:
parallel -a $LIST1 -j 48 "(printf "%s\t%s\n" {} $(zcat {}_R1.fq.gz | wc -l)) >> raw_reads_count.table1"
------------------------------------------------------
a 0 >> raw_reads_count.table1
bb 0 >> raw_reads_count.table1
ccc 0 >> raw_reads_count.table1
xyz 0 >> raw_reads_count.table1

The problem seems with the parallel placeholder expansion. Is it because of the too many layers of parenthesis () ? Need to get myself familiar with quoting in bash.
Thanks for any help!
======================================================================================
It seems to me this is the final solution:

Code:
parallel -a $LIST1 -j 48 "(echo -n {}' '; (zcat ${RAW_DIR1}/{}_R1.fq.gz | wc -l)) > {}_counts.tmp"
cat *_counts.tmp >> raw_reads_count.table1

Thanks you all for the help!

Last edited by yifangt; 03-15-2019 at 03:36 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

awk | stop after specified number of results

I am searching some rather large text files using grep and or awk. What I would like to know is if there is a way (either with grep, awk, or realy any other unix tool) to stop the search when a predifined number of results are returned. I would like to do this for speed purpuses. When i get... (6 Replies)
Discussion started by: evan108
6 Replies

2. UNIX for Advanced & Expert Users

pine does'nt attach files

Hello All, I am maintaining a server and I use pine as MUA and sendmail as MTA. Suddenly many users in the network face the problem of not being able to attach files using pine. I checked the sendmail.cf file and found a variable "MaxMessageSize = 1000000". Eventhough the message size... (2 Replies)
Discussion started by: maybemedic
2 Replies

3. Shell Programming and Scripting

attach 2 files using mailx

if test.dat is the file cat test.dat|uuencode test.dat|mailx -s "subject" mailid can be used for attaching test.dat how can i attach more than one file to a mail using mailx (2 Replies)
Discussion started by: anumkoshy
2 Replies

4. Shell Programming and Scripting

attach multiple files in email

I am trying to send multiple files as attachment in one email, I tried to search but couldn't find. Please let me know if any solutions. (2 Replies)
Discussion started by: mgirinath
2 Replies

5. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ... (3 Replies)
Discussion started by: natalie23
3 Replies

6. Shell Programming and Scripting

How to attach two files in unix script

Hi, My script has to send 2 files as a separate attachment(Note : files to be sent without zipping) to the specified email id. Below code was used but it is not attaching the file as expected instead the file contents are displayed in the body of the email. Kindly,help with your... (22 Replies)
Discussion started by: meva
22 Replies

7. UNIX for Dummies Questions & Answers

massive tarred grib files totally unacceptable

Hi, I have 7 terabytes of tar files, one for every single day since 1980. Inside these tar files are GRIB files, each with 100+ variables. There's 8 GRIBs in each tar, corresponding to different times of the day. I need 6 friggin variables..., and it takes TWO WEEKS TO EXTRACT ALL THE TAR FILES... (3 Replies)
Discussion started by: sammysoil
3 Replies

8. Shell Programming and Scripting

counting the number of characters in the filename of all files in a directory?

I am trying to display the output of ls and also print the number of characters in EVERY file name. This is what I have so far: #!/bin/sh for x in `ls`; do echo The number of characters in x | wc -m done Any help appreciated (1 Reply)
Discussion started by: LinuxNubBrah
1 Replies

9. Shell Programming and Scripting

How to count number of results found?

Hi guys, I'm struggling with this one, any help is appreciated. I have File1 with hundreds of unique words, like this: word1 word2 word3 I want to count each word from file1 in file2 and return how many times each word is found. I tried something like this: for i in $(cat file1); do... (13 Replies)
Discussion started by: demmel
13 Replies

10. Shell Programming and Scripting

Adding filename and line number from multiple files to final file

Hi all, I have 20 files (file001.txt upto file020.txt) and I want to read them from 3rd line upto end of file (line 1002). But in the final file they should appear to start from line 1. I need following kind of output in a single file: Filename Line number 2ndcolumn 4thcolumn I... (14 Replies)
Discussion started by: bioinfo
14 Replies
SAVEPOINT(7)                                                       SQL Commands                                                       SAVEPOINT(7)

NAME
SAVEPOINT - define a new savepoint within the current transaction SYNOPSIS
SAVEPOINT savepoint_name DESCRIPTION
SAVEPOINT establishes a new savepoint within the current transaction. A savepoint is a special mark inside a transaction that allows all commands that are executed after it was established to be rolled back, restoring the transaction state to what it was at the time of the savepoint. PARAMETERS
savepoint_name The name to give to the new savepoint. NOTES
Use ROLLBACK TO SAVEPOINT [rollback_to_savepoint(7)] to rollback to a savepoint. Use RELEASE SAVEPOINT [release_savepoint(7)] to destroy a savepoint, keeping the effects of commands executed after it was established. Savepoints can only be established when inside a transaction block. There can be multiple savepoints defined within a transaction. EXAMPLES
To establish a savepoint and later undo the effects of all commands executed after it was established: BEGIN; INSERT INTO table1 VALUES (1); SAVEPOINT my_savepoint; INSERT INTO table1 VALUES (2); ROLLBACK TO SAVEPOINT my_savepoint; INSERT INTO table1 VALUES (3); COMMIT; The above transaction will insert the values 1 and 3, but not 2. To establish and later destroy a savepoint: BEGIN; INSERT INTO table1 VALUES (3); SAVEPOINT my_savepoint; INSERT INTO table1 VALUES (4); RELEASE SAVEPOINT my_savepoint; COMMIT; The above transaction will insert both 3 and 4. COMPATIBILITY
SQL requires a savepoint to be destroyed automatically when another savepoint with the same name is established. In PostgreSQL, the old savepoint is kept, though only the more recent one will be used when rolling back or releasing. (Releasing the newer savepoint will cause the older one to again become accessible to ROLLBACK TO SAVEPOINT and RELEASE SAVEPOINT.) Otherwise, SAVEPOINT is fully SQL conforming. SEE ALSO
BEGIN [begin(7)], COMMIT [commit(7)], RELEASE SAVEPOINT [release_savepoint(7)], ROLLBACK [rollback(7)], ROLLBACK TO SAVEPOINT [roll- back_to_savepoint(7)] SQL - Language Statements 2010-05-14 SAVEPOINT(7)
All times are GMT -4. The time now is 02:47 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy