Sponsored Content
Top Forums Shell Programming and Scripting Attach filename to wc results on massive number of files Post 303032281 by bakunin on Thursday 14th of March 2019 02:46:34 PM
Old 03-14-2019
Quote:
Originally Posted by yifangt
The problem with my script is the "echo -n $f" "; always accomplishes first, and the ${f}_R1.fq.gz | wc -l part is behind too much and the result was not aligned as expected.
Actually this is a very interesting problem. It is hard simulate without actually create some terabytes of files that are similar in size to what you have to process, therefore, before i start to actually do that, i'd like to offer a few theories first which you may verify:

my suspicion is that the problem is the buffered nature of <stdout>. From time to time this buffer is flushed and because the output of echo is available already it gets written into the file but since the zcat still runs at that time it will be written at a much later time. Maybe the following might help. I used printf instead of echo, but that is not the point: to execute the output statement the subshell has to be finished, therefore the line should get printed completely or not at all. Because the whole process gets put in background the original order of the filenames will no longer be retained - maybe no concern to you but you should be aware of that.

Another point is the number of processes you start: starting an (in principle unlimited) amount of background processes at the same time is always a bit of an hazard. The script might work well with 10 or 20 files generating 10 or 20 background processes but a directory may as well hold millions of files. No system would survive an attempt to start a million background processes, no matter how small they are and how many processors you have. You may want to implement some logic to only have some maximum number of bround processes running concurrently.

Code:
$(printf "%s\t%s\n" "$f" $(zcat ${f}_R1.fq.gz | wc -l) ) >> raw_reads_count.table1 &

I hope this helps.

bakunin
This User Gave Thanks to bakunin For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

awk | stop after specified number of results

I am searching some rather large text files using grep and or awk. What I would like to know is if there is a way (either with grep, awk, or realy any other unix tool) to stop the search when a predifined number of results are returned. I would like to do this for speed purpuses. When i get... (6 Replies)
Discussion started by: evan108
6 Replies

2. UNIX for Advanced & Expert Users

pine does'nt attach files

Hello All, I am maintaining a server and I use pine as MUA and sendmail as MTA. Suddenly many users in the network face the problem of not being able to attach files using pine. I checked the sendmail.cf file and found a variable "MaxMessageSize = 1000000". Eventhough the message size... (2 Replies)
Discussion started by: maybemedic
2 Replies

3. Shell Programming and Scripting

attach 2 files using mailx

if test.dat is the file cat test.dat|uuencode test.dat|mailx -s "subject" mailid can be used for attaching test.dat how can i attach more than one file to a mail using mailx (2 Replies)
Discussion started by: anumkoshy
2 Replies

4. Shell Programming and Scripting

attach multiple files in email

I am trying to send multiple files as attachment in one email, I tried to search but couldn't find. Please let me know if any solutions. (2 Replies)
Discussion started by: mgirinath
2 Replies

5. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ... (3 Replies)
Discussion started by: natalie23
3 Replies

6. Shell Programming and Scripting

How to attach two files in unix script

Hi, My script has to send 2 files as a separate attachment(Note : files to be sent without zipping) to the specified email id. Below code was used but it is not attaching the file as expected instead the file contents are displayed in the body of the email. Kindly,help with your... (22 Replies)
Discussion started by: meva
22 Replies

7. UNIX for Dummies Questions & Answers

massive tarred grib files totally unacceptable

Hi, I have 7 terabytes of tar files, one for every single day since 1980. Inside these tar files are GRIB files, each with 100+ variables. There's 8 GRIBs in each tar, corresponding to different times of the day. I need 6 friggin variables..., and it takes TWO WEEKS TO EXTRACT ALL THE TAR FILES... (3 Replies)
Discussion started by: sammysoil
3 Replies

8. Shell Programming and Scripting

counting the number of characters in the filename of all files in a directory?

I am trying to display the output of ls and also print the number of characters in EVERY file name. This is what I have so far: #!/bin/sh for x in `ls`; do echo The number of characters in x | wc -m done Any help appreciated (1 Reply)
Discussion started by: LinuxNubBrah
1 Replies

9. Shell Programming and Scripting

How to count number of results found?

Hi guys, I'm struggling with this one, any help is appreciated. I have File1 with hundreds of unique words, like this: word1 word2 word3 I want to count each word from file1 in file2 and return how many times each word is found. I tried something like this: for i in $(cat file1); do... (13 Replies)
Discussion started by: demmel
13 Replies

10. Shell Programming and Scripting

Adding filename and line number from multiple files to final file

Hi all, I have 20 files (file001.txt upto file020.txt) and I want to read them from 3rd line upto end of file (line 1002). But in the final file they should appear to start from line 1. I need following kind of output in a single file: Filename Line number 2ndcolumn 4thcolumn I... (14 Replies)
Discussion started by: bioinfo
14 Replies
ICON(1) 						      General Commands Manual							   ICON(1)

NAME
icon - interpret or compile Icon programs SYNOPSIS
icont [ option ... ] file ... [ -x arg ... ] iconc [ option ... ] file ... [ -x arg ... ] DESCRIPTION
icont and iconc each convert an Icon source program into executable form. icont translates quickly and provides interpretive execution. iconc takes longer to compile but produces programs that execute faster. icont and iconc for the most part can be used interchangeably. This manual page describes both icont and iconc. Where there there are differences in usage between icont and iconc, these are noted. File Names: Files whose names end in .icn are assumed to be Icon source files. The .icn suffix may be omitted; if it is not present, it is supplied. The character - can be used to indicate an Icon source file given in standard input. Several source files can be given on the same command line; if so, they are combined to produce a single program. The name of the executable file is the base name of the first input file, formed by deleting the suffix, if present. stdin is used for source programs given in standard input. Processing: As noted in the synopsis above, icont and iconc accept options followed by file names, optionally followed by -x and arguments. If -x is given, the program is executed automatically and any following arguments are passed to it. icont: The processing performed by icont consists of two phases: translation and linking. During translation, each Icon source file is translated into an intermediate language called ucode. Two ucode files are produced for each source file, with base names from the source file and suffixes .u1 and .u2. During linking, the one or more pairs of ucode files are combined to produce a single icode file. The ucode files are deleted after the icode file is created. Processing by icont can be terminated after translation by the -c option. In this case, the ucode files are not deleted. The names of .u1 files from previous translations can be given on the icont command line. These files and the corresponding .u2 files are included in the linking phase after the translation of any source files. The suffix .u can be used in place of .u1; in this case the 1 is supplied auto- matically. Ucode files that are explicitly named are not deleted. iconc: The processing performed by iconc consists of two phases: code generation and compilation and linking. The code generation phase produces C code, consisting of a .c and a .h file, with the base name of the first source file. These files are then compiled and linked to produce an executable binary file. The C files normally are deleted after compilation and linking. Processing by iconc can be terminated after code generation by the -c option. In this case, the C files are not deleted. OPTIONS
The following options are recognized by icont and iconc: -c Stop after producing intermediate files and do not delete them. -e file Redirect standard error output to file. -f s Enable full string invocation. -o name Name the output file name. -s Suppress informative messages. Normally, both informative messages and error messages are sent to standard error output. -t Arrange for &trace to have an initial value of -1 when the program is executed and for iconc enable debugging features. -u Issue warning messages for undeclared identifiers in the program. -v i Set verbosity level of informative messages to i -E Direct the results of preprocessing to standard output and inhibit further processing. The following additional options are recognized by iconc: -f string Enable features as indicated by the letters in string: a all, equivalent to delns d enable debugging features: display(), name(), variable(), error trace back, and the effect of -f n (see below) e enable error conversion l enable large-integer arithmetic n produce code that keeps track of line numbers and file names in the source code s enable full string invocation -n string Disable specific optimizations. These are indicated by the letters in string: a all, equivalent to cest c control flow optimizations other than switch statement optimizations e expand operations in-line when reasonable (keywords are always put in-line) s optimize switch statements associated with operation invocations t type inference -p arg Pass arg on to the C compiler used by iconc -r path Use the run-time system at path, which must end with a slash. -C prg Have iconc use the C compiler given by prg ENVIRONMENT VARIABLES
When an Icon program is executed, several environment variables are examined to determine certain execution parameters. Values in paren- theses are the default values. BLKSIZE (500000) The initial size of the allocated block region, in bytes. COEXPSIZE (2000) The size, in words, of each co-expression block. DBLIST The location of data bases for iconc to search before the standard one. The value of DBLIST should be a blank-separated string of the form p1 p2 ... pn where the pi name directories. ICONCORE If set, a core dump is produced for error termination. ICONX The location of iconx, the executor for icode files, is built into an icode file when it is produced. This location can be overridden by setting the environment variable ICONX. If ICONX is set, its value is used in place of the location built into the icode file. IPATH The location of ucode files specified in link declarations for icont. IPATH is a blank-separated list of directories. The current directory is always searched first, regardless of the value of IPATH. LPATH The location of source files specified in preprocessor $include directives and in link declarations for iconc. LPATH is otherwise sim- ilar to IPATH. MSTKSIZE (10000) The size, in words, of the main interpreter stack for icont. NOERRBUF By default, &errout is buffered. If this variable is set, &errout is not buffered. QLSIZE (5000) The size, in bytes, of the region used for pointers to strings during garbage collection. STRSIZE (500000) The initial size of the string space, in bytes. TRACE The initial value of &trace. If this variable has a value, it overrides the translation-time -t option. FILES
icont Icon translator iconc Icon compiler iconx Icon executor SEE ALSO
The Icon Programming Language, Ralph E. Griswold and Madge T. Griswold, Prentice-Hall Inc., Englewood Cliffs, New Jersey, Second Edition, 1990. Version 9.1 of Icon, Ralph E. Griswold, Clinton L. Jeffery, and Gregg M. Townsend, IPD267, Department of Computer Science, The University of Arizona, 1995. Version 9 of the Icon Compiler, Ralph E. Griswold, IPD237, Department of Computer Science, The University of Arizona, 1995. icon_vt(1) LIMITATIONS AND BUGS
The icode files for the interpreter do not stand alone; the Icon run-time system (iconx) must be present. Stack overflow is checked using a heuristic that is not always effective. 1 November 1995 IPD244b ICON(1)
All times are GMT -4. The time now is 08:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy