Sponsored Content
Top Forums Shell Programming and Scripting Attach filename to wc results on massive number of files Post 303032367 by Corona688 on Friday 15th of March 2019 04:12:53 PM
Old 03-15-2019
Parallel is not a go-faster button for files. Unless your CPU is maxing out, there's no benefit.

GNU parallel is just doing individual files like you were doing anyway. It has to, lacking magic mechanisms to predict future filesize and move things where they belong.

If your CPU is maxing out, pigz may work faster one-at-a-time than you were trying to do parallel.

Last edited by Corona688; 03-15-2019 at 05:19 PM..
This User Gave Thanks to Corona688 For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

awk | stop after specified number of results

I am searching some rather large text files using grep and or awk. What I would like to know is if there is a way (either with grep, awk, or realy any other unix tool) to stop the search when a predifined number of results are returned. I would like to do this for speed purpuses. When i get... (6 Replies)
Discussion started by: evan108
6 Replies

2. UNIX for Advanced & Expert Users

pine does'nt attach files

Hello All, I am maintaining a server and I use pine as MUA and sendmail as MTA. Suddenly many users in the network face the problem of not being able to attach files using pine. I checked the sendmail.cf file and found a variable "MaxMessageSize = 1000000". Eventhough the message size... (2 Replies)
Discussion started by: maybemedic
2 Replies

3. Shell Programming and Scripting

attach 2 files using mailx

if test.dat is the file cat test.dat|uuencode test.dat|mailx -s "subject" mailid can be used for attaching test.dat how can i attach more than one file to a mail using mailx (2 Replies)
Discussion started by: anumkoshy
2 Replies

4. Shell Programming and Scripting

attach multiple files in email

I am trying to send multiple files as attachment in one email, I tried to search but couldn't find. Please let me know if any solutions. (2 Replies)
Discussion started by: mgirinath
2 Replies

5. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ... (3 Replies)
Discussion started by: natalie23
3 Replies

6. Shell Programming and Scripting

How to attach two files in unix script

Hi, My script has to send 2 files as a separate attachment(Note : files to be sent without zipping) to the specified email id. Below code was used but it is not attaching the file as expected instead the file contents are displayed in the body of the email. Kindly,help with your... (22 Replies)
Discussion started by: meva
22 Replies

7. UNIX for Dummies Questions & Answers

massive tarred grib files totally unacceptable

Hi, I have 7 terabytes of tar files, one for every single day since 1980. Inside these tar files are GRIB files, each with 100+ variables. There's 8 GRIBs in each tar, corresponding to different times of the day. I need 6 friggin variables..., and it takes TWO WEEKS TO EXTRACT ALL THE TAR FILES... (3 Replies)
Discussion started by: sammysoil
3 Replies

8. Shell Programming and Scripting

counting the number of characters in the filename of all files in a directory?

I am trying to display the output of ls and also print the number of characters in EVERY file name. This is what I have so far: #!/bin/sh for x in `ls`; do echo The number of characters in x | wc -m done Any help appreciated (1 Reply)
Discussion started by: LinuxNubBrah
1 Replies

9. Shell Programming and Scripting

How to count number of results found?

Hi guys, I'm struggling with this one, any help is appreciated. I have File1 with hundreds of unique words, like this: word1 word2 word3 I want to count each word from file1 in file2 and return how many times each word is found. I tried something like this: for i in $(cat file1); do... (13 Replies)
Discussion started by: demmel
13 Replies

10. Shell Programming and Scripting

Adding filename and line number from multiple files to final file

Hi all, I have 20 files (file001.txt upto file020.txt) and I want to read them from 3rd line upto end of file (line 1002). But in the final file they should appear to start from line 1. I need following kind of output in a single file: Filename Line number 2ndcolumn 4thcolumn I... (14 Replies)
Discussion started by: bioinfo
14 Replies
wc(1)							      General Commands Manual							     wc(1)

NAME
wc - Counts the lines, words, characters, and bytes in a file SYNOPSIS
wc [-c | -m] [-lw] [file...] The wc command counts the lines, words, and characters or bytes in a file, or in the standard input if you do not specify any files, and writes the results to standard output. It also keeps a total count for all named files. STANDARDS
Interfaces documented on this reference page conform to industry standards as follows: wc: XCU5.0 Refer to the standards(5) reference page for more information about industry standards and associated tags. OPTIONS
Counts bytes in the input. Counts lines in the input. Counts characters in the input. Counts words in the input. OPERANDS
Specifies the pathname of the input file. If this operand is omitted, standard input is used. DESCRIPTION
A word is defined as a string of characters delimited by white space as defined in the X/Open Base Definitions for XCU4. The wc command counts lines, words, and bytes by default. Use the appropriate options to limit wc output. Specifying wc without options is the equivalent of specifying wc -lwc. If any options are specified, only the requested information is output. The order in which counts appear in the output line is lines, words, bytes. If an option is omitted, then the corresponding field in the output is omitted. If the -m option is used, then character counts replace byte counts. When you specify one or more files, wc displays the names of the files along with the counts. If standard input is used, then no file name is displayed. EXIT STATUS
The following exit values are returned: Successful completion. An error occurred. EXAMPLES
To display the number of lines, words, and bytes in the file text, enter: wc text This results in the following output: 27 185 722 text The numbers 27, 185, and 722 are the number of lines, words, and bytes, respectively, in the file text. To display only one or two of the three counts include the appropriate options. For example, the following command displays only line and byte counts: wc -cl text 27 722 text To count lines, words, and bytes in more than one file, use wc with more than one input file or with a file name pat- tern. For example, the following command can be issued in a directory containing the files text, text1, and text2: wc -l text* 27 text 112 text1 5 text2 144 total The numbers 27, 112, and 5 are the numbers of lines in the files text, text1, and text2, respectively, and 144 is the total number of lines in the three files. The file name is always appended to the output. To obtain a pure number for things like reporting purposes, pipe all input to the wc command using cat. For example, the following command will report the total count of characters in all files in a directory. echo There are `cat *.c | wc -c` characters in *.c files There are 1869 characters in *.c files ENVIRONMENT VARIABLES
The following environment variables affect the execution of wc: Provides a default value for the internationalization variables that are unset or null. If LANG is unset or null, the corresponding value from the default locale is used. If any of the internationalization vari- ables contain an invalid setting, the utility behaves as if none of the variables had been defined. If set to a non-empty string value, overrides the values of all the other internationalization variables. Determines the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multibyte characters in arguments and input files) and which characters are defined as white space characters. Determines the locale for the format and contents of diagnostic messages written to standard error and informative messages written to standard output. Determines the location of message catalogues for the processing of LC_MESSAGES. SEE ALSO
Commands: cksum(1), ls(1) Standards: standards(5) wc(1)
All times are GMT -4. The time now is 03:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy