Sponsored Content
Top Forums Shell Programming and Scripting Split files with formatted numbers Post 302907152 by bobbygsk on Thursday 26th of June 2014 09:22:01 AM
Old 06-26-2014
I get 2 to 10 mil records file. I have to split them with 100,000 records in each file. Assuming that i mostly get 3 mil records, so I have to split the file in 300 files. What is the limit that awk can handle certain number of file descriptors.

Besides, how do I get header (n records) and trailer with file number or some content in it.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need to remove improperly formatted fortran output line from files, tried sed

I have been trying to remove some improperly formatted lines of output from fortran code I have been using. The problem is that I have some singularities in the math for some points that causes an incorrectly large value to be reported that exceeds the normal formating set in the code resulting in... (2 Replies)
Discussion started by: gillesc_mac
2 Replies

2. Shell Programming and Scripting

Generating formatted reports from log files

Given that I have a log file of the format: DATE ID LOG_LEVEL | EVENT 2009-07-23T14:05:11Z T-4030097550 D | MessX 2009-07-23T14:10:44Z T-4030097550 D | MessY 2009-07-23T14:34:08Z T-7298651656 D | MessX 2009-07-23T14:41:00Z T-7298651656 D | MessY 2009-07-23T15:05:10Z T-4030097550 D | MessZ... (5 Replies)
Discussion started by: daccad
5 Replies

3. UNIX for Dummies Questions & Answers

Split Function Prefix Numbers

Hello, Hello, I use the following command to split a file: split -Number_of_Lines Input_File MyPrefix_ output is MyPrefix_a MyPrefix_b MyPrefix_c ...... Instead, how can I get numerical values like: MyPrefix_1 MyPrefix_2 MyPrefix_3 ...... (2 Replies)
Discussion started by: Gussifinknottle
2 Replies

4. Shell Programming and Scripting

Extracting formatted text and numbers

Hello, I have a file of text and numbers from which I want to extract certain fields and write it to a new file. I would use awk but unfortunately the input data isn't always formatted into the correct columns. I am using tcsh. For example, given the following data I want to extract: and... (3 Replies)
Discussion started by: DFr0st
3 Replies

5. UNIX for Dummies Questions & Answers

Breaking a fasta formatted file into multiple files containing each gene separately

Hey, I've been trying to break a massive fasta formatted file into files containing each gene separately. Could anyone help me? I've tried to use the following code but i've recieved errors every time: for i in *.rtf.out do awk '/^>/{f=++d".fasta"} {print > $i.out}' $i done (1 Reply)
Discussion started by: Ann Mc Cartney
1 Replies

6. Shell Programming and Scripting

Split a file into multiple files based on line numbers and first column value

Hi All I have one query,say i have a requirement like the below code should be move to diffent files whose maximum lines can be of 10 lines.Say in the below example,it consist of 14 lines. This should be moved logically using the data in the fisrt coloumn to file1 and file 2.The data of first... (2 Replies)
Discussion started by: sarav.shan
2 Replies

7. Shell Programming and Scripting

awk split numbers

I would like to split a string of numbers "1-2,4-13,16,19-20,21-25,31-32" and output these with awk into -dFirstPage=1 -dLastPage=2 file.pdf -dFirstPage=4 -dLastPage=13 file.pdf -dFirstPage=16 -dLastPage=16 file.pdf file.pdf -dFirstPage=19 -dLastPage=20 file.pdf -dFirstPage=21 -dLastPage=25... (3 Replies)
Discussion started by: sdf
3 Replies

8. UNIX for Beginners Questions & Answers

Split and Rename Split Files

Hello, I need to split a file by number of records and rename each split file with actual filename pre-pended with 3 digit split number. What I have tried is the below command with 2 digit numeric value split -l 3 -d abc.txt F (# Will Produce split Files as F00 F01 F02) How to produce... (19 Replies)
Discussion started by: techedipro
19 Replies

9. Shell Programming and Scripting

Sum up formatted numbers with comma separation

I need to sum up the values in field nr 5 in a data file that contains some file listing. The 5th field denotes the size of each file and following are some sample values. 1,775,947,633 4,738 7,300 16,610 15,279 0 0 I tried the following code in a shell script. awk '{sum+=$5} END{print... (4 Replies)
Discussion started by: krishmaths
4 Replies
envelopes(5)							File Formats Manual						      envelopes(5)

NAME
envelopes - sender/recipient lists attached to messages INTRODUCTION
Electronic mail messages are delivered in envelopes. An envelope lists a sender and one or more recipients. Usually these envelope addresses are the same as the addresses listed in the mes- sage header: (envelope) from djb to root From: djb To: root In more complicated situations, though, the envelope addresses may differ from the header addresses. ENVELOPE EXAMPLES
When a message is delivered to several people at different locations, it is first photocopied and placed into several envelopes: (envelope) from djb to root From: djb Copy #1 of message To: root, god@brl.mil (envelope) from djb to god@brl.mil From: djb Copy #2 of message To: root, god@brl.mil When a message is delivered to several people at the same location, the sender doesn't have to photocopy it. He can instead stuff it into one envelope with several addresses; the recipients will make the photocopy: (envelope) from djb to god@brl.mil, angel@brl.mil From: djb To: god@brl.mil, angel@brl.mil, joe, frde Bounced mail is sent back to the envelope sender address. The bounced mail doesn't list an envelope sender, so bounce loops are impossi- ble: (envelope) from <> to djb From: MAILER-DAEMON To: djb Subject: unknown user frde The recipient of a message may make another copy and forward it in a new envelope: (envelope) from djb to joe From: djb Original message To: joe (envelope) from joe to fred From: djb Forwarded message To: joe A mailing list works almost the same way: (envelope) from djb to sos-list From: djb Original message To: sos-list (envelope) from sos-owner to god@brl.mil From: djb Forwarded message To: sos-list to recipient #1 (envelope) from sos-owner to frde From: djb Forwarded message To: sos-list to recipient #2 Notice that the mailing list is set up to replace the envelope sender with something new, sos-owner. So bounces will come back to sos- owner: (envelope) from <> to sos-owner From: MAILER-DAEMON To: sos-owner Subject: unknown user frde It's a good idea to set up an extra address, sos-owner, like this: the original envelope sender (djb) has no way to fix bad sos-list addresses, and of course bounces must not be sent to sos-list itself. HOW ENVELOPE ADDRESSES ARE STORED
Envelope sender and envelope recipient addresses are transmitted and recorded in several ways. When a user injects mail through qmail-inject, he can supply a Return-Path line or a -f option for the envelope sender; by default the envelope sender is his login name. The envelope recipient addresses can be taken from the command line or from various header fields, depending on the options to qmail-inject. Similar comments apply to sendmail. When a message is transferred from one machine to another through SMTP, the envelope sender is given in a MAIL FROM command, the envelope recipients are given in RCPT TO commands, and the message is supplied separately by a DATA command. When a message is delivered by qmail to a single local recipient, qmail-local records the recipient in Delivered-To and the envelope sender in Return-Path. It uses Delivered-To to detect mail forwarding loops. sendmail normally records the envelope sender in Return-Path. It does not record envelope recipient addresses, on the theory that they are redundant: you received the mail, so you must have been one of the envelope recipients. Note that, if the header doesn't have any recipient addresses, sendmail will move envelope recipient addresses back into the header. This situation occurs if all addresses were originally listed as Bcc, since Bcc is automatically removed. When sendmail sees this, it creates a new Apparently-To header field with the envelope recipient addresses. This has the strange effect that each blind-carbon-copy recipient will see a list of all recipients on the same machine. When a message is stored in mbox format, the envelope sender is recorded at the top of the message as a UUCP-style From (no colon) line. Note that this line is less reliable than the Return-Path line added by qmail-local or sendmail. SEE ALSO
qmail-header(5), qmail-local(8), qmail-inject(8) envelopes(5)
All times are GMT -4. The time now is 12:44 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy