Sponsored Content
Top Forums Shell Programming and Scripting Sort and Split file with header and custom name Post 302957048 by Saanvi1 on Tuesday 6th of October 2015 01:11:55 PM
Old 10-06-2015
Sort and Split file with header and custom name

Hi,

I am using SUN SOLARIS (SunOS sun4v sparc SUNW, T5240).
I have a huge data file with header and trailer. This file gets used into an ETL process. ETL skips the header record (which is the first record of the file) and loads the rest of the record. The file can be delimited (comma, tab, pipe) or fixed width.

I am trying to write a script that:
  1. Sorts the records by first field. The value of first field is “Q” or “E”. So all “Q”record should sort first then “E” records.
  2. After the sort above is complete, I want to split the file on every 30,000 records with rules below:
A. The first record in every split file is a header record or blank record- starting with "H"
B. The split file needs to have same name as master file with __(double underscore) and the number.

For e.g:
Master File name is: MasterFile-HP-import-20151006.txt
Split file name needs to be : MasterFile-HP-import-20151006__1.txt ; MasterFile-HP-import-20151006__2.txt and so on.


I tried the two things below for file split:

Code:
split -dl 30000 MasterFile-HP-import-20151006.txt MasterFile-HP-import-20151006__

But getting error "split: illegal option – d error"

Also, tried the solution that I got from internet:
Code:
awk '{filename = " MasterFile-HP-import-20151006__" int((NR-1)/30000) ".txt"; print >> filename}' MasterFile-HP-import-20151006.txt

But getting an error: awk: too many output files 10 . It generates the first 10 files and gives error on 11th file.


I would appreciate any help.


Thanks

Last edited by Saanvi1; 10-06-2015 at 02:40 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split large file and add header and footer to each file

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (1 Reply)
Discussion started by: ashish4422
1 Replies

2. UNIX for Dummies Questions & Answers

Sort a tab file with header.

How to sort a tab delimited file first on col1 and then on col2. Also I need to keep the header intact. file.txt val1 val2 val3 val4 a b c d m n o p e f g h i j k l ... (3 Replies)
Discussion started by: mary271
3 Replies

3. UNIX for Dummies Questions & Answers

Split a file and give custom names

Hello, I want to split a file based on an input list file that contains the lines each split should have + a corresponding file name. #!/bin/sh # sed -n 'start_line_#,end_line_#p' my_input_file > lines_extracted_output_file while read a b c do sed -n '$a,$bp' myLarge.file > $c.split... (2 Replies)
Discussion started by: Gussifinknottle
2 Replies

4. Shell Programming and Scripting

Split and add header and trailer from input file

I need to split the file based on pattern from position 34-37 while retaining the header and trailer records in each individual split file Also is it possible to output the TOM and PAT records in the same output file ? I need the output file names same as xyz_pattern_Datetimestamp.txt ... (23 Replies)
Discussion started by: techedipro
23 Replies

5. UNIX for Dummies Questions & Answers

Sort a las file keep the header as it is

I have several las files with a header and each file start Version and text and before the data starts end up with ~Ascii, then the numbers starts: ------------------------------------------------------------------------- Code: ~Version .....text.... ~Ascii 2 abc 230 1 name 1 abc ... (1 Reply)
Discussion started by: tk2000
1 Replies

6. UNIX for Dummies Questions & Answers

Sort a las file keep the header as it is

I have several las files with a header and each file start Version and text and before the data starts end up with ~Ascii, then the numbers starts: ------------------------------------------------------------------------- ~Version .....text.... ~Ascii 2 abc 230 1 name 1 abc 400 1... (17 Replies)
Discussion started by: tk2000
17 Replies

7. Shell Programming and Scripting

Split a file based on encountering header

I need to split a file based on headers found Input file file1 content: ADD john mickey DROP matt sam output of file F1 john mickey output of file F2 matt sam (5 Replies)
Discussion started by: Diddy
5 Replies

8. Shell Programming and Scripting

Split file by column value, each with header

Hello all, I have a csv with with different testcase values in column 5. year,min,max,Instrument,Testcase 2016,201,1003,GEOTROPH-02116,TATA7980 2016,53,1011,GEOTROPH-01963,TATA7980 2016,3,1024,GEOTROPH-02067,TATA7980 2016,203,1027,GEOTROPH-02011,TATA7980... (16 Replies)
Discussion started by: senhia83
16 Replies

9. UNIX for Beginners Questions & Answers

How do I custom sort the files in a directory using the filenames in a text file.?

Hi all, (5 Replies)
Discussion started by: KMusunuru
5 Replies

10. Shell Programming and Scripting

Sort file data according to a custom list of string

I have a string of pre defined ip address list which will always remain constant their order will never change like in below sample: iplist=8.8.5.19,9.7.5.14,12.9.9.23,8.8.8.14,144.1.113 In the above example i m considering only 5 ips but there could be many more. Now i have a file which... (15 Replies)
Discussion started by: mohtashims
15 Replies
audit(2)							   System Calls 							  audit(2)

NAME
audit - write a record to the audit log SYNOPSIS
cc [ flag ... ] file ... -lbsm -lsocket -lnsl [ library... ] #include <sys/param.h> #include <bsm/libbsm.h> int audit(caddr_t record, int length); DESCRIPTION
The audit() function queues a record for writing to the system audit log. The data pointed to by record is queued for the log after a mini- mal consistency check, with the length parameter specifying the size of the record in bytes. The data should be a well-formed audit record as described by audit.log(4). The kernel validates the record header token type and length, and sets the time stamp value before writing the record to the audit log. The kernel does not do any preselection for user-level generated events. If the audit policy is set to include sequence or trailer tokens, the kernel will append them to the record. RETURN VALUES
Upon successful completion, 0 is returned. Otherwise, -1 is returned and errno is set to indicate the error. ERRORS
The audit() function will fail if: E2BIG The record length is greater than the maximum allowed record length. EFAULT The record argument points outside the process's allocated address space. EINVAL The header token in the record is invalid. ENOTSUP Solaris Audit is not defined for this system. EPERM The {PRIV_PROC_AUDIT} privilege is not asserted in the effective set of the calling process. USAGE
Only privileged processes can successfully execute this call. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Interface Stability |Committed | +-----------------------------+-----------------------------+ |MT-Level |MT-Safe | +-----------------------------+-----------------------------+ SEE ALSO
bsmconv(1M), audit(1M), auditd(1M), svcadm(1M), auditon(2), getaudit(2), audit.log(4), attributes(5), privileges(5) NOTES
The functionality described in this man page is available only if the Solaris Auditing has been enabled and the audit daemon auditd(1M) has not been disabled by audit(1M) or svcadm(1M). See bsmconv(1M) for more information. SunOS 5.11 16 Apr 2008 audit(2)
All times are GMT -4. The time now is 09:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy