Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Sort and Split file with header and custom name
Top Forums Shell Programming and Scripting Sort and Split file with header and custom name Post 302957048 by Saanvi1 on Tuesday 6th of October 2015 01:11:55 PM
Old 10-06-2015
Sort and Split file with header and custom name
Hi,

I am using SUN SOLARIS (SunOS sun4v sparc SUNW, T5240).
I have a huge data file with header and trailer. This file gets used into an ETL process. ETL skips the header record (which is the first record of the file) and loads the rest of the record. The file can be delimited (comma, tab, pipe) or fixed width.

I am trying to write a script that:
  1. Sorts the records by first field. The value of first field is “Q” or “E”. So all “Q”record should sort first then “E” records.
  2. After the sort above is complete, I want to split the file on every 30,000 records with rules below:
A. The first record in every split file is a header record or blank record- starting with "H"
B. The split file needs to have same name as master file with __(double underscore) and the number.

For e.g:
Master File name is: MasterFile-HP-import-20151006.txt
Split file name needs to be : MasterFile-HP-import-20151006__1.txt ; MasterFile-HP-import-20151006__2.txt and so on.

I tried the two things below for file split:

Code:
split -dl 30000 MasterFile-HP-import-20151006.txt MasterFile-HP-import-20151006__

But getting error "split: illegal option – d error"

Also, tried the solution that I got from internet:
Code:
awk '{filename = " MasterFile-HP-import-20151006__" int((NR-1)/30000) ".txt"; print >> filename}' MasterFile-HP-import-20151006.txt

But getting an error: awk: too many output files 10 . It generates the first 10 files and gives error on 11th file.


I would appreciate any help.


Thanks
Last edited by Saanvi1; 10-06-2015 at 02:40 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split large file and add header and footer to each file

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (1 Reply)
Discussion started by: ashish4422
1 Replies

2. UNIX for Dummies Questions & Answers

Sort a tab file with header.

How to sort a tab delimited file first on col1 and then on col2. Also I need to keep the header intact. file.txt val1 val2 val3 val4 a b c d m n o p e f g h i j k l ... (3 Replies)
Discussion started by: mary271
3 Replies

3. UNIX for Dummies Questions & Answers

Split a file and give custom names

Hello, I want to split a file based on an input list file that contains the lines each split should have + a corresponding file name. #!/bin/sh # sed -n 'start_line_#,end_line_#p' my_input_file > lines_extracted_output_file while read a b c do sed -n '$a,$bp' myLarge.file > $c.split... (2 Replies)
Discussion started by: Gussifinknottle
2 Replies

4. Shell Programming and Scripting

Split and add header and trailer from input file

I need to split the file based on pattern from position 34-37 while retaining the header and trailer records in each individual split file Also is it possible to output the TOM and PAT records in the same output file ? I need the output file names same as xyz_pattern_Datetimestamp.txt ... (23 Replies)
Discussion started by: techedipro
23 Replies

5. UNIX for Dummies Questions & Answers

Sort a las file keep the header as it is

I have several las files with a header and each file start Version and text and before the data starts end up with ~Ascii, then the numbers starts: ------------------------------------------------------------------------- Code: ~Version .....text.... ~Ascii 2 abc 230 1 name 1 abc ... (1 Reply)
Discussion started by: tk2000
1 Replies

6. UNIX for Dummies Questions & Answers

Sort a las file keep the header as it is

I have several las files with a header and each file start Version and text and before the data starts end up with ~Ascii, then the numbers starts: ------------------------------------------------------------------------- ~Version .....text.... ~Ascii 2 abc 230 1 name 1 abc 400 1... (17 Replies)
Discussion started by: tk2000
17 Replies

7. Shell Programming and Scripting

Split a file based on encountering header

I need to split a file based on headers found Input file file1 content: ADD john mickey DROP matt sam output of file F1 john mickey output of file F2 matt sam (5 Replies)
Discussion started by: Diddy
5 Replies

8. Shell Programming and Scripting

Split file by column value, each with header

Hello all, I have a csv with with different testcase values in column 5. year,min,max,Instrument,Testcase 2016,201,1003,GEOTROPH-02116,TATA7980 2016,53,1011,GEOTROPH-01963,TATA7980 2016,3,1024,GEOTROPH-02067,TATA7980 2016,203,1027,GEOTROPH-02011,TATA7980... (16 Replies)
Discussion started by: senhia83
16 Replies

9. UNIX for Beginners Questions & Answers

How do I custom sort the files in a directory using the filenames in a text file.?

Hi all, (5 Replies)
Discussion started by: KMusunuru
5 Replies

10. Shell Programming and Scripting

Sort file data according to a custom list of string

I have a string of pre defined ip address list which will always remain constant their order will never change like in below sample: iplist=8.8.5.19,9.7.5.14,12.9.9.23,8.8.8.14,144.1.113 In the above example i m considering only 5 ips but there could be many more. Now i have a file which... (15 Replies)
Discussion started by: mohtashims
15 Replies

LEARN ABOUT OPENSOLARIS

device_remap

device_remap(1M)					  System Administration Commands					  device_remap(1M)

NAME

       device_remap - administer the Solaris I/O remapping feature

SYNOPSIS

       /usr/platform/sun4v/sbin/device_remap [-v | -R dir]

DESCRIPTION

       Certain	multi-node  sun4v  platforms,  such as T5440 and T5240 servers, have an integrated PCI topology that cause the I/O device paths to
       change in a CPU node failover condition. The device remapping script, device_remap, remaps the device paths in /etc/path_to_inst  file  and
       the symlinks under /dev to match the hardware.

OPTIONS

       The following options are supported:

       -v

	   Displays the /etc/path_to_inst and /dev symlink changes.

       -R dir

	   Perform remapping on the /etc/path_to_inst and /etc/path_to_inst files in the root image at dir.

USAGE

       The  primary  function of device_remap is to remap the device paths in the /etc/path_to_inst file and the symlinks under /dev in a CPU node
       failover condition to match the hardware.

       After adding CPU node(s) or removing CPU node(s), boot the system to the OBP prompt and use the following procedure:

	   1.	  Boot either the failsafe miniroot using: boot -F failsafe , or an install miniroot using boot net -s or similar command.

	   2.	  Mount the root disk as /mnt.

	   3.	  Change directory to the mounted root disk:

		    # cd /mnt

	   4.	  Run device_remap script:

		    # /mnt/usr/platform/sun4v/sbin/device_remap

	   5.	  Boot the system from disk.

       All the error messages are self-explanatory, except for the error message "missing ioaliases node" which means the firmware on  the  system
       does not support device remapping.

EXAMPLES

       Example 1 Displaying Changes Following Failover

       The following command displays the path_to_inst and /dev changes following a CPU node failover.

	 # device_remap -v

       Example 2 Changing Directory Prior to Any Changes

       The following command changes the directory on which the boot image is mounted prior to making any changes.

	 # device_remap -R /newroot

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWkvm.v 		   |
       +-----------------------------+-----------------------------+
       |Interface Stability	     |Unstable			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       boot(1M), attributes(5)

SunOS 5.11							    24 Dec 2008 						  device_remap(1M)
All times are GMT -4. The time now is 07:30 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy