Sponsored Content
Operating Systems HP-UX Need to split a large data file using a Unix script Post 302069842 by SAIK on Wednesday 29th of March 2006 01:54:23 PM
Old 03-29-2006
Need to split a large data file using a Unix script

Greetings all:
I am still new to Unix environment and I need help with the following requirement.

I have a large sequential file sorted on a field (say store#) that is being split into several smaller files, one for each store. That means if there are 500 stores, there will be 500 files. This is being done using a SQR program right now. How is this done using a Unix script? Any Pseudocode will be appreciated.

In the below example, the first two records are written to a file and when there's a change in the store#, it writes to an other file. The names of the files are lgXXX where XXX is the store number (i.e, lg002, lg003 and so on).


Format of the input file:
Store# City ZIP
--------------------
002 XXX 01601 ..> written to lg002 file
002 YYY 01601 ..> written to lg002 file
003 AAA 11111 ..> written to lg003 file
004 BBB 11222 ..> written to lg004 file
:
:
:
555 XYZ 99999 ..> written to lg555 file

Thank you!
SaiK
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split A Large File

Hi, I have a large file(csv format) that I need to split into 2 files. The file looks something like Original_file.txt first name, family name, address a, b, c, d, e, f, and so on for over 100,00 lines I need to create two files from this one file. The condition is i need to ensure... (4 Replies)
Discussion started by: nbvcxzdz
4 Replies

2. Shell Programming and Scripting

Split a large file with patterns and size

Hi, I have a large file with a repeating pattern in it. Now i want the file split into the block of patterns with a specified no. of lines in each file. i.e. The file is like 1... 2... 2... 3... 1... 2... 3... 1... 2... 2... 2... 2... 2... 3... where 1 is the start of the block... (5 Replies)
Discussion started by: sudhamacs
5 Replies

3. Shell Programming and Scripting

Split Large File

HI, i've to split a large file which inputs seems like : Input file name_file.txt 00001|AAAA|MAIL|DATEOFBIRTHT|....... 00001|AAAA|MAIL|DATEOFBIRTHT|....... 00002|BBBB|MAIL|DATEOFBIRTHT|....... 00002|BBBB|MAIL|DATEOFBIRTHT|....... 00003|CCCC|MAIL|DATEOFBIRTHT|.......... (1 Reply)
Discussion started by: AMARA
1 Replies

4. Shell Programming and Scripting

Splitting a large file, split command will not do.

Hello Everyone, I have a large file that needs to be split into many seperate files, however the text in between the blank lines need to be intact. The file looks like SomeText SomeText SomeText SomeOtherText SomeOtherText .... Since the number of lines of text are different for... (3 Replies)
Discussion started by: jwillis0720
3 Replies

5. Shell Programming and Scripting

Split a large file

I have a 3 GB text file that I would like to split. How can I do this? It's a giant comma-separated list of numbers. I would like to make it into about 20 files of ~100 MB each, with a custom header and footer. The file can only be split on commas, but they're plentiful. Something like... (3 Replies)
Discussion started by: CRGreathouse
3 Replies

6. UNIX for Dummies Questions & Answers

Split large file to smaller fastly

hi , I have a requirement input file: 1 1111111111111 108 1 1111111111111 109 1 1111111111111 109 1 1111111111111 110 1 1111111111111 111 1 1111111111111 111 1 1111111111111 111 1 1111111111111 112 1 1111111111111 112 1 1111111111111 112 The output should be, (19 Replies)
Discussion started by: mechvijays
19 Replies

7. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

8. UNIX for Advanced & Expert Users

How to split large file with different record delimiter?

Hi, I have received a file which is 20 GB. We would like to split the file into 4 equal parts and process it to avoid memory issues. If the record delimiter is unix new line, I could use split command either with option l or b. The problem is that the line terminator is |##| How to use... (5 Replies)
Discussion started by: Ravi.K
5 Replies

9. Shell Programming and Scripting

Modification of perl script to split a large file into chunks of 5000 chracters

I have a perl script which splits a large file into chunks.The script is given below use strict; use warnings; open (FH, "<monolingual.txt") or die "Could not open source file. $!"; my $i = 0; while (1) { my $chunk; print "process part $i\n"; open(OUT, ">part$i.log") or die "Could... (4 Replies)
Discussion started by: gimley
4 Replies

10. UNIX for Beginners Questions & Answers

Trying To Split a Large File

Trying to split a 35gb file into 1000mb parts. My research shows I should you this. split -b 1000m file.txt and my return is "split: cannot open 'crunch1.txt' for reading: No such file or directory" so I tried split -b 1000m Documents/Wordlists/file.txt and I get nothing other than the curser just... (3 Replies)
Discussion started by: sub terra
3 Replies
ALE(1)																	    ALE(1)

NAME
ale - a synthetic capture engine and renderer SYNOPSIS
ale [OPTIONS] original-frame [supplemental-frames ...] output-file ale [HELP OPTIONS] ale --version DESCRIPTION
ALE is a free software synthetic capture engine and renderer accepting, as input, a sequence of image frames, and producing, as output, rendered images, registration data, and 3D models (currently experimental). EXAMPLES
ale --hu Print the help page on program usage. ale 001.ppm 002.ppm 003.ppm out.ppm Render output file out.ppm from inputs 001.ppm, 002.ppm, and 003.ppm. ale --scale 2 001.ppm 002.ppm 003.ppm out.png Render output file out.png at 2x scale. ale --perturb-upper=0 --trans-load=a.t --projective frame*.png out.png Render output file out.png from inputs frame*.png, using projective transformations as described in the file a.t. ale --3dv 0 out.png --projective frame*.png null: Render output file out.png from inputs frame*.png, using a 3D scene model to determine transformations. ale --visp last:sinc*lanc:6 sf:0 out/ .jpg in/*.jpg null: Render a video stream from files in/*.jpg to the directory out/, stabilizing to the first frame, with a lanczos-windowed sinc filter. USAGE
ale is a wrapper script that calls the executable ale-bin after extracting exposure EXIF information from image files (when the exiftool utility is available) and converting from raw formats (when the dcraw utility is available). When ale-bin has been linked with ImageMagick, input and output files may be of any kind supported by ImageMagick. Otherwise, only PPM files can be read and written by ale-bin. OPTIONS
OPTIONS are summarized via built-in help pages (see HELP OPTIONS below). More detailed manuals describing options and internal algorithms are available for certain versions of ALE. See SEE ALSO below for more details on other manuals. HELP OPTIONS
A listing of available help options can be obtained by invoking ale --hu. ENVIRONMENT
The wrapper script ale recognizes the following environment variables: ALE_BIN ALE executable to use. DCRAW dcraw raw extraction tool to use. EXIF_UTILITY EXIF extraction utility to use. ALE_COUNT_THREADS Displays statistics on thread counts when set to a non-zero value. PAGER Text pager to use (e.g., for displaying help pages). SEE ALSO
More detailed manuals for certain versions of ALE are distributed with source packages, and may also be available from: http://auricle.dyndns.org/ALE/manuals/ 03/31/2009 ALE(1)
All times are GMT -4. The time now is 11:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy