Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: split large file based on field criteria
Top Forums Shell Programming and Scripting split large file based on field criteria Post 302327688 by durden_tyler on Monday 22nd of June 2009 10:41:40 AM
Old 06-22-2009
Code:
$
$ cat data.txt
2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1
2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1
2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0
2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1
2009/06/10,20:59:59.950,XAG/USD,A,0,, 14.3011,100,1
2009/06/10,21:00:00.100,CHF/JPY,Q,0,0, , 0,0,0
2009/06/10,21:00:00.100,CHF/JPY,Q,1,0, 70.26, 60,2,2
2009/06/10,21:00:00.150,CHF/JPY,D,0, 70.14, 20,XC05, ,NYD9,US,NYA1
$
$ perl -ne 'BEGIN{open(F1,">file1"); open(F2,">file2")}
>   { split/[,:]/;
>     if($_[1] <= 20){print F1 $_} else{print F2 $_}
>   }
> END {close(F1); close(F2)}' data.txt
$
$ cat file1
2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1
2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1
2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0
2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1
2009/06/10,20:59:59.950,XAG/USD,A,0,, 14.3011,100,1
$
$ cat file2
2009/06/10,21:00:00.100,CHF/JPY,Q,0,0, , 0,0,0
2009/06/10,21:00:00.100,CHF/JPY,Q,1,0, 70.26, 60,2,2
2009/06/10,21:00:00.150,CHF/JPY,D,0, 70.14, 20,XC05, ,NYD9,US,NYA1
$
$

tyler_durden
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split file based on field

Hi I have a large file 2.6 million records and I am trying to split the file based on last column. I am doing awk -F"|" '{ print > $NF }' filename1 After around 1000 splits it gives me a error awk: can't open file 3332332423 input record number 1068, file filename1 source... (6 Replies)
Discussion started by: s_adu
6 Replies

2. Solaris

Split a file which a word criteria in two files with awk

Hello, I'm searching with the Awk command to split a file into two others files. I explain : in the file N°1 I search the word "NameVirtual" and since that word to the end of the file I want to store all lines in a new file N°2 Also from that word to the beginning of the file I want to... (11 Replies)
Discussion started by: steiner
11 Replies

3. Shell Programming and Scripting

Split large file based on last digit from a column

Hello, What's the best way to split a large into multiple files based on the last digit in the first column. input file: f 2738483300000x0y03772748378831x1y13478378358383x2y23743878383802x3y33787828282820x4y43748838383881x5y5 Desired Output: f0 3738483300000x0y03787828282820x4y4 f1... (9 Replies)
Discussion started by: alain.kazan
9 Replies

4. UNIX for Dummies Questions & Answers

remove duplicates based on a field and criteria

Hi, I have a file with fields like below: A;XYZ;102345;222 B;XYZ;123243;333 C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 desired output: C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 ie, if the 4rth field is a duplicate.. i need only those records where... (5 Replies)
Discussion started by: wanderingmind16
5 Replies

5. Shell Programming and Scripting

Splitting large file and renaming based on field

I am trying to update an older program on a small cluster. It uses individual files to send jobs to each node. However the newer database comes as one large file, containing over 10,000 records. I therefore need to split this file. It looks like this: HMMER3/b NAME 1-cysPrx_C ACC ... (2 Replies)
Discussion started by: fozrun
2 Replies

6. Shell Programming and Scripting

Split a file into multiple files based on field value

Hi, I've one requirement. I have to split one comma delimited file into multiple files based on one of the column values. How can I achieve this Unix Here is the sample data. In this case I have split the files based on date column(c4) Input file c1,c2,c3,c4,c5... (1 Reply)
Discussion started by: manasvi24
1 Replies

7. Shell Programming and Scripting

How to split file into multiple files using awk based on 1 field in the file?

Good day all I need some helps, say that I have data like below, each field separated by a tab DATE NAME ADDRESS 15/7/2012 LX a.b.c 15/7/2012 LX1 a.b.c 16/7/2012 AB a.b.c 16/7/2012 AB2 a.b.c 15/7/2012 LX2 a.b.c... (2 Replies)
Discussion started by: alexyyw
2 Replies

8. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Help needed urgently please. I have a large file - a few hundred thousand lines. Sample CP START ACCOUNT 1234556 name 1 CP END ACCOUNT CP START ACCOUNT 2224444 name 1 CP END ACCOUNT CP START ACCOUNT 333344444 name 1 CP END ACCOUNT I need to split this file each time "CP START... (7 Replies)
Discussion started by: frustrated1
7 Replies

9. Shell Programming and Scripting

Split Large Files Based On Row Pattern..

Hi all. I've tried searching the web but could not find similar problem to mine. I have one large file to be splitted into several files based on the matching pattern found in each row. For example, let's say the file content: ... (13 Replies)
Discussion started by: aimy
13 Replies

10. Shell Programming and Scripting

Split file based on a column/field value

Hi All, I have a requirement to split file into 2 sets of file. Below is a sample data of the file AU;PTN;24EX;25-AUG-14;AU;123;SE;123;Test NN;;;;ASD; AU;PTN;24EX;25-AUG-14;AU;456;SE;456;Test NN;;;;ASD; AU;PTN;24EX;25-AUG-14;AU;147;SE;147;Test NN;;;;ASD;... (6 Replies)
Discussion started by: galaxy_rocky
6 Replies

LEARN ABOUT DEBIAN

gdcmtar

gdcmtar(1)							       GDCM								gdcmtar(1)

NAME

       gdcmtar - Concatenate/Extract DICOM files.

SYNOPSIS

       gdcmtar [options] file-in file-out

DESCRIPTION

       The gdcmtar is a command line tool used to tar/untar multi-frames images (including SIEMENS MOSAIC file)

PARAMETERS

       file-in	 DICOM input filename

       file-out  DICOM output filename

options
   options
	    --enhance	 enhance (default)
	 -U --unenhance  unenhance
	 -M --mosaic	 Split SIEMENS Mosaic image into multiple frames.
	 -p --pattern	 Specify trailing file pattern.
	    --root-uid	      Root UID.

   general options
	 -h   --help
		print this help text and exit

	 -v   --version
		print version information and exit

	 -V   --verbose
		verbose mode (warning+error).

	 -W   --warning
		warning mode, print warning information

	 -E   --error
		error mode, print error information

	 -D   --debug
		debug mode, print debug information

   environment variable
	 GDCM_ROOT_UID Root UID

Typical usage
   SIEMENS Mosaic
       $ gdcminfo MR-sonata-3D-as-Tile.dcm

       MediaStorage is 1.2.840.10008.5.1.4.1.1.4 [MR Image Storage]
       TransferSyntax is 1.2.840.10008.1.2.1 [Explicit VR Little Endian]
       NumberOfDimensions: 2
       Dimensions: (384,384,1)
       ...

       $ gdcmtar --mosaic -i MR-sonata-3D-as-Tile.dcm -o mosaic --pattern %03d.dcm

       Will output:

       -rw-r--r--   1 mathieu mathieu 72882 2009-08-10 11:14 mosaic000.dcm
       -rw-r--r--   1 mathieu mathieu 72886 2009-08-10 11:14 mosaic001.dcm
       -rw-r--r--   1 mathieu mathieu 72886 2009-08-10 11:14 mosaic002.dcm
       -rw-r--r--   1 mathieu mathieu 72886 2009-08-10 11:14 mosaic003.dcm
       -rw-r--r--   1 mathieu mathieu 72886 2009-08-10 11:14 mosaic004.dcm
       -rw-r--r--   1 mathieu mathieu 72886 2009-08-10 11:14 mosaic005.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic006.dcm
       -rw-r--r--   1 mathieu mathieu 72882 2009-08-10 11:14 mosaic007.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic008.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic009.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic010.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic011.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic012.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic013.dcm
       -rw-r--r--   1 mathieu mathieu 72882 2009-08-10 11:14 mosaic014.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic015.dcm
       -rw-r--r--   1 mathieu mathieu 72882 2009-08-10 11:14 mosaic016.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic017.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic018.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic019.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic020.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic021.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic022.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic023.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic024.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic025.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic026.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic027.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic028.dcm
       -rw-r--r--   1 mathieu mathieu 72884 2009-08-10 11:14 mosaic029.dcm
       -rw-r--r--   1 mathieu mathieu 72882 2009-08-10 11:14 mosaic030.dcm

       $ gdcminfo mosaic000.dcm

       MediaStorage is 1.2.840.10008.5.1.4.1.1.4 [MR Image Storage]
       TransferSyntax is 1.2.840.10008.1.2.1 [Explicit VR Little Endian]
       NumberOfDimensions: 2
       Dimensions: (64,64,1)
       ...

SEE ALSO

       gdcmdump(1), gdcmraw(1), gdcminfo(1)

COPYRIGHT

       Copyright (c) 2006-2011 Mathieu Malaterre

Version 2.2.0							  Tue Feb 5 2013							gdcmtar(1)
All times are GMT -4. The time now is 12:15 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy