Sponsored Content
Top Forums Shell Programming and Scripting Split a huge data into few different files?! Post 302367199 by Scrutinizer on Sunday 1st of November 2009 11:10:48 PM
Old 11-02-2009
Quote:
Originally Posted by patrick87
Hi Scrutinizer, do you have any idea to get my desired output result?
I try to replace the space of header with "_" and try your suggested code.
Unfortunately, it still can't work Smilie
Thanks a lot for your advise.
Hi patrick87,

The problem is, I put random spaces and : characters inside the labels of your input examples you gave and both scripts still work as expected. I have to assume your real world data sets somehow do not correspond with the input format you provided. You would have to take a small part (say 7 records) of an actual, anonymized, file, then run my scripts on them to see if they also produce the strange results and then post that example input file here, and also list the strange resulting file names and their content, so I can have a look.

S.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl script error to split huge data one by one.

Below is my perl script: #!/usr/bin/perl open(FILE,"$ARGV") or die "$!"; @DATA = <FILE>; close FILE; $join = join("",@DATA); @array = split( ">",$join); for($i=0;$i<=scalar(@array);$i++){ system ("/home/bin/./program_name_count_length MULTI_sequence_DATA_FILE -d... (5 Replies)
Discussion started by: patrick87
5 Replies

2. Shell Programming and Scripting

Problem running Perl Script with huge data files

Hello Everyone, I have a perl script that reads two types of data files (txt and XML). These data files are huge and large in number. I am using something like this : foreach my $t (@text) { open TEXT, $t or die "Cannot open $t for reading: $!\n"; while(my $line=<TEXT>){ ... (4 Replies)
Discussion started by: ad23
4 Replies

3. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

4. Shell Programming and Scripting

how to split a huge file by every 100 lines

into small files. i need to add a head.txt and tail.txt into small files at the begin and end, and give a name as q1.xml q2.xml q3.xml .... thank you very much. (2 Replies)
Discussion started by: dtdt
2 Replies

5. Shell Programming and Scripting

Split a file into several files using a data

Hi All, I have file(File1) with data like below: 102100|LName|Gender|Company|Branch|Bday|Salary|Age 102100|bbbb|male|cccc|dddd|19900814|15000|20| 102101|asdg|male|gggg|ksgu|19911216||| 102102|bdbm|male|kkkk|acke|19931018||23| 102102|kfjg|male|kkkc|gkgg|19921213|14000|24|... (2 Replies)
Discussion started by: sarav.shan
2 Replies

6. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

7. Shell Programming and Scripting

Split a folder with huge number of files in n folders

We have a folder XYZ with large number of files (>350,000). how can i split the folder and create say 10 of them XYZ1 to XYZ10 with 35,000 files each. (doesnt matter which files go where). (12 Replies)
Discussion started by: AlokKumbhare
12 Replies

8. Shell Programming and Scripting

Split JSON to different data files

Hi Gurus, I have below JSON file, now I want to rewrite this file into a new file. I will appreciate if anyone can help me to provide the solution...I can't use jq. { "_id": "3ad893cb4cf1560add7b4caffd4b6126", "_rev": "1-1f0ce165e1d210319cf6e9f9c6ff654f", "name":... (4 Replies)
Discussion started by: manas_ranjan
4 Replies

9. UNIX for Advanced & Expert Users

File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this

I have 2 large file (.dat) around 70 g, 12 columns but the data not sorted in both the files.. need your inputs in giving the best optimized method/command to achieve this and redirect the not macthing lines to the thrid file ( diff.dat) File 1 - 15 columns File 2 - 15 columns Data is... (9 Replies)
Discussion started by: kartikirans
9 Replies

10. Solaris

Split huge File System

Gents I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server. How to can I do that without changing anything in the source. Please your support. (1 Reply)
Discussion started by: AbuAliiiiiiiiii
1 Replies
DH-EXEC(1)							      dh-exec								DH-EXEC(1)

NAME
dh-exec - Debhelper executable file helpers SYNOPSIS
#! /usr/bin/dh-exec src/libfoo-*.so.* debian/foo-plugins/usr/lib/foo/${DEB_HOST_MULTIARCH}/ etc/example.conf => debian/foo/etc/foo/foo.conf DESCRIPTION
dh-exec is a simple program, meant to be used as the interpreter for executable debhelper config files. It is a wrapper around the various other sub-commands (see below), and will pipe the input file through all of them in turn, using an ordering that makes most sense in the vast majority of cases. The order as of now is dh-exec-subst gets run first, followed by dh-exec-install, so that variable expansion happens before files need to be copied. ARCHITECTURE
dh-exec is built up from three layers: there is the dh-exec utility, its single entry point, the only thing one will need to call. Below that, there are the various sub-commands, such as dh-exec-subst, dh-exec-installs and dh-exec-illiterate, which are thin wrappers around the various dh-exec scripts, that make sure they only run those that need to be run. And the lowest layer are the various scripts that do the actual work. One can control which sub-commands to run, or if even more granularity is desired, one can limit which scripts shall be run, too. See below for the options! OPTIONS
--with=command[,command ...] Replace the list of sub-commands to run the input through with a custom list (where entries are separated by whitespace or commas). This option will always replace the existing list with whatever is specified. This can be used to explicitly set which sub-commands to use. The list must not include the dh-exec- prefix. Defaults to subst,install. --without=command[,command ...] Inversely to the option above, this lists all the sub-commands which should not be used. The list must not include the dh-exec- prefix. --with-scripts=script[,script ...] Replace the list of scripts to run the input through with a custom list (where entries are separated by whitespace or commas). This option will always replace the existing list with whatever is specified. This can be used to explicitly specify which scripts to use, limiting even beyond what the --with option is capable of. The list must not include the dh-exec- prefix. By default it is empty, meaning there is no filtering done, and whatever scripts the sub-commands find, will be run. --no-act Do not really do anything, but print the pipeline that would have been run instead. --list List the available sub-commands and scripts, grouped by sub-command. --help, --version Display a short help or the package version, respectively. SUB-COMMANDS dh-exec-subst Substitutes various variables (either from the environment, or from dpkg-architecture(1)). dh-exec-install An extension to dh_install(1), that supports renaming files during the copy process, using a special syntax. ENVIRONMENT
DH_EXEC_LIBDIR The directory in which the wrapped sub-commands reside. Defaults to /usr/lib/dh-exec/. DH_EXEC_SCRIPTDIR The directory in which the scripts that do the heavy work live. Defaults to /usr/share/dh-exec/. FILES
$DH_EXEC_LIBDIR/dh-exec-* The various sub-commands. $DH_EXEC_SCRIPTDIR/dh-exec-* The various scripts ran by the sub-commands. SEE ALSO
debhelper(1), dh-exec-subst(1), dh-exec-install(1) AUTHOR
dh-exec is copyright (C) 2011-2012 by Gergely Nagy <algernon@madhouse-project.org>. 2012-05-03 DH-EXEC(1)
All times are GMT -4. The time now is 06:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy