Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Split a huge data into few different files?!
Top Forums Shell Programming and Scripting Split a huge data into few different files?! Post 302366690 by Scrutinizer on Friday 30th of October 2009 06:31:20 AM
Old 10-30-2009
Code:
awk -F'>' '$2{c--} c<=0{f=$2; c=3; $0=FS$2} {print>f}' infile



---------- Post updated at 02:31 AM ---------- Previous update was at 01:57 AM ----------

shell equivalent:
Code:
c=2
while read line; do
  case $line in
    \>*) c=$((c+1));;          # if label is found then increase counter
  esac
  if [ $c -eq 3 ]; then        # if 3 labels have been found then
    exec>${line#>}             # redirect output to file "label"
    c=0                        # reset counter
  fi
  echo $line                   # print output to current output file
done<infile

Last edited by Scrutinizer; 10-30-2009 at 07:40 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl script error to split huge data one by one.

Below is my perl script: #!/usr/bin/perl open(FILE,"$ARGV") or die "$!"; @DATA = <FILE>; close FILE; $join = join("",@DATA); @array = split( ">",$join); for($i=0;$i<=scalar(@array);$i++){ system ("/home/bin/./program_name_count_length MULTI_sequence_DATA_FILE -d... (5 Replies)
Discussion started by: patrick87
5 Replies

2. Shell Programming and Scripting

Problem running Perl Script with huge data files

Hello Everyone, I have a perl script that reads two types of data files (txt and XML). These data files are huge and large in number. I am using something like this : foreach my $t (@text) { open TEXT, $t or die "Cannot open $t for reading: $!\n"; while(my $line=<TEXT>){ ... (4 Replies)
Discussion started by: ad23
4 Replies

3. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

4. Shell Programming and Scripting

how to split a huge file by every 100 lines

into small files. i need to add a head.txt and tail.txt into small files at the begin and end, and give a name as q1.xml q2.xml q3.xml .... thank you very much. (2 Replies)
Discussion started by: dtdt
2 Replies

5. Shell Programming and Scripting

Split a file into several files using a data

Hi All, I have file(File1) with data like below: 102100|LName|Gender|Company|Branch|Bday|Salary|Age 102100|bbbb|male|cccc|dddd|19900814|15000|20| 102101|asdg|male|gggg|ksgu|19911216||| 102102|bdbm|male|kkkk|acke|19931018||23| 102102|kfjg|male|kkkc|gkgg|19921213|14000|24|... (2 Replies)
Discussion started by: sarav.shan
2 Replies

6. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

7. Shell Programming and Scripting

Split a folder with huge number of files in n folders

We have a folder XYZ with large number of files (>350,000). how can i split the folder and create say 10 of them XYZ1 to XYZ10 with 35,000 files each. (doesnt matter which files go where). (12 Replies)
Discussion started by: AlokKumbhare
12 Replies

8. Shell Programming and Scripting

Split JSON to different data files

Hi Gurus, I have below JSON file, now I want to rewrite this file into a new file. I will appreciate if anyone can help me to provide the solution...I can't use jq. { "_id": "3ad893cb4cf1560add7b4caffd4b6126", "_rev": "1-1f0ce165e1d210319cf6e9f9c6ff654f", "name":... (4 Replies)
Discussion started by: manas_ranjan
4 Replies

9. UNIX for Advanced & Expert Users

File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this

I have 2 large file (.dat) around 70 g, 12 columns but the data not sorted in both the files.. need your inputs in giving the best optimized method/command to achieve this and redirect the not macthing lines to the thrid file ( diff.dat) File 1 - 15 columns File 2 - 15 columns Data is... (9 Replies)
Discussion started by: kartikirans
9 Replies

10. Solaris

Split huge File System

Gents I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server. How to can I do that without changing anything in the source. Please your support. (1 Reply)
Discussion started by: AbuAliiiiiiiiii
1 Replies

LEARN ABOUT CENTOS

grub2-mkrescue

GRUB-MKRESCUE(3)					     Library Functions Manual						  GRUB-MKRESCUE(3)

NAME

       grub-mkrescue -- Generate a GRUB rescue image using GNU Xorriso.

SYNOPSIS

       grub-mkrescue [-o | --output=FILE] [--modules=MODULES]
		      [--install-modules=MODULES] [--themes=THEMES]
		      [--fonts=FONTS] [--locales=LOCALES]
		      [--compress[=no,xz,gz,lzo]] [-d | --directory=DIR]
		      [--grub-mkimage=FILE] [--rom-directory=DIR]
		      [--xorriso=FILE] [--grub-glue-efi=FILE]
		      [--grub-render-label=FILE] [--label-font=FILE]
		      [--label-color=COLOR] [--label-bgcolor=FILE]
		      [--product-name=STRING] [--product-version=STRING]
		      [--sparc-boot] [--arcs-boot]

DESCRIPTION

       grub-mkrescue can be used to generate a rescue image with the GRUB bootloader.

OPTIONS

       --output=FILE
	      Write the generated file to FILE.  The default is to write to standard output.

       --modules=MODULES
	      Pre-load modules specified by MODULES.

       --install-modules=MODULES
	      Install only MODULES and their dependencies.  The default is to install all available modules.

       --themes=THEMES
	      Install THEMES.  The default is to install the starfield theme, if available.

       --fonts=FONTS
	      Install FONTS.  The default is to install the unicode font.

       --locales=LOCALES
	      Install only locales listed in LOCALES.  The default is to install all available locales.

       --compress[=no,xz,gz,lzo]
	      Compress GRUB files using the specified compression algorithm.

       --directory=DIR
	      Use images and modules in DIR.

       --grub-mkimage=FILE
	      Use FILE as grub-mkimage(1).  The default is /usr/bin/grub-mkimage.

       --rom-directory=DIR
	      Save ROM images in DIR.

       --xorriso=FILE
	      Use FILE as xorriso.

       --grub-glue-efi=FILE
	      Use FILE as grub-glue-efi(3).

       --grub-render-label=FILE
	      Use FILE as grub-render-label(3).

       --label-font=FILE
	      Use FILE as the font file for generated labels.

       --label-color=COLOR
	      Use COLOR as the color for generated labels.

       --label-bgcolor=COLOR
	      Use COLOR as the background color for generated labels.

       --product-name=STRING
	      Use STRING as the product name in generated labels.

       --product-version=STRING
	      Use STRING as the product version in generated labels.

       --sparc-boot
	      Enable booting the SPARC platform.  This disables HFS+, APM, ARCS, and "boot as disk image" on the i386-pc target platform.

       --arcs-boot
	      Enable  ARCS  booting.  This is typically for big-endian MIPS machines, and disables HFS+, APM, sparc64, and "boot as disk image" on
	      the i386-pc target platform.

       --     All options after a -- will be passed directly to xorriso's command line when generating the image.

SEE ALSO

       info grub

								  Wed Feb 26 2014						  GRUB-MKRESCUE(3)
All times are GMT -4. The time now is 07:15 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy