Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Split a huge data into few different files?!
Top Forums Shell Programming and Scripting Split a huge data into few different files?! Post 302366690 by Scrutinizer on Friday 30th of October 2009 06:31:20 AM
Old 10-30-2009
Code:
awk -F'>' '$2{c--} c<=0{f=$2; c=3; $0=FS$2} {print>f}' infile



---------- Post updated at 02:31 AM ---------- Previous update was at 01:57 AM ----------

shell equivalent:
Code:
c=2
while read line; do
  case $line in
    \>*) c=$((c+1));;          # if label is found then increase counter
  esac
  if [ $c -eq 3 ]; then        # if 3 labels have been found then
    exec>${line#>}             # redirect output to file "label"
    c=0                        # reset counter
  fi
  echo $line                   # print output to current output file
done<infile

Last edited by Scrutinizer; 10-30-2009 at 07:40 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl script error to split huge data one by one.

Below is my perl script: #!/usr/bin/perl open(FILE,"$ARGV") or die "$!"; @DATA = <FILE>; close FILE; $join = join("",@DATA); @array = split( ">",$join); for($i=0;$i<=scalar(@array);$i++){ system ("/home/bin/./program_name_count_length MULTI_sequence_DATA_FILE -d... (5 Replies)
Discussion started by: patrick87
5 Replies

2. Shell Programming and Scripting

Problem running Perl Script with huge data files

Hello Everyone, I have a perl script that reads two types of data files (txt and XML). These data files are huge and large in number. I am using something like this : foreach my $t (@text) { open TEXT, $t or die "Cannot open $t for reading: $!\n"; while(my $line=<TEXT>){ ... (4 Replies)
Discussion started by: ad23
4 Replies

3. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

4. Shell Programming and Scripting

how to split a huge file by every 100 lines

into small files. i need to add a head.txt and tail.txt into small files at the begin and end, and give a name as q1.xml q2.xml q3.xml .... thank you very much. (2 Replies)
Discussion started by: dtdt
2 Replies

5. Shell Programming and Scripting

Split a file into several files using a data

Hi All, I have file(File1) with data like below: 102100|LName|Gender|Company|Branch|Bday|Salary|Age 102100|bbbb|male|cccc|dddd|19900814|15000|20| 102101|asdg|male|gggg|ksgu|19911216||| 102102|bdbm|male|kkkk|acke|19931018||23| 102102|kfjg|male|kkkc|gkgg|19921213|14000|24|... (2 Replies)
Discussion started by: sarav.shan
2 Replies

6. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

7. Shell Programming and Scripting

Split a folder with huge number of files in n folders

We have a folder XYZ with large number of files (>350,000). how can i split the folder and create say 10 of them XYZ1 to XYZ10 with 35,000 files each. (doesnt matter which files go where). (12 Replies)
Discussion started by: AlokKumbhare
12 Replies

8. Shell Programming and Scripting

Split JSON to different data files

Hi Gurus, I have below JSON file, now I want to rewrite this file into a new file. I will appreciate if anyone can help me to provide the solution...I can't use jq. { "_id": "3ad893cb4cf1560add7b4caffd4b6126", "_rev": "1-1f0ce165e1d210319cf6e9f9c6ff654f", "name":... (4 Replies)
Discussion started by: manas_ranjan
4 Replies

9. UNIX for Advanced & Expert Users

File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this

I have 2 large file (.dat) around 70 g, 12 columns but the data not sorted in both the files.. need your inputs in giving the best optimized method/command to achieve this and redirect the not macthing lines to the thrid file ( diff.dat) File 1 - 15 columns File 2 - 15 columns Data is... (9 Replies)
Discussion started by: kartikirans
9 Replies

10. Solaris

Split huge File System

Gents I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server. How to can I do that without changing anything in the source. Please your support. (1 Reply)
Discussion started by: AbuAliiiiiiiiii
1 Replies

LEARN ABOUT OSF1

grub-mkrescue

GRUB-MKRESCUE(1)						   User Commands						  GRUB-MKRESCUE(1)

NAME

       grub-mkrescue - make a GRUB rescue image

SYNOPSIS

       grub-mkrescue [OPTION...] [OPTION] SOURCE...

DESCRIPTION

       Make GRUB CD-ROM, disk, pendrive and floppy bootable image.

       --compress=no|xz|gz|lzo
	      compress GRUB files [optional]

       -d, --directory=DIR
	      use images and modules under DIR [default=/usr/lib/grub/<platform>]

       --fonts=FONTS
	      install FONTS [default=unicode]

       --install-modules=MODULES
	      install only MODULES and their dependencies [default=all]

       -k, --pubkey=FILE
	      embed FILE as public key for signature checking

       --locale-directory=DIR use translations under DIR
	      [default=/usr/share/locale]

       --locales=LOCALES
	      install only LOCALES [default=all]

       --modules=MODULES
	      pre-load specified modules MODULES

       --themes=THEMES
	      install THEMES [default=starfield]

       -v, --verbose
	      print verbose messages.

       --arcs-boot
	      enable ARCS (big-endian mips machines, mostly SGI) boot. Disables HFS+, APM, sparc64 and boot as disk image for i386-pc

       --core-compress=xz|none|auto
	      choose the compression to use for core image

       --label-bgcolor=COLOR
	      use COLOR for label background

       --label-color=COLOR
	      use COLOR for label

       --label-font=FILE
	      use FILE as font for label

       -o, --output=FILE
	      save output in FILE [required]

       --product-name=STRING
	      use STRING as product name

       --product-version=STRING
	      use STRING as product version

       --rom-directory=DIR
	      save ROM images in DIR [optional]

       --sparc-boot
	      enable sparc boot. Disables HFS+, APM, ARCS and boot as disk image for i386-pc

       --xorriso=FILE
	      use FILE as xorriso [optional]

       -?, --help
	      give this help list

       --usage
	      give a short usage message

       -V, --version
	      print program version

       Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options.

       Generates  a  bootable CD/USB/floppy image.  Arguments other than options to this program are passed to xorriso, and indicate source files,
       source directories, or any of the mkisofs options listed by the output of `xorriso -as mkisofs -help'.

       Option -- switches to native xorriso command mode.

       Mail xorriso support requests to <bug-xorriso@gnu.org>.

REPORTING BUGS

       Report bugs to <bug-grub@gnu.org>.

SEE ALSO

       grub-mkimage(1)

       The full documentation for grub-mkrescue is maintained as a Texinfo manual.  If the info and grub-mkrescue programs are properly  installed
       at your site, the command

	      info grub-mkrescue

       should give you access to the complete manual.

grub-mkrescue (GRUB) 2.02-2ubuntu8.3				     July 2018							  GRUB-MKRESCUE(1)
All times are GMT -4. The time now is 03:05 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy