Sponsored Content
Top Forums Shell Programming and Scripting how to split a huge file by every 100 lines Post 302550764 by tukuyomi on Saturday 27th of August 2011 03:56:19 AM
Old 08-27-2011
See -l option of the split command : Man Page for split (Linux Section 1) - The UNIX and Linux Forums
Then, after split is done, do something like (code not tested):
Code:
#!/bin/sh

i=0
#split default names split files as xaa, xab, xac, ..., hence the x* in the command below
for f in x*; do
 i=$((i+1))
 cat head.txt "$f" tail.txt > "q$i.xml"
done

exit 0


Last edited by tukuyomi; 08-27-2011 at 05:04 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split a huge line into multiple 120 characters lines with sed?

Hello , I'm trying to split a file which contains a single very long line. My aim is to split this single line each 120 characters. I tried with the sed command : `cat ${MYPATH}/${FILE}|sed -e :a -e 's/^.\{1,120\}$/&\n/;ta' >{MYPATH}/${DEST}` but when I wc -l the destination file it is... (2 Replies)
Discussion started by: jerome_1664
2 Replies

2. Shell Programming and Scripting

Delete lines from huge file

I have to delete 1st 7000 lines of a file which is 12GB large. As it is so large, i can't open in vi and delete these lines. Also I found one post here which gave solution using perl, but I don't have perl installed. Also some solutions were redirecting the o/p to a different file and renaming it.... (3 Replies)
Discussion started by: rahulrathod
3 Replies

3. UNIX for Dummies Questions & Answers

copy and paste certain many lines of huge file in linux

Dear All, I am working with windoes OS but remote a linux machine. I wonder the way to copy an paste some part of a huge file in linux machine. the contain of file like as follow: ... dump annealling all custom 10 anneal_*.dat id type x y z q timestep 0.02 run 200000 Memory... (2 Replies)
Discussion started by: ariesto
2 Replies

4. Shell Programming and Scripting

NAWK array to store lines from huge file

Hi, I would like to clarify about the NAWK array to store multiple lines from huge file. The file is having an unique REF.NO, I wants to store the lines (it may be 100+ lines) till I found the new REF.NO. How can I apply NAWK - arrays for the above? Rgds, sharif. (1 Reply)
Discussion started by: sharif
1 Replies

5. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

6. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

7. UNIX for Dummies Questions & Answers

How to split a huge file into small pieces (per 2000 columns)?

Dear all, I have a big file:2879(rows)x400,170 (columns) like below. I 'd like to split the file into small pieces:2879(rows)x2000(columns) per file (the last small piece will be 2879x170. So far, I only know how to create one samll piece at one time. But actually I need to repeat this work... (6 Replies)
Discussion started by: forevertl
6 Replies

8. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

9. UNIX for Advanced & Expert Users

How to split a large file with the first 100 lines of each condition?

I have a huge file with the following input: Case1 Specific_Info Specific_Info Case1 Specific_Info Specific_Info Case3 Specific_Info Specific_Info Case4 Specific_Info Specific_Info Case1 Specific_Info Specific_Info Case2 Specific_Info Specific_Info Case2 Specific_Info Specific_Info... (2 Replies)
Discussion started by: laurigo
2 Replies

10. Solaris

Split huge File System

Gents I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server. How to can I do that without changing anything in the source. Please your support. (1 Reply)
Discussion started by: AbuAliiiiiiiiii
1 Replies
STAG-PARSE(1p)						User Contributed Perl Documentation					    STAG-PARSE(1p)

NAME
stag-parse - parses a file and fires events (e.g. sxpr to xml) SYNOPSIS
# convert XML to IText stag-parse -p xml -w itext file1.xml file2.xml # use a custom parser/generator and a custom writer/generator stag-parse -p MyMod::MyParser -w MyMod::MyWriter file.txt DESCRIPTION
script wrapper for the Data::Stag modules feeds in files into a parser object that generates nestarray events, and feeds the events into a handler/writer class ARGUMENTS
-p|parser FORMAT FORMAT is one of xml, sxpr or itext, or the name of a perl module this is the class that parsers the input file(s) and generates stag events xml assumed as default -w|writer FORMAT FORMAT is one of xml, sxpr or itext, or the name of a perl module this is the class that catches the events thrown by the parser; it can be any class, but the class is typically a writer xml assumed as default -o|out FILE the writer will use this file (defaults to STDOUT) -e|errf FILE file to store parse error handler output -errhandler FORMAT/MODULE FORMAT is one of xml, sxpr or itext, or the name of a perl module all parse error events go to this module -r|root NODE_NAME if this is specified, NODE_NAME becomes the root of the stag tree, and anything that was previously the root is placed below this. this happens automatically if more than one file is parsed (because there can only be one tree root) -color Works only if the output handler is able to provide ASCII-colors (currently supported for itext and xml) SEE ALSO
Data::Stag This script is a wrapper for the method Data::Stag->parse() perl v5.10.0 2008-12-23 STAG-PARSE(1p)
All times are GMT -4. The time now is 02:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy