04-27-2011
Merge files into groups of 10000
Hi Guys,
First post! I've seen a few options but dont know the most efficient:
I have a directory with a 150,000+ text files in it
I want to merge them into files contain 10,000 files with a carriage return in between.
Thanks
P
The following is an example but doesnt limit the number of files to merge:
$ paste -d "\n" * >> x.txt
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
hi,
i am facing a problem in merging two files using awk,
the problem is as stated below,
file1:
A|B|C|D|E|F|G|H|I|1
M|N|O|P|Q|R|S|T|U|2
AA|BB|CC|DD|EE|FF|GG|HH|II|1
....
....
....
file2 :
1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies
2. Shell Programming and Scripting
Hi
i have many sets of files as shown below(here i have shown 2 sets)
basel_aa_20091030.txt
basel_aa_20091130.txt
basel_aa_20091230.txt
basel_bb_20091030.txt
basel_bb_20091130.txt
basel_bb_20091230.txt
from each set of files i need to select the latest updated file(there are... (3 Replies)
Discussion started by: jagadeeshn04
3 Replies
3. Web Development
Please till me how to stop or to limit some IP which download .rm and .mp3 files for more than 10000 times ...
I have two cases :
1. code 206 up to 20/second 5 GB or more than that ...
2. code 206 up to 20/second but less than 0.5 GB
I used Ddos and mod_evasive20.so
<IfModule... (0 Replies)
Discussion started by: nonowa
0 Replies
4. Shell Programming and Scripting
I have five directories, dir1 to dir5
for each directory, I have all same number-named folders. There are four types of folders, {1..10}, {20..30}, { 40..50}, {60..70}
Now for each types of folder, I will do the same thing, here is the code
for i in {1..5}
do
cd dir$i
mkdir temp1
for... (5 Replies)
Discussion started by: ksgreen
5 Replies
5. UNIX for Advanced & Expert Users
Hi
this is my SQL script
$ wc -l insert_into_customers.sql
85601 insert_into_customers.sqlI wish to cut this file into 9 files each 10000 lines (the last one less)
$ wc -l insert_into_customers_00*.sql
10000 insert_into_customers_001.sql
10000 insert_into_customers_002.sql
... (1 Reply)
Discussion started by: slashdotweenie
1 Replies
6. Shell Programming and Scripting
Problem Statement:-
I need to search a particular `String Pattern` in around `10000 files` and find the records which contains that `particular pattern`. I can use `grep` here, but it is taking lots of time.
Below is the command I am using to search a `particular string pattern` after... (3 Replies)
Discussion started by: raihan26
3 Replies
7. Shell Programming and Scripting
Hi,
I am just new to scripting but got to write a complex scipt please help. i need a shell script which can check the list of data listed in a txt doc and see if they belong to any of the groups that are listed in other list file.... (5 Replies)
Discussion started by: draghun9
5 Replies
8. Shell Programming and Scripting
G'day all,
I'm have tons of image files I need to process, but I don't need to process all of them and it would take a long time to process them all if I don't have to.
The images are arranged in folders like this...
folder1/RawData
folder2/RawData
folder3/RawData
...
folderN/RawData
... (2 Replies)
Discussion started by: Dan_S
2 Replies
9. Shell Programming and Scripting
Hi,
My requirement is,there is a directory location like:
:camp/current/
In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated... (10 Replies)
Discussion started by: srikanth_sagi
10 Replies
10. Shell Programming and Scripting
I have three files as an input and I need to rearrange this input to match the rules by which the processing program consumes the data.
My files are:
/tmp$ cat F
# file -1-
FS00|0|zero-zero|
FSTA|0|10|
FSTA|0|12|
FSTA|0|15|
FSTA|0|17|
FS00|3|negative|
FSTA|3|-1|
FS00|5|regular|... (2 Replies)
Discussion started by: migurus
2 Replies
LEARN ABOUT XFREE86
git-merge-index
GIT-MERGE-INDEX(1) Git Manual GIT-MERGE-INDEX(1)
NAME
git-merge-index - Run a merge for files needing merging
SYNOPSIS
git merge-index [-o] [-q] <merge-program> (-a | [--] <file>*)
DESCRIPTION
This looks up the <file>(s) in the index and, if there are any merge entries, passes the SHA-1 hash for those files as arguments 1, 2, 3
(empty argument if no file), and <file> as argument 4. File modes for the three files are passed as arguments 5, 6 and 7.
OPTIONS
--
Do not interpret any more arguments as options.
-a
Run merge against all files in the index that need merging.
-o
Instead of stopping at the first failed merge, do all of them in one shot - continue with merging even when previous merges returned
errors, and only return the error code after all the merges.
-q
Do not complain about a failed merge program (a merge program failure usually indicates conflicts during the merge). This is for
porcelains which might want to emit custom messages.
If git merge-index is called with multiple <file>s (or -a) then it processes them in turn only stopping if merge returns a non-zero exit
code.
Typically this is run with a script calling Git's imitation of the merge command from the RCS package.
A sample script called git merge-one-file is included in the distribution.
ALERT ALERT ALERT! The Git "merge object order" is different from the RCS merge program merge object order. In the above ordering, the
original is first. But the argument order to the 3-way merge program merge is to have the original in the middle. Don't ask me why.
Examples:
torvalds@ppc970:~/merge-test> git merge-index cat MM
This is MM from the original tree. # original
This is modified MM in the branch A. # merge1
This is modified MM in the branch B. # merge2
This is modified MM in the branch B. # current contents
or
torvalds@ppc970:~/merge-test> git merge-index cat AA MM
cat: : No such file or directory
This is added AA in the branch A.
This is added AA in the branch B.
This is added AA in the branch B.
fatal: merge program failed
where the latter example shows how git merge-index will stop trying to merge once anything has returned an error (i.e., cat returned an
error for the AA file, because it didn't exist in the original, and thus git merge-index didn't even try to merge the MM thing).
GIT
Part of the git(1) suite
Git 2.17.1 10/05/2018 GIT-MERGE-INDEX(1)