Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Merge files into groups of 10000 Post 302518066 by jim mcnamara on Thursday 28th of April 2011 01:51:37 PM
Old 04-28-2011
Code:
#!/bin/ksh
cnt=0
i=1
base=bigfile.dat
ls -1 * > $HOME/listfile
while read fname
do
     cat $fname >> base${i}
     echo "" >> base{$i}
     cnt=$(( $cnt + 1 ))
      [ $(( $cnt % 10000 )) -eq 0 ] &&  i=$(( $i + 1 ))  
done  < $HOME/listfile

This will make files: bigfile.dat1 ... bigfile.dat15

Most of your problems with 'efficiency' are related to having way too many files in a directory. ls will take a lot of time to return before the loop engages. You should rebuild the directory after you consoildate the files into fifteen big files.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

2. Shell Programming and Scripting

to find the last updated file from different groups of files.

Hi i have many sets of files as shown below(here i have shown 2 sets) basel_aa_20091030.txt basel_aa_20091130.txt basel_aa_20091230.txt basel_bb_20091030.txt basel_bb_20091130.txt basel_bb_20091230.txt from each set of files i need to select the latest updated file(there are... (3 Replies)
Discussion started by: jagadeeshn04
3 Replies

3. Web Development

Please till me how to stop or to limit some IP which download files for more than 10000

Please till me how to stop or to limit some IP which download .rm and .mp3 files for more than 10000 times ... I have two cases : 1. code 206 up to 20/second 5 GB or more than that ... 2. code 206 up to 20/second but less than 0.5 GB I used Ddos and mod_evasive20.so <IfModule... (0 Replies)
Discussion started by: nonowa
0 Replies

4. Shell Programming and Scripting

multiple groups of files processing

I have five directories, dir1 to dir5 for each directory, I have all same number-named folders. There are four types of folders, {1..10}, {20..30}, { 40..50}, {60..70} Now for each types of folder, I will do the same thing, here is the code for i in {1..5} do cd dir$i mkdir temp1 for... (5 Replies)
Discussion started by: ksgreen
5 Replies

5. UNIX for Advanced & Expert Users

SQL script with 86000 lines: new files with only 10000 lines (per file)

Hi this is my SQL script $ wc -l insert_into_customers.sql 85601 insert_into_customers.sqlI wish to cut this file into 9 files each 10000 lines (the last one less) $ wc -l insert_into_customers_00*.sql 10000 insert_into_customers_001.sql 10000 insert_into_customers_002.sql ... (1 Reply)
Discussion started by: slashdotweenie
1 Replies

6. Shell Programming and Scripting

Searching a particular string pattern in 10000 files

Problem Statement:- I need to search a particular `String Pattern` in around `10000 files` and find the records which contains that `particular pattern`. I can use `grep` here, but it is taking lots of time. Below is the command I am using to search a `particular string pattern` after... (3 Replies)
Discussion started by: raihan26
3 Replies

7. Shell Programming and Scripting

Urgent...Need a shell script to list files which belong to particular groups

Hi, I am just new to scripting but got to write a complex scipt please help. i need a shell script which can check the list of data listed in a txt doc and see if they belong to any of the groups that are listed in other list file.... (5 Replies)
Discussion started by: draghun9
5 Replies

8. Shell Programming and Scripting

Move groups of files

G'day all, I'm have tons of image files I need to process, but I don't need to process all of them and it would take a long time to process them all if I don't have to. The images are arranged in folders like this... folder1/RawData folder2/RawData folder3/RawData ... folderN/RawData ... (2 Replies)
Discussion started by: Dan_S
2 Replies

9. Shell Programming and Scripting

Checking in a directory how many files are present and basing on that merge all the files

Hi, My requirement is,there is a directory location like: :camp/current/ In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated... (10 Replies)
Discussion started by: srikanth_sagi
10 Replies

10. Shell Programming and Scripting

Rearrange groups of lines from several files

I have three files as an input and I need to rearrange this input to match the rules by which the processing program consumes the data. My files are: /tmp$ cat F # file -1- FS00|0|zero-zero| FSTA|0|10| FSTA|0|12| FSTA|0|15| FSTA|0|17| FS00|3|negative| FSTA|3|-1| FS00|5|regular|... (2 Replies)
Discussion started by: migurus
2 Replies
PCAP_LOOP(3PCAP)														  PCAP_LOOP(3PCAP)

NAME
pcap_loop, pcap_dispatch - process packets from a live capture or savefile SYNOPSIS
#include <pcap/pcap.h> typedef void (*pcap_handler)(u_char *user, const struct pcap_pkthdr *h, const u_char *bytes); int pcap_loop(pcap_t *p, int cnt, pcap_handler callback, u_char *user); int pcap_dispatch(pcap_t *p, int cnt, pcap_handler callback, u_char *user); DESCRIPTION
pcap_loop() processes packets from a live capture or ``savefile'' until cnt packets are processed, the end of the ``savefile'' is reached when reading from a ``savefile'', pcap_breakloop() is called, or an error occurs. It does not return when live read timeouts occur. A value of -1 or 0 for cnt is equivalent to infinity, so that packets are processed until another ending condition occurs. pcap_dispatch() processes packets from a live capture or ``savefile'' until cnt packets are processed, the end of the current bufferful of packets is reached when doing a live capture, the end of the ``savefile'' is reached when reading from a ``savefile'', pcap_breakloop() is called, or an error occurs. Thus, when doing a live capture, cnt is the maximum number of packets to process before returning, but is not a minimum number; when reading a live capture, only one bufferful of packets is read at a time, so fewer than cnt packets may be processed. A value of -1 or 0 for cnt causes all the packets received in one buffer to be processed when reading a live capture, and causes all the packets in the file to be processed when reading a ``savefile''. (In older versions of libpcap, the behavior when cnt was 0 was undefined; different platforms and devices behaved differently, so code that must work with older versions of libpcap should use -1, nor 0, as the value of cnt.) callback specifies a pcap_handler routine to be called with three arguments: a u_char pointer which is passed in the user argument to pcap_loop() or pcap_dispatch(), a const struct pcap_pkthdr pointer pointing to the packet time stamp and lengths, and a const u_char pointer to the first caplen (as given in the struct pcap_pkthdr a pointer to which is passed to the callback routine) bytes of data from the packet. The struct pcap_pkthdr and the packet data are not to be freed by the callback routine, and are not guaranteed to be valid after the callback routine returns; if the code needs them to be valid after the callback, it must make a copy of them. RETURN VALUE
pcap_loop() returns 0 if cnt is exhausted, -1 if an error occurs, or -2 if the loop terminated due to a call to pcap_breakloop() before any packets were processed. It does not return when live read timeouts occur; instead, it attempts to read more packets. pcap_dispatch() returns the number of packets processed on success; this can be 0 if no packets were read from a live capture (if, for example, they were discarded because they didn't pass the packet filter, or if, on platforms that support a read timeout that starts before any packets arrive, the timeout expires before any packets arrive, or if the file descriptor for the capture device is in non-blocking mode and no packets were available to be read) or if no more packets are available in a ``savefile.'' It returns -1 if an error occurs or -2 if the loop terminated due to a call to pcap_breakloop() before any packets were processed. If your application uses pcap_breakloop(), make sure that you explicitly check for -1 and -2, rather than just checking for a return value < 0. If -1 is returned, pcap_geterr() or pcap_perror() may be called with p as an argument to fetch or display the error text. SEE ALSO
pcap(3PCAP), pcap_geterr(3PCAP), pcap_breakloop(3PCAP) 24 December 2008 PCAP_LOOP(3PCAP)
All times are GMT -4. The time now is 07:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy