11-15-2011
Combine multiple Files into one big file
Hi Ppl,
I have a requirement like i will be getting files of huge size daily and if the file size is so huge ,the files will be split into many parts and sent.The first file will have the header details followed by detail records and the consecutive files will have detail records and the last file will have the trailer details.I need to merge all these files into one single file in the order Header record then detail records then trailer record.
Example,
Filename pattern is Daily_report_1111_20111115 & Daily_report_2222_20111115.
If the file size is huge it will be split and sent as
Daily_report_1111_20111115_1
Daily_report_1111_20111115_2
Daily_report_1111_20111115_3
Daily_report_1111_20111115_4
Daily_report_2222_20111115_1
Daily_report_2222_20111115_2
Daily_report_2222_20111115_3
Daily_report_2222_20111115_4.
I need to concatenate all the Daily_report_1111_20111115_1,2,3,4 files into one big file Daily_report_1111_20111115 with contents of Daily_report_1111_20111115_1 in the top followed by the contents of the files Daily_report_1111_20111115_2,3,4.So the final output file should be Daily_report_2222_20111115 with contents of all 4 files in the order received(the order of the files will be present in the filename like _ 1,2,3,4)
Similarly for Daily_report_2222_20111115 files.
The requirement is the date 20111115(which is in the filename) will keep changing and we might get multiple days files on the same day.so i need to write and automate the script so that it looks out for the files with the same date and check the filename pattern(either Daily_report_1111_YYYYMMDD or Daily_report_2222_YYYYMMDD and concatenate them into one single file based on the filenames like _1,2,3 etc.
Please help me out!
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi friends,
I have data in flat file as following, first filed is the customer number. We have almost 50-100 customers in the system
100 ABC A123
100 BVC D234
100 BNC N324
200 CBC A122
200 AVC D294
200 HNC N324
300 GBC A173
300 FVC D234
300 DNC N344
I want to split the file and... (5 Replies)
Discussion started by: monicasgupta
5 Replies
2. Shell Programming and Scripting
Hi there,
I was wondering if someone can help me with this.
I am trying the combine multiple columns from multiple files into one file.
Example file 1:
c0t0d0 c0t2d0 # hostname vgname
c0t0d1 c0t2d1 # hostname vgname
c0t0d2 c0t2d2 # hostname vgname
c0t1d0 c0t3d0 # hostname vgname1... (5 Replies)
Discussion started by: martva
5 Replies
3. UNIX for Dummies Questions & Answers
I need to compile a large amount of data with a common string from individual text files throughout many directories.
An example data file is below. I want to search for the following string, "cc_sectors_1" and combine all the data from each file which contains this string, into one new... (2 Replies)
Discussion started by: GradStudent2010
2 Replies
4. Shell Programming and Scripting
Hi All
I am trying to combine columns from multiple text files into a single file using paste command but the record length being unequal in the different files the data is running over to the closest empty cell on the left. Please see below.
What can i do to resolve this ?
File 1 File... (15 Replies)
Discussion started by: venky_ibm
15 Replies
5. Shell Programming and Scripting
I have multiple files; each file contains a certain data in a column view
simply i want to combine all those files into one file in columns
example
file1:
a
b
c
d
file 2:
1
2
3
4
file 3:
G (4 Replies)
Discussion started by: ahmedamro
4 Replies
6. Shell Programming and Scripting
Can anybody help on the script to combine/concatenate columns from multiple files
input1
4 135
5 185
6 85
11 30
16 72
17 30
21 52
22 76
input2
2 50
4 50
6 33
8 62
10 25
12 46
14 42
15 46output (2 Replies)
Discussion started by: sdf
2 Replies
7. Shell Programming and Scripting
I am trying to combine 4 .dat files into one single Output file
Inputs are:- file123.dat, file256.dat, file378.dat & file490
Expected Output:-
FileName=file1
{text from file1}
EOF
{blank line}
FileName=file2
{text from file2}
EOF
{blank line}
FileName=file3
{text from file3}
EOF... (4 Replies)
Discussion started by: lancesunny
4 Replies
8. Shell Programming and Scripting
I have a requirement to split a huge file to smaller text files based on first four characters which look like
ABCD
1234
DFGH
RREX
:
:
:
:
:
0000
Each of these records are OF EQUAL bytes with a different internal layout based on the above first digit identifier..
Any help to start... (5 Replies)
Discussion started by: etldev
5 Replies
9. Shell Programming and Scripting
Good morning all,
I have a problem that is one step beyond a standard awk compare.
I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies
10. Shell Programming and Scripting
this thread is a continuation from previous thread
https://www.unix.com/shell-programming-and-scripting/223901-split-big-file-into-multiple-files-based-first-four-characters.html
..I am using awk to split file and I have a syntax error while executing the below code
I am using AIX 7.2... (4 Replies)
Discussion started by: etldev
4 Replies
LEARN ABOUT LINUX
free_hugepages
ALLOC_HUGEPAGES(2) Linux Programmer's Manual ALLOC_HUGEPAGES(2)
NAME
alloc_hugepages, free_hugepages - allocate or free huge pages
SYNOPSIS
void *alloc_hugepages(int key, void *addr, size_t len,
int prot, int flag);
int free_hugepages(void *addr);
DESCRIPTION
The system calls alloc_hugepages() and free_hugepages() were introduced in Linux 2.5.36 and removed again in 2.5.54. They existed only on
i386 and ia64 (when built with CONFIG_HUGETLB_PAGE). In Linux 2.4.20 the syscall numbers exist, but the calls fail with the error ENOSYS.
On i386 the memory management hardware knows about ordinary pages (4 KiB) and huge pages (2 or 4 MiB). Similarly ia64 knows about huge
pages of several sizes. These system calls serve to map huge pages into the process's memory or to free them again. Huge pages are locked
into memory, and are not swapped.
The key argument is an identifier. When zero the pages are private, and not inherited by children. When positive the pages are shared
with other applications using the same key, and inherited by child processes.
The addr argument of free_hugepages() tells which page is being freed: it was the return value of a call to alloc_hugepages(). (The memory
is first actually freed when all users have released it.) The addr argument of alloc_hugepages() is a hint, that the kernel may or may not
follow. Addresses must be properly aligned.
The len argument is the length of the required segment. It must be a multiple of the huge page size.
The prot argument specifies the memory protection of the segment. It is one of PROT_READ, PROT_WRITE, PROT_EXEC.
The flag argument is ignored, unless key is positive. In that case, if flag is IPC_CREAT, then a new huge page segment is created when
none with the given key existed. If this flag is not set, then ENOENT is returned when no segment with the given key exists.
RETURN VALUE
On success, alloc_hugepages() returns the allocated virtual address, and free_hugepages() returns zero. On error, -1 is returned, and
errno is set appropriately.
ERRORS
ENOSYS The system call is not supported on this kernel.
FILES
/proc/sys/vm/nr_hugepages Number of configured hugetlb pages. This can be read and written.
/proc/meminfo Gives info on the number of configured hugetlb pages and on their size in the three variables HugePages_Total,
HugePages_Free, Hugepagesize.
CONFORMING TO
These calls are specific to Linux on Intel processors, and should not be used in programs intended to be portable.
NOTES
These system calls are gone; they existed only in Linux 2.5.36 through to 2.5.54. Now the hugetlbfs file system can be used instead. Mem-
ory backed by huge pages (if the CPU supports them) is obtained by using mmap(2) to map files in this virtual file system.
The maximal number of huge pages can be specified using the hugepages= boot parameter.
COLOPHON
This page is part of release 3.27 of the Linux man-pages project. A description of the project, and information about reporting bugs, can
be found at http://www.kernel.org/doc/man-pages/.
Linux 2007-05-31 ALLOC_HUGEPAGES(2)