Shell Programming and Scripting

hi,
I have the following problem:
- 6 different files that have one key in common.
- this six files must be aggregated in one output file sorted by the key.
- the main file has to be writen twice, one in the beggining of the new output file and another in the end, for each key.
- add one identifier to the output file in order to do a sort with the expected order of the lines.
- a friend told me to use AWK to gather them, but i have never seen this and only after finishing my current task i wil check about this.
- i really have to check about performance because each file has about 800.000 lines and i have 150 files!


if someone can give me some tips or where to get one example to do my work fast i would be appreciated ( this way i would probably start my weekend at decent times).
I think the biggest headdik will be because there can the lines without key (in witch i must use the last readed key), the fact i have to read twice the first file and that the sort must be done by the identifier in order to get the expected order!

example from files to see what i have to gather:

1º:

Code:

410000000000001,dummy, data,separated,by,comma,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,
410000000000002,,dummy, data,separated,by,comma,,,,,,,,
410000000000003,,dummy, data,separated,by,comma,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,

2º

Code:

410000000000001,,,dummy, data,separated,by,comma,,,,,,,,
410000000000002,,,dummy, data,separated,by,comma,,,,,,,,

3º

Code:

410000000000001,,,dummy, data,separated,by,comma,,,,,,,,
410000000000002,,,dummy, data,separated,by,comma,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,
410000000000003,,,dummy, data,separated,by,comma,,,,,,,,
410000000000004,,,dummy, data,separated,by,comma,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,

4º

Code:

41000000000001,dummy, data,separated,by,comma,,,,,,,,
41000000000002,dummy, data,separated,by,comma,,,,,,,,
,,,,,,,
,,,,,,,
,,,,,,,
41000000000003,dummy, data,separated,by,comma,,,,,,,,
41000000000004,dummy, data,separated,by,comma,,,,,,,,
,,,,,,,

5º

Code:

410000000000001,dummy, data,separated,by,comma,,,,,,,,
410000000000002,dummy, data,separated,by,comma,,,,,,,,

6º

Code:

410000000000001,dummy, data,separated,by,comma,,,,,,,,
410000000000002,dummy, data,separated,by,comma,,,,,,,,

in:

Code:

dummy, data,separated,by,comma

i will have a variable number of fields, depending on the file

Thanks for the help.

Best regards,
Ricardo Tomás

Sorry I am not sure of fully understanding what you are trying to achieve. Is is about concatenating the 6 files in one? can you not use cat file1 >> OutputFile and then cat file2 >> OutputFile and so on?

As per the common key, you mean there is a field content that is shared by all the files? Can you provide a shorter example (just 2 or 3 files) with some contents and the expected output file? Like:

File1
xxxxxxxxxx

File2
xxxxxxxxxx

Output File
xxxxxxxxxxxxxxxxxxx

If you want to extract the first field from your files using awk then you can do

cat theFile | awk -F, '{print $1}'

but I am not sure if that's what you are looking for.

explaining my situation best:




the information may came like this:

410000000000001,dummy, data,separated,by,comma,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,
410000000000002,,dummy, data,separated,by,comma,,,,,,,,
410000000000003,,dummy, data,separated,by,comma,,,,,,,,
,,,,,,,,,,,,,,,,,,,,,,,,,

and in some files i cames with the first value in all the lines.
like this:
410000000000002,,dummy, data,separated,by,comma,,,,,,,,
410000000000003,,dummy, data,separated,by,comma,,,,,,,,

i want it to stay like for each file:

X,410000000000001,dummy, data,separated,by,comma,,,,,,,,
X,410000000000001,,,,,,,,,,,,,,,,,,,,,,,,,
X,410000000000001,,,,,,,,,,,,,,,,,,,,,,,,,
X,410000000000001,,,,,,,,,,,,,,,,,,,,,,,,,
X,410000000000002,,dummy, data,separated,by,comma,,,,,,,,
X,410000000000003,,dummy, data,separated,by,comma,,,,,,,,
X,410000000000003,,,,,,,,,,,,,,,,,,,,,,,,,

or like this:
410000000000001,X,dummy, data,separated,by,comma,,,,,,,,
410000000000001,X,,,,,,,,,,,,,,,,,,,,,,,,,,
410000000000001,X,,,,,,,,,,,,,,,,,,,,,,,,,,
410000000000001,X,,,,,,,,,,,,,,,,,,,,,,,,,,
410000000000002,X,,dummy, data,separated,by,comma,,,,,,,,
410000000000003,X,,dummy, data,separated,by,comma,,,,,,,,
410000000000003,X,,,,,,,,,,,,,,,,,,,,,,,,,,

the final result depends on how the sort function works because i need the sort to be done by the value 4100000000000XXX.

And for final result i need this:
1,410000000000001,,dummy, data,separated,by,comma,,,,,,,,
2,410000000000001,,dummy, data,separated,by,comma,,,,,,,,
3,410000000000001,,dummy, data,separated,by,comma,,,,,,,,
...
7,410000000000001,,dummy, data,separated,by,comma,,,,,,,,
1,410000000000002,,dummy, data,separated,by,comma,,,,,,,,
2,410000000000002,,dummy, data,separated,by,comma,,,,,,,,


which means, i need to sort the files by the number 4100000000000XX in order to process all the information for the same number sequentially ( it goes to a oracle database after processing)

I hope this explains my situation best and that you can help me.

best regards,
Ricardo Tomás

Each file has one or more lines
Each line has one instance of 410...XXX followed by one or more fields

You want one resulting file, with each line:
<a number>, followed by the 410....XXX, followed by fields
<a different number>, followed by the 410....XXX, followed by fields.


Where are you coming up with the number that you want at the front of each line for your final text file?
How do you want the common numbered lines (from different files) to be grouped?
How do you want the fields in the common numbered lines (from different files) to be grouped? This will be more difficult unless you can provide information about how each file has it's fields ordered:

Code:

file1
:16digitnumber:, :field1:, :field2:, :field3:

file2
:16digitnumber:, :field4:, :field5:, field6

file3
:field7:, :16digitnumber:, :field1:, field8:

...

Consolidated file
:somenumber:, :16digitnumber:, :field1:, :field2:, :field3:
:somenumber:, :16digitnumber:, :field4:, :field5:, :field6:
:somenumber:, :16digitnumber:, :filed7:, :field8:
...
and so on....