Shell Programming and Scripting

Hi,

I have a file (named as values) which contains around 10 columns with an unique number in the first column. I wanted to segregate the file into 10 files based on the range of the Unique number. It varies from 0 to 10 Million. So, I need to split the file into 10 files with ranges from 0 to 1 million .... 9 million to 10 million.

I also have a file (named as headeronly) that contains the header which I should use to merge the splitted file with a common header . "awk and cat commands are used inside a for loop". The issue is the script is getting executed and when the splitted files checked only the header is printed but not the columns of the numerical values.

Code used:

Code:

for (i=0;i<=9;i=i+1)
 do
   awk'{if(($i*1000000<$1)&&($1<($i+1)*1000000)){print$0}}' values > $i
   cat headeronly $i > a$i
done

Moderator's Comments:

Please view this code tag video for how to use code tags when posting code and data.

Kindly give suggestion.

Variables do not work inside single quotes.

Code:

$ echo '$i'

$i

$

Instead you input variables into awk like

Code:

awk -v I="$i" '{ ... }'

Also, you can simplify your awk code a lot. A raw expression by itself prints lines whenever true, and doesn't whenever false.

So:

Code:

for i in 0 1 2 3 4 5 6 7 8 9
do
   awk -v I="$i" '(((I*1000000)<$1)&&($1<((I+1)*1000000)))' values > $i
   cat headeronly $i > a$i
done

$i inside the awk script is not $i from the shell script (at least in single-quotes), and awk doesn't use a $ to access most variables ($i will evaluate to a field, 0 in this case).

EDIT: Also, I'm guessing you want the header first into a$i, and append the other values to the same file? What is the $i in the cat for though?

Try something like:

Code:

for (i=0;i<=9;i=i+1)
 do
   cat headeronly $i > a$i
   awk -vidx=$i '{if ((idx*1000000<$1)&&($1<(idx+1)*1000000)){print}}' values >> a$i
done

can you provide sample data of input and output for your requirement

@ raj_saini20

Say a file (named as values) has following data (i have given only 2 columns for simplication) with a header as :

The values of the simulation are as follows with the unique number and potential as

unique number potential

1 5.63
2 9.62
3 0.23
4 5.01
5 6.36
6 2.40
7 5.62
8 1.24
9 4.23
10 7.25
11 1.98
. .
. .
. .
. .
100 7.04

Now , i need to separate above file into 10 files based on unique number with range from 1 to 10 , 11 to 20 ..... 91 to 100 and include the common header to all these 10 files .
I should do this by using "awk and cat" command.

---------- Post updated at 01:09 AM ---------- Previous update was at 12:58 AM ----------

@CarloM : I am just trying to first split the files into 10 number based on their unique number. So , I am first using the "awk" command and put it into a new 10 files with names as 0 1 2 ...9. That's why is use $i in the last of the awk .

Code:

for (i=0;i<=9;i=i+1)
 do
   awk'{if(($i*1000000<$1)&&($1<($i+1)*1000000)){print$0}}' values > $i

Then, i wish to add header to all my newly created files (i.e) to files named 0 1 2 .....9 using the cat command. since , i have referred $i as the new file in the above awk command, i am doing same in the cat command also .

Code:

cat headeronly $i > a$i
done

Try something like this...

Code:

for ((i=0;i<=9;i=i+1))
 do
   cat headeronly $i > a$i
   range_s="10" #Here you can give your range directly so no need to change in the code.. 10 is for 0-100 range..
   awk -v idx="$i" -v RG="$range_s" '{if ((idx*RG<$1)&&($1<=(idx+1)*RG)){print}}' test_file >> a$i   
done

check this if it is working for splitting into 10 files.
if this code is working i will further modify it for insertion of header in all the ten files.
One more thing in how many lines your header including column name is coming

Code:

awk '{for(i=1;i<=10;i++){if(($1>=(i-1)*1000000)&&($1<i*1000000)){print > i".txt"}}}' values.txt