copying file information using awk & grep


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting copying file information using awk & grep
# 1  
Old 10-09-2011
copying file information using awk & grep

Hi,

TASK 1:
I have been using this code to print the information of files kept at "/castor/cern.ch/user/s/sudha/forPooja" in some text file name FILE.txt.
Code:
rfdir /castor/cern.ch/user/s/sudha/forPooja | grep data | awk '{print "rfio:///castor/cern.ch/user/s/sudha/forPooja/"$9}' > FILE.txt

Need to mention that "rfdir" is special command in my work. It works same as "ls".

Code, which I am working.
I have some script :
Code:
#!/bin/bash
SOURCEPATH=$1
TARGETPATH=$2
GREP=$3
echo "Copying \"$1 | grep $3\" to $2"
for FILE in `nsls $SOURCEPATH | grep $GREP`
do
        if [ -f $TARGETPATH/$FILE ]; then
                echo "Skipping $FILE."
        else
                echo "Copying $FILE ..."
                awk '{print "$2"$9}' > $2
#               rfcp $SOURCEPATH/$FILE $TARGETPATH
        fi
done

TASK 2 :
In addition, I have a script which does the rest of the part
Code:
#!/bin/bash

split -10 **.txt chunk
i=0
for file in chunk*
do
    ((i=i+1))
    new_file="datafilename"$i".list"

    #sed '/^$/d' $file > $new_file
    #or use the perl command
    perl -0pe 's/\n$//' $file >  $new_file

    #to remove the processed file
    rm -rf $file
    echo 'succesful'
done

Here in this code, I am using that FILE.txt .

It would be really useful if I have single script doing both.
Thanks in advance
pooja.
# 2  
Old 10-09-2011
Task 1:
Do you want to print the filenames having the string "data" in the filename with rfio:///castor/cern.ch/user/s/sudha/forPooja/ as prefix into FILE.txt?

And are the files in Task1 and Task2 related in anyway?


And what is the purpose of this?
Code:
awk '{print "$2"$9}' > $2

what are the input arguments to the script?

Task1 is not at all clear! Smilie

--ahamed

---------- Post updated at 01:21 AM ---------- Previous update was at 01:17 AM ----------

Something like this?

Code:
#!/bin/bash

SOURCEPATH=$1
TARGETPATH=$2
search=$3
for file in $SOURCEPATH/*${search}* 
do
        file_name=$(basename $file)
        test -f $TARGETPATH/$file_name && continue
        echo "rfio://$SOURCEPATH/$file_name" >> FILE.txt
        cp $SOURCEPATH/$file_name $TARGETPATH
done

split -10 **.txt chunk
i=0
for file in chunk*
do
    ((i=i+1))
    new_file="datafilename"$i".list"

    #sed '/^$/d' $file > $new_file
    #or use the perl command
    perl -0pe 's/\n$//' $file >  $new_file

    #to remove the processed file
    rm -rf $file
    echo 'succesful'
done

Everything based on assumptions!

--ahamed

Last edited by ahamed101; 10-09-2011 at 05:43 AM..
# 3  
Old 10-09-2011
Hi,

The files kept in the "/castor/cern.ch/user/s/sudha/forPooja" has field :
Code:
[lxplus415] /afs/cern.ch/user/p/pooja/test > rfdir /castor/cern.ch/user/s/sudha/forPooja
-rw-r--r--   1 cms003   zh                  969064642 Jul 29 17:08 data_10_0_dBw.root
-rw-r--r--   1 cms003   zh                 1021157382 Jul 29 18:46 data_11_0_rRG.root
-rw-r--r--   1 cms003   zh                 1136693129 Jul 30 09:33 data_13_0_Ljr.root
-rw-r--r--   1 cms003   zh                 1136693192 Jul 30 12:50 data_13_2_VEo.root
-rw-r--r--   1 cms003   zh                  458432186 Jul 30 04:58 data_14_0_5TU.root
-rw-r--r--   1 cms003   zh                   75970008 Jul 29 14:17 data_15_0_WKR.root
-rw-r--r--   1 cms003   zh                 1279818743 Jul 30 14:04 data_16_1_Ohw.root
-rw-r--r--   1 cms003   zh                  573268613 Jul 29 16:11 data_18_0_znC.root
-rw-r--r--   1 cms003   zh                  692655767 Jul 29 22:17 data_19_0_KMo.root

I want the information written in FILE.txt is like:
Code:
rfio:///castor/cern.ch/user/s/sudha/forPooja/data_10_0_dBw.root
rfio:///castor/cern.ch/user/s/sudha/forPooja/data_19_0_KMo.root

In awk comand
Code:
awk '{print "$2"$9}' > $2

$9 is the column for data**.root information and same for $2.

Yes you are right with this statement.
"Do you want to print the filenames having the string "data" in the filename with rfio:///castor/cern.ch/user/s/sudha/forPooja/ as prefix into FILE.txt?"

hope it is clear now.
Many thanks for reply..
pooja
# 4  
Old 10-09-2011
Isn't this working?
Code:
SOURCEPATH=$1
TARGETPATH=$2
search=$3
for file in $SOURCEPATH/*${search}* 
do
        file_name=$(basename $file)
        test -f $TARGETPATH/$file_name && continue
        echo "rfio://$TARGETPATH/$file_name" >> FILE.txt
        cp $SOURCEPATH/$file_name $TARGETPATH
done

--ahamed

Last edited by ahamed101; 10-09-2011 at 05:52 AM..
# 5  
Old 10-09-2011
Hi,

No, the code is not working
Code:
#!/bin/bash

SOURCEPATH=$1
TARGETPATH=$2
search=$3
for file in $SOURCEPATH/*${search}* 
do
        file_name=$(basename $file)
        test -f $TARGETPATH/$file && continue
        echo "rfio://$SOURCEPATH/$file_name" >> FILE.txt
        cp $SOURCEPATH/$file_name $TARGETPATH
done

Error mesage is following
Code:
[lxplus427] /afs/cern.ch/user/p/pooja/test > ./pooja.sh /castor/cern.ch/user/s/sudha/forPooja FILE.txt data
cp: cannot stat `/castor/cern.ch/user/s/sudha/forPooja/*data*': No such file or directory
[lxplus427] /afs/cern.ch/user/p/pooja/test >

Actually, the problem is with the source directory. It is special one and normal command does not work there. As i mentioned "rfdir == ls" similarly, "rfcp == cp"

I have some hint, find below the working code (1.sh) which copy the root files kept in the "/castor/cern.ch/user/s/sudha/forPooja" to some target folder.
Code:
#!/bin/bash
SOURCEPATH=$1
TARGETPATH=$2
GREP=$3

echo "Copying \"$1 | grep $3\" to $2"

for FILE in `nsls $SOURCEPATH | grep $GREP`
do
        if [ -f $TARGETPATH/$FILE ]; then
                echo "Skipping $FILE."
        else
                echo "Copying $FILE ..."
                rfcp $SOURCEPATH/$FILE $TARGETPATH
        fi
done

I execute it following way
[code]
$./1.sh /castor/cern.ch/user/s/sudha/forPooja/ data /afs/cern.ch/user/p/pooja/

Thanks
pooja..

---------- Post updated at 10:37 AM ---------- Previous update was at 03:54 AM ----------

Hi,
I have a request, could you please explain me the meaning of these command lines.
Code:
for file in $SOURCEPATH/*${search}* 
do
        file_name=$(basename $file)
        test -f $TARGETPATH/$file_name && continue
        echo "rfio://$TARGETPATH/$file_name" >> FILE.txt
        cp $SOURCEPATH/$file_name $TARGETPATH
done


I did not understand the usage of these 2 lines:
Code:
  file_name=$(basename $file)
        test -f $TARGETPATH/$file_name && continue

waiting..
pooja..
# 6  
Old 10-09-2011
Can you tell me where the data files will be? And where you want to move it?

Following code will check if the data file is present at target folder, if so it will continue with the next file.
Code:
test -f $TARGETPATH/$file_name && continue

--ahamed

---------- Post updated at 09:10 AM ---------- Previous update was at 09:08 AM ----------

You will need to explain the task 1 more with more clarity.
What I have understood is that, there are data files at the source directory, you want take each data file from the source directory and check if it present in the target directory. If not present copy it to target and create a FILE.txt file with rfio://... entry.

--ahamed
# 7  
Old 10-09-2011
Hi,
The source area from where I am trying to copy the file information is not regular one. As I already mentioned that usual terminal commands does nor work there.


Thanks for ur help..
pooja..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed / awk / grep to extract information from log

Hi all, I have a query that runs that outputs data in the following format - 01/09/12 11:43:40,ADMIN,4,77,Application Group Load: Name(TESTED) LoadId(5137-1-0-1XX-15343-15343) File(/dir/dir/File.T03.CI2.RYR.2012009.11433350806.ARD) InputSize(5344) OutputSize(1359) Rows(2) Time(1.9960)... (8 Replies)
Discussion started by: jeffs42885
8 Replies

2. Shell Programming and Scripting

Include information in certain columns using grep and awk

HI all, I have data in a file that looks like this: 1 HOW _ NNP NNP _ 3 nn _ _ 2 DRUGS _ NNP NNP _ 3 nn _ _ 3 ACT _ NNP NNP _ 0 null _ _ 4 : _ ... (3 Replies)
Discussion started by: owwow14
3 Replies

3. UNIX for Dummies Questions & Answers

Grep? - using a file of terms to search another file when the information is on a different line

I have a flat file that looks like this, let's call it Chromosome_9.txt: FT /Gene_Name="Guanyl-Acetylase 9" FT /Gene_Number"36952" FT /Gene_Name="Endoplasmic Luciferase" FT /Gene_Number"36953" FT ... (4 Replies)
Discussion started by: Twinklefingers
4 Replies

4. Shell Programming and Scripting

Copying the Header & footer Information to the Outfile.

Hi I am writing a perl script which checks for the specific column values from a file and writes to the OUT file. So the feed file has a header information and footer information. I header information isaround107 lines i.e. Starts with START-OF-FILE ....... so on .... ... (11 Replies)
Discussion started by: filter
11 Replies

5. Shell Programming and Scripting

Grep'ing information from a log file on SUN OS 5

Hi Guys, I'm trying to write an script that will be launched by a user. The script will look at a log file and check for alerts with the date (supplied by user) and a machine's hostname (also supplied by the user). I'm trying to get the output formatted just like the log file. The logfile looks... (5 Replies)
Discussion started by: illgetit
5 Replies

6. Shell Programming and Scripting

Copying Information from One File to Another File in Shell

Hello, I'm new to scripting and I need help moving text from one file to another file. Here are examples what the files look like. File 1: Ac-223 2.10m A 1 0 0 0 Fr-219 358 9.9000E-01 0 0.0 0 0.0 0 0.0... (1 Reply)
Discussion started by: tamachan414
1 Replies

7. Shell Programming and Scripting

awk (gawk) grep & columns

Hi, I'm working with gawk (on DOS) today. A goal is: find a string for-instance '123', cut a line in two columns and write second one. The problem is: command line works OK, awk file doesn't. But I would like to work with file because there are many strings to find. input: line command: awk... (4 Replies)
Discussion started by: frajer
4 Replies

8. Shell Programming and Scripting

grep & awk

Hi all, I'm figuring on how to grep only specific data I want below: Bin Total % ----- ------- ----- 1 15 42.9 Bin Total % ----- ------- ----- 2 15 ... (3 Replies)
Discussion started by: *Jess*
3 Replies

9. UNIX for Dummies Questions & Answers

Copying file names returned from a grep command into another directory

When I do the following : grep -l "string" *, I get a list of file names returned. Is there a way to copy the files returned from the list into another directory ?. Thanks. (4 Replies)
Discussion started by: Kartheg
4 Replies
Login or Register to Ask a Question