Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

To check the file and remove header before copying

Shell Programming and Scripting


Reply    
 
Thread Tools Search this Thread Display Modes
    #8  
Old Unix and Linux 06-15-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 20 August 2017, 1:07 AM EDT
Location: San Jose, CA, USA
Posts: 10,464
Thanks: 533
Thanked 3,660 Times in 3,123 Posts
Quote:
Originally Posted by rohit_shinez View Post
Hi Rudic,

Thanks for your inputs, can this be modified in my code which runs in bash


Code:
something like this

sed '1d' $file > $file

The tail command in RudiC's suggestion copies your source files to their destinations removing "$HDCNT" lines of headers during the copy.

Your code above will do two things:
  • The shell will see the redirection and truncate the file named by the expansion of $file to size 0, and then
  • The sed command will delete the 1st line (of the empty file) named by the expansion of $file and copy the remaining (non-existent) lines to the output.
Do you really want to change your input files before you copy their contents to their destinations, or do you just need to remove the header lines from the copied files? If you don't need to modify the source files:

Code:
sed 1d "$file" > /output/aus/"$file"

will do what you want. If you need to change the source files too:

Code:
sed 1d "$file" > "$file.$$" && cp "$file.$$" "$file" && rm "$file.$$"

is a safe way to do what you want without breaking links to your source file (if any existed).
Sponsored Links
    #9  
Old Unix and Linux 06-15-2017
rohit_shinez rohit_shinez is offline
Registered User
 
Join Date: Jul 2013
Last Activity: 20 August 2017, 10:16 AM EDT
Posts: 172
Thanks: 57
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by Don Cragun View Post
The tail command in RudiC's suggestion copies your source files to their destinations removing "$HDCNT" lines of headers during the copy.

Your code above will do two things:
  • The shell will see the redirection and truncate the file named by the expansion of $file to size 0, and then
  • The sed command will delete the 1st line (of the empty file) named by the expansion of $file and copy the remaining (non-existent) lines to the output.
Do you really want to change your input files before you copy their contents to their destinations, or do you just need to remove the header lines from the copied files? If you don't need to modify the source files:

Code:
sed 1d "$file" > /output/aus/"$file"

will do what you want. If you need to change the source files too:

Code:
sed 1d "$file" > "$file.$$" && cp "$file.$$" "$file" && rm "$file.$$"

is a safe way to do what you want without breaking links to your source file (if any existed).
Hi don,

Before I copy I need to change the source files which ends with _01.txt I.e. To remove the header
Sponsored Links
    #10  
Old Unix and Linux 06-15-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 20 August 2017, 1:07 AM EDT
Location: San Jose, CA, USA
Posts: 10,464
Thanks: 533
Thanked 3,660 Times in 3,123 Posts
Quote:
Originally Posted by rohit_shinez View Post
Hi don,

Before I copy I need to change the source files which ends with _01.txt I.e. To remove the header
Fine. And, since you copy the modified source file to the destination (not move it), when you run your script again a few seconds (or an hour, or day, or week, or month, or year) later, it will update that same file again (this time removing a line of data instead of the header). And then it will copy the updated file again to its destination discarding a line of data there as well. And, after you have run your script enough times, your source and destination files will both be empty.

This doesn't seem like a logical way to handle things to me, but you can do it that way if you want to.
    #11  
Old Unix and Linux 06-15-2017
rohit_shinez rohit_shinez is offline
Registered User
 
Join Date: Jul 2013
Last Activity: 20 August 2017, 10:16 AM EDT
Posts: 172
Thanks: 57
Thanked 0 Times in 0 Posts
Hi don,

Once i have copied i wil remove the files or i will move the files from filepath to temp path and remove the header from that temp path
Sponsored Links
    #12  
Old Unix and Linux 06-15-2017
lightman66 lightman66 is offline
Registered User
 
Join Date: Mar 2017
Last Activity: 12 July 2017, 4:46 AM EDT
Posts: 2
Thanks: 0
Thanked 1 Time in 1 Post
Quote:
Originally Posted by rohit_shinez View Post
Hi Guys,

I have below directory where there are certain files. Something like below

Code:
country_dir

aus_01.txt
nz_01.txt
aus_02.txt
bd.txt

property.txt
aus
nz
bd

I need to remove the header of the file which ends with _01.txt while copying from country directory to another directory

Code which i am using


Code:
for i in `cat property.txt`
do
file=$(ls -t1 $filepath|  grep -i $file)

if [ "$i" == "aus" ]
then
if [ -f $file ]
then

      	cp $file /output/aus/
else 
	echo "No aus files"

fi

elif [ "$i" == "nz" ]
if [ -f $file ]
then

      	cp $file /output/nz/
else 
	echo "No nz files"

fi

fi
....
....
done

in the above code i need to remove the header for the files aus_01.txt
nz_01.txt before copying to output directory. Can the changes be done in same code
Hi,

I'm quite new to scripting too.

just curious if the current code you are using is already working and you wanted to enhanced the code to omit the header in individual file_0X.txt?

may i also check if the header string in each individual *.txt is unique or different?

if its unique, maybe this might work?


Code:
for i in `cat property.txt | grep -v <HEADER>`
do
file=$(ls -t1 $filepath|  grep -i $file)

if [ "$i" == "aus" ]
then
if [ -f $file ]
then

      	cp $file /output/aus/
else 
	echo "No aus files"

fi

elif [ "$i" == "nz" ]
if [ -f $file ]
then

      	cp $file /output/nz/
else 
	echo "No nz files"

fi

fi
....
....
done

just a thought
Sponsored Links
    #13  
Old Unix and Linux 06-16-2017
rohit_shinez rohit_shinez is offline
Registered User
 
Join Date: Jul 2013
Last Activity: 20 August 2017, 10:16 AM EDT
Posts: 172
Thanks: 57
Thanked 0 Times in 0 Posts
Thanks
But yours wont work because there is no header like unique and in property file there is no header string like that i need to remove the header from source file before copying
Sponsored Links
    #14  
Old Unix and Linux 06-16-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 20 August 2017, 1:07 AM EDT
Location: San Jose, CA, USA
Posts: 10,464
Thanks: 533
Thanked 3,660 Times in 3,123 Posts
Quote:
Originally Posted by rohit_shinez View Post
Hi don,

Once i have copied i wil remove the files or i will move the files from filepath to temp path and remove the header from that temp path
Huh? You have to remove the header before you copy the source file and then you will copy the source file and then you will copy the source file (to a country directory) and then you will remove the source file OR you will remove the header before you copy the source file and then you will copy the source file (to a country directory) and then you will move the source file to a temporary file an thend you will remove the header from the temporary file again and then you will remove the temporary file. If either of these multistep processes are interrupted and you restart the process, zero of more lines of data will be lost.

Either of those sound like a lot of extra work when it sounds like you get the same results with MUCH less processing if you just copy the source file except for the header to a country directory and then remove the source file. If this processing is interrupted and restarted, no data will be lost.

Why is it that you want to make this so complicated? Why do you need to remove the header BEFORE copying the file? Why isn't removing the header WHILE you are copying the file sufficient?
The Following User Says Thank You to Don Cragun For This Useful Post:
rohit_shinez (06-17-2017)
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Check for empty line at end of a dynamic header in each file tpk UNIX for Dummies Questions & Answers 6 04-16-2015 03:54 AM
Remove the file content based on the Header of the file i150371485 Shell Programming and Scripting 5 08-03-2012 02:35 AM
Remove last few characters in a file but keeping Header and trailer intact nvuradi Shell Programming and Scripting 2 04-12-2012 02:16 PM
Check header of file, without opening the file lydiaflamp UNIX for Dummies Questions & Answers 2 10-03-2010 05:41 PM
Copy all the files with time stamp and remove header,trailer from file ksrams UNIX for Dummies Questions & Answers 35 07-30-2007 02:15 PM



All times are GMT -4. The time now is 05:07 PM.