Sponsored Content
Full Discussion: AWS S3 with Bash Script
Top Forums Shell Programming and Scripting AWS S3 with Bash Script Post 302936770 by krux_rap on Friday 27th of February 2015 01:39:37 PM
Old 02-27-2015
AWS S3 with Bash Script

Hello All,

I am trying few stuff on Amazon S3 move files to EC2 (Ubuntu Server)

I am receiving daily 2 files in S# bucket

Code:
XYZ_2015-02-26_ 200.csv
ABC_2015-02-26_ 200.csv

I want to move daily received files to local EC2(Ubuntu server) and rename files to
Code:
XYZ_2015-02-26.csv
ABC_2015-02-26.csv

Instead of Renaming files after moving to EC2 , if possible i want to rename files on S3 itself.

and send an email notification with files received or not on daily basis

Is this possible using Bash? If yes, please guide.

Thanks
Krux
 

4 More Discussions You Might Find Interesting

1. Virtualization and Cloud Computing

Computing in the Clouds with AWS

Tim Bass 07-25-2008 02:34 AM The admin*team at The UNIX Forums*have been considering moving the UNIX and*Linux*Forums to the clouds - the Amazon Web Services (AWS) cloud.* Amazon EC2 is one option to scale the forums, which is a*LAMP application.* Amazon EC2 allows*us to rent dedicated... (3 Replies)
Discussion started by: Linux Bot
3 Replies

2. Virtualization and Cloud Computing

How to grow an EBS Array at AWS

This process was collected by trial and error (mostly error) since most of the documents I found on the 'net were incomplete. The cookbook/walkthrough on how to grow an Array on EBS at AWS... # walkthrough by John Jones (jjones at cirrhus9 dot com) umount /dev/md0 mdadm --stop /dev/md0... (0 Replies)
Discussion started by: Habitual
0 Replies

3. Shell Programming and Scripting

AWS Cloud EBS limit script!

Hi, I am trying to correct the following script to get total EBS usage from Amazon cloud. Can someone please help in correcting it. It appears that its not working properly. t=0; for i in `ec2-describe-volumes | grep VOLUME | grep standard | awk '{print $3}'`; do t=`expr $t + $i`; done;... (2 Replies)
Discussion started by: amar0000
2 Replies

4. Shell Programming and Scripting

How to block first bash script until second bash script script launches web server/site?

I'm new to utilities like socat and netcat and I'm not clear if they will do what I need. I have a "compileDeployStartWebServer.sh" script and a "StartBrowser.sh" script that are started by emacs/elisp at the same time in two different processes. I'm using Cygwin bash on Windows 10. My... (3 Replies)
Discussion started by: siegfried
3 Replies
S3MKBUCKET(1p)						User Contributed Perl Documentation					    S3MKBUCKET(1p)

NAME
s3mkbucket - Create Amazon AWS S3 buckets SYNOPSIS
s3mkbucket [options] [bucket ...] Options: --access-key AWS Access Key ID --secret-key AWS Secret Access Key --acl-short private|public-read|public-read-write|authenticated-read Environment: AWS_ACCESS_KEY_ID AWS_ACCESS_KEY_SECRET OPTIONS
--help Print a brief help message and exits. --man Prints the manual page and exits. --verbose Print a message for each created bucket. --access-key and --secret-key Specify the "AWS Access Key Identifiers" for the AWS account. --access-key is the "Access Key ID", and --secret-key is the "Secret Access Key". These are effectively the "username" and "password" to the AWS account, and should be kept confidential. The access keys MUST be specified, either via these command line parameters, or via the AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_SECRET environment variables. Specifying them on the command line overrides the environment variables. --secure Uses SSL/TLS HTTPS to communicate with the AWS service, instead of HTTP. --acl-short Apply a "canned ACL" to the bucket when it is created. To set a more complex ACL, use the "s3acl" tool after the bucket is created. The following canned ACLs are currently defined by S3: private Owner gets "FULL_CONTROL". No one else has any access rights. This is the default. public-read Owner gets "FULL_CONTROL". The anonymous principal is granted "READ" access. public-read-write Owner gets "FULL_CONTROL". The anonymous principal is granted "READ" and "WRITE" access. This is a useful policy to apply to a bucket, if you intend for any anonymous user to PUT objects into the bucket. authenticated-read Owner gets "FULL_CONTROL" . Any principal authenticated as a registered Amazon S3 user is granted "READ" access. bucket One or more bucket names. As many as possible will be created. A user may have no more than 100 buckets. Bucket names must be between 3 and 255 characters long, and can only contain alphanumeric characters, underscore, period, and dash. Bucket names are case sensitive. Buckets with names containing uppercase characters or underscores are not accessible using the virtual hosting method. Buckets are unique in a global namespace. That means if someone has created a bucket with a given name, someone else cannot create another bucket with the same name. If a bucket name begins with one or more dashes, it might be mistaken for a command line option. If this is the case, separate the command line options from the bucket names with two dashes, like so: s3mkbucket --verbose -- --bucketname ENVIRONMENT VARIABLES
AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_SECRET Specify the "AWS Access Key Identifiers" for the AWS account. AWS_ACCESS_KEY_ID contains the "Access Key ID", and AWS_ACCESS_KEY_SECRET contains the "Secret Access Key". These are effectively the "username" and "password" to the AWS service, and should be kept confidential. The access keys MUST be specified, either via these environment variables, or via the --access-key and --secret-key command line parameters. If the command line parameters are set, they override these environment variables. CONFIGURATION FILE
The configuration options will be read from the file "~/.s3-tools" if it exists. The format is the same as the command line options with one option per line. For example, the file could contain: --access-key <AWS access key> --secret-key <AWS secret key> --secure This example configuration file would specify the AWS access keys and that a secure connection using HTTPS should be used for all communications. DESCRIPTION
Create buckets in the Amazon Simple Storage Service (S3). BUGS
Report bugs to Mark Atwood mark@fallenpegasus.com. Making a bucket that already exists and is owned by the user does not fail. It is unclear whether this is a bug or not. Occasionally the S3 service will randomly fail for no externally apparent reason. When that happens, this tool should retry, with a delay and a backoff. Access to the S3 service can be authenticated with a X.509 certificate, instead of via the "AWS Access Key Identifiers". This tool should support that. It might be useful to be able to specify the "AWS Access Key Identifiers" in the user's "~/.netrc" file. This tool should support that. Errors and warnings are very "Perl-ish", and can be confusing. AUTHOR
Written by Mark Atwood mark@fallenpegasus.com. Many thanks to Wotan LLC <http://wotanllc.com>, for supporting the development of these S3 tools. Many thanks to the Amazon AWS engineers for developing S3. SEE ALSO
These tools use the Net::Amazon:S3 Perl module. The Amazon Simple Storage Service (S3) is documented at <http://aws.amazon.com/s3>. perl v5.10.0 2009-03-08 S3MKBUCKET(1p)
All times are GMT -4. The time now is 09:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy