Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [amazon-s3]

The tag has no usage guidance.

2
votes
1answer
116 views

egrep in bash script not working

I have this set of commands : s3ls="aws s3 ls $ssyss3/ --recursive --human-readable --summarize" egrepCommand="egrep '$currentDate|Total'" totalSize=$(echo $s3ls|$egrepCommand| awk -F 'Total Size:' '{...
3
votes
1answer
53 views

Find files then move them and rename them at the same time?

Currently I use three steps to gzip some static assets and then use s3cmd to an S3 bucket (technically it's a Digital Ocean Spaces bucket). Here's what I do: $ find . -type f -name '*.css' | xargs -I{...
0
votes
1answer
33 views

Find specific folder path in s3 bucket

I'm searching a folder in s3 bucket using this command aws s3 ls s3://bucketname/dir1/dir2/dir3 --recursive | grep -i 'dir3' It's getting results like dir1/dir2/dir3/1/aaa.txt dir1/dir2/dir3/1/bbb....
-1
votes
1answer
48 views

How to switch to the last added folder in that particular folder in S3 bucket

I have a **S3 bucket in which starting of every month ... will be added. And in the month lder daily **new files will be added .
0
votes
3answers
645 views

How can we generate the session token in aws

i have aws access key and secret key with me. i wanted session token to be updated in aws credential file (~/.aws/credentials), how will i get it? I want them to be generated in command line.
6
votes
2answers
637 views

Is it possible to get aws account id with only aws access key and secret key in command line (CLI)

Is it possible to get aws account id with only aws access key and secret key in command line (CLI) I have access key and secret key with me. Is it possible to get the account id using those in ...
0
votes
1answer
380 views

Only copy files from a particular date from s3 storage

I would like to only copy files from S3 that are from today out of a certain bucket with 100s of files. I tried the following: $ aws s3 ls s3://cve-etherwan/ --recursive --region=us-west-2 | grep ...
0
votes
1answer
398 views

Limiting recursive 'aws s3 ls' searches by number of items in the folder

Here's the scenario: I need to recursively search through tons and tons of folders and subfolders and spit out the results to a log file using the ls command, BUT, I need to stop searching a folder ...
0
votes
1answer
480 views

autofs : dynamic mounting rule for s3 bucket

I successfully implement autofs to automatically mount s3 bucket to the server which running ubuntu server 14.04.5 by following this tutorials. but the number of bucket (that needs to automatically ...
1
vote
1answer
2k views

xargs with aws cli (amazon s3 copy)

I am trying to speed up the process of copying hundreds of file between 2 S3 buckets in my account. Bucket1 dt=2017-01-01/ file1, file2, file3 ... file100 .. .. dt=2017-01-31/ Each file is a ~...
0
votes
1answer
141 views

Goodsync (gsync) linux always overwriting files to S3 (cant identify similarity)

I have been using Linux gsync, as described below, by creating the jobs in OSX (local dir to S3) and then exporting the tix file out. That more-or-less seemed to work. HOWEVER--- The gsync on Linux ...
1
vote
1answer
825 views

How to test speed of an s3 bucket mounted via s3fs-fuse?

I have an s3 share that's mounted via [s3fs-fuse][1] and would like to run some speed tests to compare throughput on DreamHost's DreamObjects vs Amazon S3. Everything is mounted and working just fine ...
5
votes
3answers
14k views

Check if file exists in S3 Bucket

This directory /data/files/ has thousands files like: 1test 2test 3test [...] 60000test 60001test I'm also sending them to a S3 Bucket (AWS), using AWS CLI. However, sometimes the S3 bucket can be ...
0
votes
3answers
336 views

Counting objects from AWS and warning if too many

I'm trying to create bash script. When I run the AWS command I get a "Total Objects" on the bottom of the results. I want to trigger a warning if its greater than 50 objects. aws s3 ls test.domain....
0
votes
1answer
437 views

S3 head object but signature does not match

I want to use curl to head objects in s3. code: s3key=xxxxx... (s3 key) s3skey=xxxxxx...(secret key) bucket=xxx file=/xx.txt date=$(date +"%a, %d %b %Y %T %z") string="HEAD\n\n\n$date\n/$bucket$file"...
1
vote
1answer
964 views

How to get private IP address of EC2 after spinning the AWS from local machine/ at Jenkins outside AWS

Assume we already spin the AWS using cloudformation plugin from Jenkins outside AWS and now, how to get private IP address after spinning the AWS at my local machine/jenkins using any API methods? I ...
1
vote
0answers
46 views

some characters show after put objects on amazon s3

I use libcurl to put objects on amazon s3 service. Here is my original contents in a.txt: aaaaaaaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbbbbbbb ...
5
votes
2answers
6k views

Search inside s3 bucket with logs

How to search for a string inside a lot of .gz files in Amazon S3 bucket subfolder? I tried to mount it via s3fs and zgrep but it's sooooo slow. Do you use any other methods? Maybe is there any ...
3
votes
1answer
2k views

Duplicity + Amazon S3 Glacier. How Much “Unfreezing” to Resume Backup

It's been a couple of months since I last did a Duplicity backup [to Amazon S3] and, in the meantime thanks to an automatic 30-day rule I have on my Amazon buckets, my S3 backup has been moved from ...
0
votes
1answer
70 views

browse local S3 storage (not Amazon) with command line tool

I wish to browse, put and get files in local S3 storage from vendor EMC Atmos Cloud Storage I have a RHEL7 and wish to do it on command line. We have no flat structure (like Amazon) we use ...
1
vote
1answer
387 views

Can set up a cron job to copy my server to Amazon S3

I have an Ubuntu 12.04 install on physical hardware. I will then install Geonode on it. The system will be running at a place with variable network capacity. My colleagues will add data and should ...
3
votes
2answers
8k views

Unable to pass a bash variable as a python argument in bash

For some reason I cannot pass the bash variable $FOLDER as a python argument on the following code. The python script downloads some files from amazon s3. Bash script: #!/bin/bash FOLDER=$(./aws get ...
1
vote
0answers
381 views

Storing Web Session on S3 bucket

I have a cluster of web servers which store web session related information in a NFS shared directory. We are trying to implement an AWS version of our product where we could use S3 bucket to store ...
2
votes
0answers
479 views

s3fs complains about SSH key or SSL cert - how to fix?

I downloaded and installed s3fs 1.73 on my Debian Wheezy system. The specific steps I took were, all as root: apt-get -u install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-...
1
vote
0answers
217 views

How to make a directory public and private in a single bucket

I am using amazon S3 bucket to store my data in two different directories in single bucket, I want that my one directory should be globally accessible and another directory should be private. Is there ...