Skip to main content

Questions tagged [amazon-s3]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
35 views

Is it impossible to overwrite a 0byte file on NFS?

I have set up NFS using AWS Storage Gateway and File Share on a Mac environment. When using a specific program to create 0byte files on this NFS, if a file with the same name already exists, the ...
derick-park's user avatar
0 votes
0 answers
248 views

xargs cat a file with aws cli (amazon s3 move)

I have a file called file.csv, which is a list of the files (or more precisely, file paths path in an S3 Bucket) that need to be moved to another folder in s3. This file lists 53,00,000 files. I have ...
hanu's user avatar
  • 1
0 votes
0 answers
6k views

aws s3 copy failing with fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Copying a file from S3 bucket to EC2 Instance.EC2 Instance is in the same region us-east-2 as S3 Bucket in.S3 Bucket is ACL disabled and Allowed for public access. # /usr/local/bin/aws s3 cp s3://...
dbadmin's user avatar
  • 23
0 votes
1 answer
200 views

How to backup many large files to single compressed file on S3

I have an application that has many thousands of files totaling over 10TB. I'd need to backup this data somewhere (probably to AWS S3). I'd like to: compress data being backed up save the backup as a ...
nick314's user avatar
0 votes
1 answer
21 views

Flask Systemd Service not able open video file under s3 bucket mounted folder

My flask app is running as service but it is not able to read video file from s3bucket mounted folder. OpenCV does not throw error when it fails to load but returns 0 frame array. So we understand ...
nerdicsapo's user avatar
0 votes
1 answer
746 views

Include Variable in S3 CP command for include block

I have a scenario where I want to include some files names in S3 cp command but it keeps failing. Below is my command:- DATE=$(date '+%Y%m%d') /usr/local/bin/aws s3 cp "/test/copy" "s3:/...
Ashmit's user avatar
  • 1
0 votes
1 answer
155 views

Uploading to S3 from memory and benchmarking it

I am running some very basic time commands on an S3 read/write. The problem is, I dont want it to be affected by system IO, and want to bench it from memory. A friend has suggested to use /dev/null as ...
Stat.Enthus's user avatar
1 vote
0 answers
28 views

Are there commercial/freeware/script systems that could back up (daily/weekly/monthly) to S3?

I used to backup 10 important machines (CentOS/RHEL) using rsnap to physical HDDs. I'd like to migrate this backup so its target would be an AWS/S3 bucket. Do I have to reinvent the wheel, or are ...
boardrider's user avatar
1 vote
0 answers
343 views

AWS libcrypto resolve messages seen when using a boto3 library, apparently after an update

I'm using the s4cmd package in Python which in turn uses boto3 to communicate with a (non Amazon) S3 service. I've started seeing these warning messages on stderr. I believe this happened after an ...
David Parks's user avatar
  • 1,150
0 votes
1 answer
3k views

Filter files on a s3 bucket based on the file name

I have an s3 bucket which has files named as such: example_test_20200612010000 example_test_20200612020000 example_test_20200612020000 example_control_20200612010000 example_control_20200612020000 ...
user3874530's user avatar
0 votes
1 answer
75 views

How to store and expand the variable in a command?

I am trying to get the size of a S3 Folder and store it in a INFLUXDB Below is my code #!/bin/bash -x FILESIZE=$(mktemp) s3cmd du -r "s3://path1/path2/" | awk '{print $1}' >> "$FILESIZE" cat "...
clarie's user avatar
  • 95
0 votes
2 answers
1k views

I want to download the latest 150 files from S3 Spaces

I want to download the latest 150 files from S3 Spaces. I used this command s3cmd get s3://obs/site1/uploads/large/ /home/ankit -r | tail -n150 but it does not do what I want; instead, it starts ...
Ankit JaiSwal's user avatar
1 vote
3 answers
1k views

How to pass each line of an output file as an argument to a for loop in the same bash script?

am trying to write a bash script to get the total size of sub folders in a S3 bucket. My bucketpath s3://path1/path2/subfolders Inside the path2 folder i have many sub-folder like 2019_06 2019_07 ...
clarie's user avatar
  • 95
3 votes
2 answers
2k views

Get the size of a S3 Bucket's sub-folder through bash script

I am trying to write a bash script to get the total size of sub folders in a S3 bucket. My bucketpath s3://path1/path2/subfolders Inside the path2 folder i have many sub-folder like 2019_06 2019_07 ...
clarie's user avatar
  • 95
3 votes
2 answers
1k views

How to sort s3cmd ls output by date?

I want to sort directories in an s3 storage by the date embedded in their name. When I run s3cmd ls s3://xyz/private/backups/mails/daily/ | awk '{print $2}' it lists the directories like s3://xyz/...
Pankaj Jha's user avatar
1 vote
1 answer
198 views

Weird aws sync behavior on sigterm

This is baffling behaviour to me. I send a terminate signal to my script while aws s3 sync command is running, and despite me handling the sigterm error, the error trap is also triggered by aws sync ...
alexfvolk's user avatar
  • 183
3 votes
1 answer
2k views

egrep in bash script not working

I have this set of commands : s3ls="aws s3 ls $ssyss3/ --recursive --human-readable --summarize" egrepCommand="egrep '$currentDate|Total'" totalSize=$(echo $s3ls|$egrepCommand| awk -F 'Total Size:' '{...
user352882's user avatar
3 votes
1 answer
123 views

Find files then move them and rename them at the same time?

Currently I use three steps to gzip some static assets and then use s3cmd to an S3 bucket (technically it's a Digital Ocean Spaces bucket). Here's what I do: $ find . -type f -name '*.css' | xargs -I{...
cfx's user avatar
  • 133
0 votes
2 answers
5k views

Find specific folder path in s3 bucket

I'm searching a folder in s3 bucket using this command aws s3 ls s3://bucketname/dir1/dir2/dir3 --recursive | grep -i 'dir3' It's getting results like dir1/dir2/dir3/1/aaa.txt dir1/dir2/dir3/1/bbb....
Jaimin Gosai's user avatar
-1 votes
1 answer
339 views

How to switch to the last added folder in that particular folder in S3 bucket

I have a **S3 bucket in which starting of every month ... will be added. And in the month lder daily **new files will be added .
disha's user avatar
  • 17
10 votes
3 answers
30k views

How can we generate the session token in aws

i have aws access key and secret key with me. i wanted session token to be updated in aws credential file (~/.aws/credentials), how will i get it? I want them to be generated in command line.
Vandhana's user avatar
  • 333
8 votes
3 answers
6k views

Is it possible to get aws account id with only aws access key and secret key in command line (CLI)

Is it possible to get aws account id with only aws access key and secret key in command line (CLI) I have access key and secret key with me. Is it possible to get the account id using those in ...
Vandhana's user avatar
  • 333
1 vote
3 answers
15k views

Only copy files from a particular date from s3 storage

I would like to only copy files from S3 that are from today out of a certain bucket with 100s of files. I tried the following: $ aws s3 ls s3://cve-etherwan/ --recursive --region=us-west-2 | grep ...
stdcerr's user avatar
  • 2,069
1 vote
1 answer
2k views

Limiting recursive 'aws s3 ls' searches by number of items in the folder

Here's the scenario: I need to recursively search through tons and tons of folders and subfolders and spit out the results to a log file using the ls command, BUT, I need to stop searching a folder ...
Sidereal's user avatar
  • 111
2 votes
3 answers
2k views

autofs : dynamic mounting rule for s3 bucket

I successfully implement autofs to automatically mount s3 bucket to the server which running ubuntu server 14.04.5 by following this tutorials. but the number of bucket (that needs to automatically ...
kahidna's user avatar
  • 21
4 votes
1 answer
7k views

xargs with aws cli (amazon s3 copy)

I am trying to speed up the process of copying hundreds of file between 2 S3 buckets in my account. Bucket1 dt=2017-01-01/ file1, file2, file3 ... file100 .. .. dt=2017-01-31/ Each file is a ~...
Abi's user avatar
  • 143
0 votes
1 answer
262 views

Goodsync (gsync) linux always overwriting files to S3 (cant identify similarity)

I have been using Linux gsync, as described below, by creating the jobs in OSX (local dir to S3) and then exporting the tix file out. That more-or-less seemed to work. HOWEVER--- The gsync on Linux ...
BurningKrome's user avatar
1 vote
1 answer
3k views

How to test speed of an s3 bucket mounted via s3fs-fuse?

I have an s3 share that's mounted via [s3fs-fuse][1] and would like to run some speed tests to compare throughput on DreamHost's DreamObjects vs Amazon S3. Everything is mounted and working just fine ...
ylluminate's user avatar
11 votes
5 answers
51k views

Check if file exists in S3 Bucket

This directory /data/files/ has thousands files like: 1test 2test 3test [...] 60000test 60001test I'm also sending them to a S3 Bucket (AWS), using AWS CLI. However, sometimes the S3 bucket can be ...
Patrick B.'s user avatar
1 vote
3 answers
633 views

Counting objects from AWS and warning if too many

I'm trying to create bash script. When I run the AWS command I get a "Total Objects" on the bottom of the results. I want to trigger a warning if its greater than 50 objects. aws s3 ls test.domain....
user209785's user avatar
0 votes
1 answer
1k views

S3 head object but signature does not match

I want to use curl to head objects in s3. code: s3key=xxxxx... (s3 key) s3skey=xxxxxx...(secret key) bucket=xxx file=/xx.txt date=$(date +"%a, %d %b %Y %T %z") string="HEAD\n\n\n$date\n/$bucket$file"...
yayaya's user avatar
  • 79
3 votes
1 answer
2k views

How to get private IP address of EC2 after spinning the AWS from local machine/ at Jenkins outside AWS

Assume we already spin the AWS using cloudformation plugin from Jenkins outside AWS and now, how to get private IP address after spinning the AWS at my local machine/jenkins using any API methods? I ...
user176867's user avatar
1 vote
0 answers
82 views

some characters show after put objects on amazon s3

I use libcurl to put objects on amazon s3 service. Here is my original contents in a.txt: aaaaaaaaaaaaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbbbbbbb ...
yayaya's user avatar
  • 79
8 votes
4 answers
18k views

Search inside s3 bucket with logs

How to search for a string inside a lot of .gz files in Amazon S3 bucket subfolder? I tried to mount it via s3fs and zgrep but it's sooooo slow. Do you use any other methods? Maybe is there any ...
Michal_Szulc's user avatar
5 votes
1 answer
3k views

Duplicity + Amazon S3 Glacier. How Much "Unfreezing" to Resume Backup

It's been a couple of months since I last did a Duplicity backup [to Amazon S3] and, in the meantime thanks to an automatic 30-day rule I have on my Amazon buckets, my S3 backup has been moved from ...
stíobhart's user avatar
0 votes
2 answers
215 views

browse local S3 storage (not Amazon) with command line tool

I wish to browse, put and get files in local S3 storage from vendor EMC Atmos Cloud Storage I have a RHEL7 and wish to do it on command line. We have no flat structure (like Amazon) we use ...
Sybil's user avatar
  • 1,923
1 vote
1 answer
448 views

Can set up a cron job to copy my server to Amazon S3

I have an Ubuntu 12.04 install on physical hardware. I will then install Geonode on it. The system will be running at a place with variable network capacity. My colleagues will add data and should ...
ragnvald's user avatar
  • 111
5 votes
2 answers
14k views

Unable to pass a bash variable as a python argument in bash

For some reason I cannot pass the bash variable $FOLDER as a python argument on the following code. The python script downloads some files from amazon s3. Bash script: #!/bin/bash FOLDER=$(./aws get ...
Bris's user avatar
  • 61
1 vote
0 answers
573 views

Storing Web Session on S3 bucket

I have a cluster of web servers which store web session related information in a NFS shared directory. We are trying to implement an AWS version of our product where we could use S3 bucket to store ...
mezi's user avatar
  • 942
2 votes
1 answer
1k views

s3fs complains about SSH key or SSL cert - how to fix?

I downloaded and installed s3fs 1.73 on my Debian Wheezy system. The specific steps I took were, all as root: apt-get -u install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-...
user's user avatar
  • 29.2k
1 vote
0 answers
924 views

How to make a directory public and private in a single bucket

I am using amazon S3 bucket to store my data in two different directories in single bucket, I want that my one directory should be globally accessible and another directory should be private. Is there ...
Nitesh B.'s user avatar
  • 573