Questions tagged [amazon-s3]
The amazon-s3 tag has no usage guidance.
41
questions
0
votes
0
answers
12
views
Contabo's Object Storage - Object Lock Retention (AWS S3 compatible)
I'm having difficulty with Contabo's Object Storage.
With a user with permission: "S3 Object Storage Read and Write" I configured a aws profile called 'backup'.
With this profile, I ran this ...
- 131
0
votes
0
answers
102
views
aws s3 copy failing with fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
Copying a file from S3 bucket to EC2 Instance.EC2 Instance is in the same region us-east-2 as S3 Bucket in.S3 Bucket is ACL disabled and Allowed for public access.
# /usr/local/bin/aws s3 cp s3://...
- 23
0
votes
1
answer
96
views
How to backup many large files to single compressed file on S3
I have an application that has many thousands of files totaling over 10TB.
I'd need to backup this data somewhere (probably to AWS S3).
I'd like to:
compress data being backed up
save the backup as a ...
- 3
0
votes
0
answers
32
views
sending presigned s3 url from server to slack
I want to push my generated url to slack, is there any possibilites to push?
curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' ...
0
votes
1
answer
12
views
Flask Systemd Service not able open video file under s3 bucket mounted folder
My flask app is running as service but it is not able to read video file from s3bucket mounted folder. OpenCV does not throw error when it fails to load but returns 0 frame array. So we understand ...
- 101
0
votes
1
answer
474
views
Include Variable in S3 CP command for include block
I have a scenario where I want to include some files names in S3 cp command but it keeps failing.
Below is my command:-
DATE=$(date '+%Y%m%d')
/usr/local/bin/aws s3 cp "/test/copy" "s3:/...
- 1
0
votes
1
answer
117
views
Uploading to S3 from memory and benchmarking it
I am running some very basic time commands on an S3 read/write. The problem is, I dont want it to be affected by system IO, and want to bench it from memory. A friend has suggested to use /dev/null as ...
- 101
1
vote
0
answers
28
views
Are there commercial/freeware/script systems that could back up (daily/weekly/monthly) to S3?
I used to backup 10 important machines (CentOS/RHEL) using rsnap to physical HDDs.
I'd like to migrate this backup so its target would be an AWS/S3 bucket.
Do I have to reinvent the wheel, or are ...
- 240
1
vote
0
answers
273
views
AWS libcrypto resolve messages seen when using a boto3 library, apparently after an update
I'm using the s4cmd package in Python which in turn uses boto3 to communicate with a (non Amazon) S3 service.
I've started seeing these warning messages on stderr. I believe this happened after an ...
- 1,060
0
votes
1
answer
2k
views
Filter files on a s3 bucket based on the file name
I have an s3 bucket which has files named as such:
example_test_20200612010000
example_test_20200612020000
example_test_20200612020000
example_control_20200612010000
example_control_20200612020000
...
0
votes
1
answer
63
views
How to store and expand the variable in a command?
I am trying to get the size of a S3 Folder and store it in a INFLUXDB
Below is my code
#!/bin/bash -x
FILESIZE=$(mktemp)
s3cmd du -r "s3://path1/path2/" | awk '{print $1}' >> "$FILESIZE"
cat "...
- 85
0
votes
2
answers
700
views
I want to download the latest 150 files from S3 Spaces
I want to download the latest 150 files from S3 Spaces. I used this command
s3cmd get s3://obs/site1/uploads/large/ /home/ankit -r | tail -n150
but it does not do what I want; instead, it starts ...
- 101
1
vote
3
answers
1k
views
How to pass each line of an output file as an argument to a for loop in the same bash script?
am trying to write a bash script to get the total size of sub folders in a S3 bucket.
My bucketpath s3://path1/path2/subfolders
Inside the path2 folder i have many sub-folder like
2019_06
2019_07
...
- 85
3
votes
2
answers
1k
views
Get the size of a S3 Bucket's sub-folder through bash script
I am trying to write a bash script to get the total size of sub folders in a S3 bucket.
My bucketpath s3://path1/path2/subfolders
Inside the path2 folder i have many sub-folder like
2019_06
2019_07
...
- 85
3
votes
2
answers
941
views
How to sort s3cmd ls output by date?
I want to sort directories in an s3 storage by the date embedded in their name.
When I run
s3cmd ls s3://xyz/private/backups/mails/daily/ | awk '{print $2}'
it lists the directories like
s3://xyz/...
- 133
1
vote
1
answer
163
views
Weird aws sync behavior on sigterm
This is baffling behaviour to me. I send a terminate signal to my script while aws s3 sync command is running, and despite me handling the sigterm error, the error trap is also triggered by aws sync ...
- 163
3
votes
1
answer
2k
views
egrep in bash script not working
I have this set of commands :
s3ls="aws s3 ls $ssyss3/ --recursive --human-readable --summarize"
egrepCommand="egrep '$currentDate|Total'"
totalSize=$(echo $s3ls|$egrepCommand| awk -F 'Total Size:' '{...
- 31
3
votes
1
answer
104
views
Find files then move them and rename them at the same time?
Currently I use three steps to gzip some static assets and then use s3cmd to an S3 bucket (technically it's a Digital Ocean Spaces bucket). Here's what I do:
$ find . -type f -name '*.css' | xargs -I{...
- 133
0
votes
2
answers
4k
views
Find specific folder path in s3 bucket
I'm searching a folder in s3 bucket using this command
aws s3 ls s3://bucketname/dir1/dir2/dir3 --recursive | grep -i 'dir3'
It's getting results like
dir1/dir2/dir3/1/aaa.txt
dir1/dir2/dir3/1/bbb....
- 11
-1
votes
1
answer
217
views
How to switch to the last added folder in that particular folder in S3 bucket
I have a **S3 bucket in which starting of every month ... will be added. And in the month lder daily **new files will be added .
- 17
8
votes
3
answers
21k
views
How can we generate the session token in aws
i have aws access key and secret key with me. i wanted session token to be updated in aws credential file (~/.aws/credentials), how will i get it?
I want them to be generated in command line.
- 313
8
votes
3
answers
5k
views
Is it possible to get aws account id with only aws access key and secret key in command line (CLI)
Is it possible to get aws account id with only aws access key and secret key in command line (CLI)
I have access key and secret key with me. Is it possible to get the account id using those in ...
- 313
1
vote
3
answers
12k
views
Only copy files from a particular date from s3 storage
I would like to only copy files from S3 that are from today out of a certain bucket with 100s of files. I tried the following: $ aws s3 ls s3://cve-etherwan/ --recursive --region=us-west-2 | grep ...
- 2,037
1
vote
1
answer
2k
views
Limiting recursive 'aws s3 ls' searches by number of items in the folder
Here's the scenario:
I need to recursively search through tons and tons of folders and subfolders and spit out the results to a log file using the ls command, BUT, I need to stop searching a folder ...
- 111
2
votes
3
answers
2k
views
autofs : dynamic mounting rule for s3 bucket
I successfully implement autofs to automatically mount s3 bucket to the server which running ubuntu server 14.04.5 by following this tutorials. but the number of bucket (that needs to automatically ...
- 21
4
votes
1
answer
6k
views
xargs with aws cli (amazon s3 copy)
I am trying to speed up the process of copying hundreds of file between 2 S3 buckets in my account.
Bucket1
dt=2017-01-01/
file1, file2, file3 ... file100
..
..
dt=2017-01-31/
Each file is a ~...
- 143
0
votes
1
answer
244
views
Goodsync (gsync) linux always overwriting files to S3 (cant identify similarity)
I have been using Linux gsync, as described below, by creating the jobs in OSX (local dir to S3) and then exporting the tix file out. That more-or-less seemed to work.
HOWEVER---
The gsync on Linux ...
- 228
1
vote
1
answer
3k
views
How to test speed of an s3 bucket mounted via s3fs-fuse?
I have an s3 share that's mounted via [s3fs-fuse][1] and would like to run some speed tests to compare throughput on DreamHost's DreamObjects vs Amazon S3. Everything is mounted and working just fine ...
- 581
11
votes
5
answers
49k
views
Check if file exists in S3 Bucket
This directory /data/files/ has thousands files like:
1test
2test
3test
[...]
60000test
60001test
I'm also sending them to a S3 Bucket (AWS), using AWS CLI. However, sometimes the S3 bucket can be ...
- 119
1
vote
3
answers
605
views
Counting objects from AWS and warning if too many
I'm trying to create bash script. When I run the AWS command I get a "Total Objects" on the bottom of the results. I want to trigger a warning if its greater than 50 objects.
aws s3 ls test.domain....
- 11
0
votes
1
answer
1k
views
S3 head object but signature does not match
I want to use curl to head objects in s3.
code:
s3key=xxxxx... (s3 key)
s3skey=xxxxxx...(secret key)
bucket=xxx
file=/xx.txt
date=$(date +"%a, %d %b %Y %T %z")
string="HEAD\n\n\n$date\n/$bucket$file"...
- 79
2
votes
1
answer
2k
views
How to get private IP address of EC2 after spinning the AWS from local machine/ at Jenkins outside AWS
Assume we already spin the AWS using cloudformation plugin from Jenkins outside AWS and now, how to get private IP address after spinning the AWS at my local machine/jenkins using any API methods? I ...
- 21
1
vote
0
answers
81
views
some characters show after put objects on amazon s3
I use libcurl to put objects on amazon s3 service. Here is my original contents in a.txt:
aaaaaaaaaaaaaaaaaaaaaaaaaaa
bbbbbbbbbbbbbbbbbbbbbbbbbbb
...
- 79
8
votes
3
answers
16k
views
Search inside s3 bucket with logs
How to search for a string inside a lot of .gz files in Amazon S3 bucket subfolder? I tried to mount it via s3fs and zgrep but it's sooooo slow. Do you use any other methods?
Maybe is there any ...
- 256
5
votes
1
answer
3k
views
Duplicity + Amazon S3 Glacier. How Much "Unfreezing" to Resume Backup
It's been a couple of months since I last did a Duplicity backup [to Amazon S3] and, in the meantime thanks to an automatic 30-day rule I have on my Amazon buckets, my S3 backup has been moved from ...
- 51
0
votes
2
answers
134
views
browse local S3 storage (not Amazon) with command line tool
I wish to browse, put and get files in local S3 storage from vendor EMC Atmos Cloud Storage
I have a RHEL7 and wish to do it on command line. We have no flat structure (like Amazon) we use ...
- 1,633
1
vote
1
answer
425
views
Can set up a cron job to copy my server to Amazon S3
I have an Ubuntu 12.04 install on physical hardware. I will then install Geonode on it. The system will be running at a place with variable network capacity.
My colleagues will add data and should ...
- 111
5
votes
2
answers
14k
views
Unable to pass a bash variable as a python argument in bash
For some reason I cannot pass the bash variable $FOLDER as a python argument on the following code. The python script downloads some files from amazon s3.
Bash script:
#!/bin/bash
FOLDER=$(./aws get ...
- 61
1
vote
0
answers
543
views
Storing Web Session on S3 bucket
I have a cluster of web servers which store web session related information in a NFS shared directory. We are trying to implement an AWS version of our product where we could use S3 bucket to store ...
- 922
2
votes
1
answer
1k
views
s3fs complains about SSH key or SSL cert - how to fix?
I downloaded and installed s3fs 1.73 on my Debian Wheezy system. The specific steps I took were, all as root:
apt-get -u install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-...
- 27.9k
1
vote
0
answers
791
views
How to make a directory public and private in a single bucket
I am using amazon S3 bucket to store my data in two different directories in single bucket, I want that my one directory should be globally accessible and another directory should be private. Is there ...
- 553