Override S3 Endpoint Using Boto3 Configuration File

OVERVIEW: I'm trying to override certain variables in boto3 using the configuration file (~/aws/confg). In my use case I want to use fakes3 service and send S3 requests to the loca

Solution 1:

You cannot set host in config file, however you can override it from your code with boto3.

importboto3session= boto3.session.Session()

s3_client = session.client(
    service_name='s3',
    aws_access_key_id='aaa',
    aws_secret_access_key='bbb',
    endpoint_url='http://localhost',
)

Then you can interact as usual.

print(s3_client.list_buckets())

Solution 2:

boto3 only reads the signature version for s3 from that config file. You may want to open a feature request, but for now here is how you can address a custom endpoint:

import boto3
from botocore.utils importfix_s3_hostresource= boto3.resource(service_name='s3', endpoint_url='http://localhost')
resource.meta.client.meta.events.unregister('before-sign.s3', fix_s3_host)

That bit about the meta is important because boto3 automatically changes the endpoint to your_bucket_name.s3.amazonaws.com when it sees fit 1. If you'll be working with both your own host and s3, you may wish to override the functionality rather than removing it altogether.

Solution 3:

using boto3 resource:

import boto3

# use third party object storage
s3 = boto3.resource('s3', endpoint_url='https://URL:443',
  aws_access_key_id = 'AccessKey',
  aws_secret_access_key = 'SecertKey')

# Print out bucket namesfor bucket in s3.buckets.all():
 print(bucket.name)

Solution 4:

Another way:

importboto3s3client= boto3.client('s3', endpoint_url='http://X.X.x.X:8080/',
        aws_access_key_id = 'XXXXXXX',
        aws_secret_access_key = 'XXXXXXXX')
bucket_name = 'aaaaa'
s3client.create_bucket(Bucket=bucket_name)

Solution 5:

I've just submitted a PR to boto3 to add an env var to allow you to override the endpoint_url if you need to use a dependent module which uses boto3 (in which case you may not be able to modify the call to boto3.client directly)

https://github.com/boto/boto3/pull/2746

https://github.com/rwillmer/boto3