Aws S3 Check If File Exists Based On A Conditional Path

I would like to check if a file exists in a separate directory of the bucket if a given file exists. I have the following directory structure- import boto3 s3 = boto3.resource('s3

Solution 1:

I ended up using this which gave a little cleaner code

import boto3
s3client = boto3.client('s3')

defall_file_exist(bucket, prefix, fileN):
    fileFound = False
    fileConditionFound = False
    theObjs = s3client.list_objects_v2(Bucket=bucket, Prefix=prefix)
    forobjectin theObjs['Contents']:
        ifobject['Key'].endswith(fileN+'_condition.jpg') :
            fileConditionFound = Trueifobject['Key'].endswith(fileN+".jpg") :
            fileFound = Trueif (fileFound and fileConditionFound) : 
        returnTruereturnFalse

all_file_exist("bucket","folder1", "test")

Solution 2:

It is not possible to specify an object key via a wildcard.

Instead, you would need to do a bucket listing (which can be against the whole bucket, or within a path) and then perform your own logic for identifying the file of interest.

If the number of objects is small (eg a few thousand), the list can be easily retrieved and kept in memory for fast comparison in a Python list.

If there are millions of objects, you might consider using Amazon S3 Inventory, which can provide a daily CSV file that lists all objects in the bucket. Using such a file would be faster than scanning the bucket itself.