amazon web services - refactor to limit to use lastModified<=15 - Stack Overflow

Yes, we can use the aws s3api... to filter the keys that is older than 15 days by means of running below command (below returns expected result):

lastModifiedMax=`date -d "15 days ago" '+%Y-%m-%d'`
aws s3api list-objects --bucket $bucket  --query "Contents[?LastModified<='${lastModifiedMax}'][].{object: Key}"

However, we want to use in python boto3, using this sample piece of the script -

Which is, we tried to replace this below lines

  while True:
                # The S3 API response is a large blob of metadata.
                # 'Contents' contains information about the listed objects.
                resp = s3.list_objects_v2(**kwargs)
                for content in resp.get('Contents', []):
                    last_modified_date = content['LastModified']
                    if (
                        content['Key'].endswith(suffixes) and
                        last_modified_rule(last_modified_min, last_modified_date, last_modified_max)
                    ):
                        yield content

With :

    end_date = date.today() - timedelta(days=15)

    for prefix in prefixes:
        kwargs['Prefix'] = prefix
        while True:
            # The S3 API response is a large blob of metadata.
            # 'Contents' contains information about the listed objects.
            resp = s3.list_objects_v2(**kwargs)
            for content in resp.get('Contents', ['?LastModified<='+str(end_date)]):
                    yield content

The above changes did not give expected result, means, returned "whole" keys (whole timetamp).

Please advise what is wrong.

Yes, we can use the aws s3api... to filter the keys that is older than 15 days by means of running below command (below returns expected result):

lastModifiedMax=`date -d "15 days ago" '+%Y-%m-%d'`
aws s3api list-objects --bucket $bucket  --query "Contents[?LastModified<='${lastModifiedMax}'][].{object: Key}"

However, we want to use in python boto3, using this sample piece of the script - https://stackoverflow/a/53877685/27921578

Which is, we tried to replace this below lines

  while True:
                # The S3 API response is a large blob of metadata.
                # 'Contents' contains information about the listed objects.
                resp = s3.list_objects_v2(**kwargs)
                for content in resp.get('Contents', []):
                    last_modified_date = content['LastModified']
                    if (
                        content['Key'].endswith(suffixes) and
                        last_modified_rule(last_modified_min, last_modified_date, last_modified_max)
                    ):
                        yield content

With :

    end_date = date.today() - timedelta(days=15)

    for prefix in prefixes:
        kwargs['Prefix'] = prefix
        while True:
            # The S3 API response is a large blob of metadata.
            # 'Contents' contains information about the listed objects.
            resp = s3.list_objects_v2(**kwargs)
            for content in resp.get('Contents', ['?LastModified<='+str(end_date)]):
                    yield content

The above changes did not give expected result, means, returned "whole" keys (whole timetamp).

Please advise what is wrong.

• amazon-web-services
• boto3

Share Improve this question Follow edited Nov 18, 2024 at 17:48 Hari Addepalli asked Nov 18, 2024 at 17:47 Hari AddepalliHari Addepalli 333 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

never mind - debugged/researched, we can achieve by means of this

           for content in resp.get('Contents',  []):
                if (
                        content['Key'] and content['LastModified'] <= datetime.now().astimezone() - timedelta(days=15)
                    ):
                    yield content