Copy Latest Files From AWS S3 Using Powershell Script

In this blog, we will create a Powershell script, which will copy latest files from AWS S3 to paste it in local. So, lets start the process.

Note: For this script, we need to install AWS CLI on local Windows machine and we need configure IAM user credentials with S3 get and put object permission.

Step 1: Create IAM user and download the access key and secret key credentials for AWS CLI configuration.

Step 2: Create a policy with S3 put and get permission, which will help script to fetch files details and copy paste it locally.

 2   "Version":"2012-10-17",
 3   "Statement":[
 4      {
 5         "Effect":"Allow",
 6         "Action": "s3:ListAllMyBuckets",
 7         "Resource":"arn:aws:s3:::*"
 8      },
 9      {
10         "Effect":"Allow",
11         "Action":["s3:ListBucket","s3:GetBucketLocation"],
12         "Resource":"arn:aws:s3:::awsBucketName"
13      },
14      {
15         "Effect":"Allow",
16         "Action":[
17            "s3:PutObject",
18            "s3:PutObjectAcl",
19            "s3:GetObject",
20            "s3:GetObjectAcl",
21            "s3:DeleteObject"
22         ],
23         "Resource":"arn:aws:s3:::awsBucketName/*"
24      }
25   ]

Step 3: Attach policy with IAM user.

Step 4: Install AWS CLI on windows server and configure credentials using below command:

1aws configure --profile my-s3

Step 5: Now, Lets start creating the script. Open a new file and paste the below code.

 1$s3Bucket = "s3://s3-bucket"  # Enter S3 bucket from where you want to copy the files.
 2$s3Prefix = "/local-path/"    # Enter the path where you want to paste the files.
 3$s3location = "${s3Bucket}${s3Prefix}"
 4$files = $(aws s3 ls $s3location --recursive  --profile my-s3 | sort | select -last 1)
 5$dlPath = "C:\TEMP-data"
 6foreach ($s3FileInfo in $files) {
 7    $filename = $s3FileInfo.Split()[-1]
 8    $path = "${s3Bucket}/${filename}"
 9    aws s3 cp $path $dlPath --profile my-s3
10    echo("Done downloading ${path} to ${dlPath}")

Step 6: Save the file with .ps1 extension.

You can also configure the script in windows scheduler to automate this. Use this link to know more.

I :heart: AWS! :smile: Enjoy