Copy Latest Files From AWS S3 Using Powershell Script
In this blog, we will create a Powershell script, which will copy latest files from AWS S3 to paste it in local. So, lets start the process.
Note: For this script, we need to install AWS CLI on local Windows machine and we need configure IAM user credentials with S3 get and put object permission.
Step 1: Create IAM user and download the access key and secret key credentials for AWS CLI configuration.
Step 2: Create a policy with S3 put and get permission, which will help script to fetch files details and copy paste it locally.
1{
2 "Version":"2012-10-17",
3 "Statement":[
4 {
5 "Effect":"Allow",
6 "Action": "s3:ListAllMyBuckets",
7 "Resource":"arn:aws:s3:::*"
8 },
9 {
10 "Effect":"Allow",
11 "Action":["s3:ListBucket","s3:GetBucketLocation"],
12 "Resource":"arn:aws:s3:::awsBucketName"
13 },
14 {
15 "Effect":"Allow",
16 "Action":[
17 "s3:PutObject",
18 "s3:PutObjectAcl",
19 "s3:GetObject",
20 "s3:GetObjectAcl",
21 "s3:DeleteObject"
22 ],
23 "Resource":"arn:aws:s3:::awsBucketName/*"
24 }
25 ]
26}
Step 3: Attach policy with IAM user.
Step 4: Install AWS CLI on windows server and configure credentials using below command:
1aws configure --profile my-s3
Step 5: Now, Lets start creating the script. Open a new file and paste the below code.
1$s3Bucket = "s3://s3-bucket" # Enter S3 bucket from where you want to copy the files.
2$s3Prefix = "/local-path/" # Enter the path where you want to paste the files.
3$s3location = "${s3Bucket}${s3Prefix}"
4$files = $(aws s3 ls $s3location --recursive --profile my-s3 | sort | select -last 1)
5$dlPath = "C:\TEMP-data"
6foreach ($s3FileInfo in $files) {
7 $filename = $s3FileInfo.Split()[-1]
8 $path = "${s3Bucket}/${filename}"
9 aws s3 cp $path $dlPath --profile my-s3
10 echo("Done downloading ${path} to ${dlPath}")
11}
Step 6: Save the file with .ps1 extension.
You can also configure the script in windows scheduler to automate this. Use this link to know more.