Find Source IP Addresses Connecting to AWS Transfer Family SFTP with CloudWatch Logs Insights

If your AWS Transfer Family SFTP server has CloudWatch logging turned on, every connection, login attempt, and file operation is already sitting in a log group — you just need the right query to pull the source IPs out of it. This comes up in three situations: someone reports failed logins and you need to know where they're coming from, you're building an IP allowlist and need a baseline of who's actually connecting today, or an audit asks "who accessed this SFTP server and from where" and the honest answer is "let me go check."

Prerequisites

• CloudWatch logging enabled for your Transfer Family server (Server details → Additional details → Logging role, pointing at a log group).
• IAM permission to run logs:StartQuery / logs:GetQueryResults against that log group.
• A rough idea of the time window you care about — CloudWatch Logs Insights charges by data scanned, and an unbounded "all time" query on a busy server gets expensive fast.

Why the log format matters before you write a single query

Transfer Family logs come in two shapes, and which one you have changes how you should query it. Newer configuration uses JSON structured logs, where a connection event looks like this (real field names, from AWS's own example logs):

 1{
 2   "role": "arn:aws:iam::500655546075:role/transfer-s3",
 3   "activity-type": "CONNECTED",
 4   "client": "SSH-2.0-OpenSSH_7.4",
 5   "source-ip": "52.94.133.133",
 6   "resource-arn": "arn:aws:transfer:us-east-1:500655546075:server/s-3fe215d89f074ed2a",
 7   "home-dir": "/test/log-me",
 8   "user": "log-me",
 9   "session-id": "9ca9a0e1cec6ad9d"
10}

Older servers still write legacy line-based logs, where the same event looks like this instead:

1user.914984e553bcddb6 CONNECTED SourceIP=1.22.111.222 User=lhr HomeDir=LOGICAL Client=SSH-2.0-OpenSSH_7.4 Role=arn:aws::iam::123456789012:role/sftp-s3-access

Same information, completely different shape — one is a JSON object with a source-ip key, the other is space-delimited text with a SourceIP= token. Don't guess which one you have; the first query below tells you in ten seconds, and it decides whether the rest of this guide is a two-line query or a regex.

Step 1: Open CloudWatch Logs Insights and check the raw format

Console → CloudWatch → Logs → Logs Insights → select your Transfer Family log group (named /aws/transfer/<server-id>) → pick a time range, then run:

1fields @timestamp, @message
2| sort @timestamp desc
3| limit 20

Look at the raw @message values. If they start with { and look like the JSON block above, you're on structured logging and should use Step 2. If they look like the space-delimited legacy format, skip to Step 5.

Step 2: Query source IPs directly (structured JSON logs)

This is the part worth knowing: CloudWatch Logs Insights automatically discovers fields from JSON log events — you don't need to parse anything out of @message at all, you can just reference the field. The only wrinkle is that source-ip contains a hyphen, and any discovered field name with a non-alphanumeric character (other than @ or .) has to be wrapped in backticks:

1fields @timestamp, `source-ip`, user, `activity-type`
2| filter `activity-type` = "CONNECTED"
3| sort @timestamp desc
4| limit 100

This is more reliable than regex-scraping @message, because you're reading the field CloudWatch already parsed for you instead of re-deriving it from free text.

Step 3: Aggregate — count, first seen, last seen, per IP

For an actual investigation you want counts and a time window, not a raw event list:

1fields `source-ip`, `activity-type`
2| filter `activity-type` = "CONNECTED"
3| stats count(*) as connections, min(@timestamp) as first_seen, max(@timestamp) as last_seen by `source-ip`
4| sort connections desc

This is the single query to keep — it answers "who's connecting, how often, and over what window" in one shot, and it's the one to reach for by default rather than the raw event dump from Step 1.

Step 4: Correlate IP with username

Because user and source-ip live on the same CONNECTED log event, you get a per-user/per-IP breakdown for free — no separate correlation step needed:

1fields `source-ip`, user
2| filter `activity-type` = "CONNECTED"
3| stats count(*) as connections by user, `source-ip`
4| sort connections desc

Watch for the same username showing up against multiple, unrelated-looking source IPs in a short window — that's the pattern that usually means shared or leaked credentials, not just someone switching networks.

Step 5: Extract the IP with regex (legacy log format, or as a portable fallback)

If Step 1 showed you the legacy SourceIP=x.x.x.x format, there's no discovered field to reference — you extract it from @message with parse, using a regex and a named capture group:

1fields @timestamp, @message
2| parse @message /SourceIP=(?<source_ip>\d{1,3}(?:\.\d{1,3}){3})/
3| stats count(*) as connections, min(@timestamp) as first_seen, max(@timestamp) as last_seen by source_ip
4| sort connections desc

CloudWatch Logs Insights' parse command runs on RE2 regex syntax, which supports the non-capturing group ((?:...)) and {1,3} quantifier used above without issue — this isn't the older, more restricted regex dialect used by CloudWatch subscription/metric filter patterns, which is a separate, more limited feature. A bare IP-matching regex with no anchor (\d{1,3}(?:\.\d{1,3}){3}) also still works against the JSON format from Step 2 in a pinch — the digits are in there too, just inside a JSON string — but anchoring to the field it actually came from (SourceIP= or "source-ip":) avoids accidentally matching a stray dotted-number sequence elsewhere in the line, and matching the discovered field directly (Step 2/3) avoids the question entirely.

Step 6: Investigate one specific suspicious IP

Once an IP stands out, pull everything it touched, across activity types, not just connections:

1fields @timestamp, `activity-type`, user, path
2| filter `source-ip` = "203.0.113.25"
3| sort @timestamp desc
4| limit 1000

Include AUTH_FAILURE events specifically if you're chasing a brute-force pattern rather than a single successful session:

1fields @timestamp, user, `activity-type`
2| filter `source-ip` = "203.0.113.25" and `activity-type` = "AUTH_FAILURE"
3| sort @timestamp desc

Step 7: Check IPs against a known allowlist

Rather than eyeballing a list of IPs against a spreadsheet, CloudWatch Logs Insights has built-in IP functions you can use directly in a query — isIpInSubnet(), isPrivateIP(), isPublicIP(), and a few others. This flags any connection from outside your expected corporate CIDR range in the same query that gathers the data:

1fields @timestamp, `source-ip`, user
2| filter `activity-type` = "CONNECTED"
3| filter not isIpInSubnet(`source-ip`, "203.0.113.0/24")
4| stats count(*) as connections by `source-ip`, user
5| sort connections desc

Swap 203.0.113.0/24 for your real allowlisted range (or chain several with or). This is a more direct way to answer "did anyone connect from outside where we expect" than exporting the IP list and diffing it by hand.

Best practices

• Default to Step 3's query, not a raw @message dump — count plus first/last seen is the shape almost every investigation actually needs.
• Narrow the time range before running anything. CloudWatch Logs Insights bills by data scanned; a targeted 24-hour window on a specific log group costs a fraction of "last 30 days, no filter."
• Prefer the discovered JSON field over regex whenever your logs are structured. It's less to get wrong, and it survives a log format tweak that would break a hand-written regex.
• Remember one public IP can be many people. NAT gateways, corporate proxies, and VPNs put multiple real users behind a single source IP — treat a spike from one IP as a lead to correlate with user and session-id, not a conclusion on its own.
• Cross-reference with CloudTrail for the control-plane side (who changed the server's IAM role, security policy, or user configuration) — CloudWatch Logs Insights on the Transfer Family log group only shows you the data-plane SFTP activity itself.

Common mistakes to avoid

• Forgetting the backticks on source-ip and activity-type. Without them, CloudWatch Logs Insights reads source-ip as the subtraction of two identifiers (source minus ip) and the query either errors or silently returns nothing useful.
• Assuming every server logs in the same format. Servers created or reconfigured at different times can be on structured JSON or legacy line logs — check with Step 1 per server, don't assume it matches the last one you queried.
• Running limit 10000 as a habit rather than a ceiling. It's the documented maximum for a single query, but pulling 10,000 rows into the console when stats ... by source-ip would collapse them to a few dozen just makes the result harder to read, not more complete.
• Treating source-IP volume alone as a security verdict. A high connection count from one IP is often a scheduled automated job, not an attacker — check user and the operation types (OPEN/CLOSE/DELETE) before escalating.

Troubleshooting

Query returns zero results even though you can see log events in the console. Check your time range first — Logs Insights uses the range picker independently of what the raw log stream view shows. Then confirm the field name casing and backtick-quoting exactly matches what Step 1 showed you.

parse runs but source_ip is empty for some rows. That's expected, not a bug — parse leaves non-matching events in the result set with the extracted field blank rather than dropping them. Add | filter ispresent(source_ip) if you want only the rows that actually matched.

Regex approach and JSON field approach give different counts. Almost always a filter mismatch, not a query bug — check whether one query is implicitly including non-CONNECTED activity types (like KEX_FAILURE, which also carries a source-ip field) that the other excludes.

FAQ

Does this work the same way for FTPS and FTP servers, not just SFTP? Yes — the JSON structured log schema and activity-type values are shared across SFTP, FTPS, and FTP protocols on Transfer Family; only some fields (like SSH-specific ones) won't populate for non-SFTP protocols.

Can I turn a legacy-format server into structured JSON logs without recreating it? Check your server's logging configuration under Additional details in the console or via update-server in the CLI — Transfer Family has been migrating servers toward structured logging, but confirm your specific server's current setting with Step 1 rather than assuming.

What's the actual maximum number of rows a query can return? 10,000, set with limit 10000 — that's a hard ceiling for a single Logs Insights query, not a configurable quota.

Can I export this to build a permanent, always-on allowlist enforcement instead of running an ad hoc query? CloudWatch Logs Insights is fine for investigation and reporting, but ongoing enforcement belongs in the connect workflow itself — a custom identity provider Lambda, or a resource/session policy — since Logs Insights only tells you after the fact what already connected.

Do these queries cost anything beyond standard CloudWatch Logs pricing? Logs Insights charges per GB of log data scanned by the query, on top of whatever you already pay for log ingestion and storage — narrowing the time range and log group is the main lever for keeping that cost down.

Key takeaways

Further Reading

• Enhancing AWS Security: A Comprehensive Guide to IAM Policies for Restricting Access by Source IP
• AWS Security Checklist: Protecting Your Cloud Environment
• CloudWatch Managed Prometheus Collectors: Setup Guide for EC2 and ECS
• Official: Example CloudWatch log entries — AWS Transfer Family
• Official: CloudWatch Logs Insights query syntax — parse command