CloudTrail is a service provided by AWS that monitors all activity in your account, including API actions made by IAM Users. It’s useful for performing security audits, but the default search console for it isn’t the greatest.
Linking CloudTrail and CloudWatch
By default, CloudTrail logs all events for the last 90 days in your account. However, to link it with CloudWatch, you’ll need to create a Trail, which keeps records of events for longer, and also has the option to keep extended logs on individual S3 writes and Lambda invocations.
To create one, head over to the CloudTrail Management Console, and under the “Trails” tab, create a new one. You can choose which regions and what kinds of events it monitors.
Under “Data Events,” you can also turn on extended monitoring for S3 buckets or Lambda functions. These are optional and will incur some additional charges, as well as taking up a lot more storage space in CloudWatch Logs.
Once the trail is created, you can turn on the CloudWatch Logs integration by clicking on the trail name under “Trails,” scrolling down to “CloudWatch Logs,” and pressing “Configure.”
The only option here is the log group name, which defaults to CloudTrail/DefaultLogGroup. The group will be created if it doesn’t already exist.
Because of how AWS’s permissions system works, you need to grant CloudTrail sufficient privileges to access CloudWatch Log Groups and create streams to start sending log events. This role is already configured, and all you need to do is press “Allow” on the next screen to link the two services together.
You should now see the log group and IAM role under the trail settings:
And, in CloudWatch, you will see a new log group and log stream created, which will begin streaming all events automatically.
CloudWatch will receive all updates going forward, but currently there isn’t a built in way to import previous events.
Linking CloudWatch and Elasticsearch
Elasticsearch is a search engine that is commonly used to analyze Linux log files, and is often paired with Kibana, a visualization engine that is able to draw graphs and plots using the data provided by Elasticsearch. With the huge amount of data an active AWS account can spit out from CloudTrail, Elasticsearch makes sense for a lot of people. Luckily, it’s fairly easy to set up.
You can read our full guide to setting up an Elasticsearch server on AWS here. However, if you’ve already got it set up, linking CloudWatch to it is pretty simple. From the CloudWatch Console, select the log group you wish to link, and select “Stream To Amazon Elasticsearch Service”:
This will bring up a dialog where you can select your ES cluster. After that, you should see all events from Elasticsearch.