Real Time Log Monitoring Using AWS CloudWatch

Spread the love


Hi,

In this post of devOpsJourney , you will see how to setup “Real Time Log Monitoring Using AWS CloudWatch”.

You can use CloudWatch Logs to monitor applications and systems using log data. For example, CloudWatch Logs can track the number of errors that occur in your application logs and send you a notification whenever the rate of errors exceeds a threshold you specify. CloudWatch Logs uses your log data for monitoring; so, no code changes are required. For example, you can monitor application logs for specific literal terms (such as “NullReferenceException”) or count the number of occurrences of a literal term at a particular position in log data (such as “404” status codes in an Apache access log). When the term you are searching for is found, CloudWatch Logs reports the data to an CloudWatch metric that you specify.

Source : http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchLogs.html

In this post, we will demonstrate how we can setup Real Time Log Stream in Cloud watch for Apache Server (httpd) which is running on AWS EC2.

Step 1: Under AWS CloudWatch Console, select log on left hand side and create a Log Group.

1_logGroup

 

Step 2 : Select your log group and create a Metric Filter.

2_metricfilter

 

Step 3 : Enter Filter Name and Filter Pattern. For Example if you want to filter 40x Error of Apache Web Server with host, time stamp and request code use the below pattern.

3_metric

Step 4: (Optional) Either you can create a stream here , give the same Stream name in coming next step where we’ll configure the agent on EC2 or you can skip this step, Stream will be configured automatically.

4_stream

 

Step 5: Create a alarm for filter . Select your Log Group and Click on Create Alarm.

6_add_alarm

 

 

 

 

 

 

 

 

Step 6: Lets configure if any of 4xx event occurs in Log File.

7_showAlarm

 

Step 7: Now we need to configure agent in AWS EC2 . We’ll configure agent for Apache logs on this Instance.

In this demonstration, we are using Amazon Linux AMI. Login to your AWS EC2 instance and install/configure following.

  • Install AWS Log Agent
  • sudo yum install -y awslogs

  • Open /etc/awslogs/awscli.conf and provide below details.
  • region = <us-east-1, us-west-1, us-west-2, eu-west-1, eu-central-1, ap-southeast-1, ap-southeast-2, or ap-northeast-1>
    aws_access_key_id = <YOUR ACCESS KEY>
    aws_secret_access_key = <YOUR SECRET KEY>

Note : If IAM role having CloudWatch Permission have been assigned to EC2, then there is no need to provide aws_access_key_id and aws_secret_access_key.

  • Install the httpd server if not already installed and place some content in /var/www/html/ directory.
  • Add below entries in the end of /etc/awslogs/awscli.conf file.
  • [/etc/httpd/logs/access_log]
    datetime_format = %b %d %H:%M:%S
    file = /etc/httpd/logs/access_log
    buffer_duration = 5000
    log_stream_name = testHttpdStream
    initial_position = start_of_file
    log_group_name = TestLogGroup

  • Restart the service of awslogs.
  • sudo service awslogs start

Step 8 : We can see the logs coming under Log Groups >> Stream for <streamName>

8_logOutput

 

Step 9: To check the event (e.g 404 Event), select your stream and check event logs.9_outputEvents

 

Step 10: Check the Alarms Section in CloudWatch. As we have setup Alarm for the above filter, We’ll get one alarm for same.

So this is how we can setup Logs on AWS CloudWatch and see Real Time Monitoring .

This was all about our post. Let us know if you have any questions, suggestions, feedback. We ‘ll surely look into this.




Thanks.

— Ishant
Share with your network.

Leave a Reply

Your email address will not be published. Required fields are marked *