Product Feedback for Azure Network Watcher

 

Feedback Summary:

Azure Network Watcher is an essential network monitoring and diagnostics tool within the Azure ecosystem. It provides various capabilities to help users monitor, troubleshoot, and diagnose network-related issues in Azure environments. While it offers powerful features for network visibility, there are areas where enhancements could further improve its usability, depth of insight, and integration with other Azure services.

---

Positive Aspects:

1. Comprehensive Network Diagnostics:

   • Azure Network Watcher offers a wide range of network monitoring features like IP flow verify, Network Security Group (NSG) flow logs, and connection troubleshoot tools. These features provide great insights into traffic patterns, security configurations, and network connectivity.
   • Connection Monitor and Topology View provide useful visualizations and diagnostic information that simplifies troubleshooting for users, especially for complex environments with multiple resources.

2. Real-Time Troubleshooting:

   • Connection Troubleshoot and VPN Troubleshoot are invaluable tools for diagnosing network connection issues. They provide real-time feedback and easy-to-understand diagnostic information that helps users quickly identify network problems.
   • The ability to troubleshoot and identify issues with resources like VMs, App Services, and Load Balancers directly within the Azure Portal streamlines problem resolution and enhances operational efficiency.

3. NSG Flow Logs and Traffic Analytics:

   • Integration with NSG flow logs allows users to capture and analyze network traffic flow across different resources. Traffic Analytics provides detailed insights into traffic patterns and helps in optimizing network performance, security, and compliance monitoring.
   • Having the ability to export NSG flow logs to Azure Storage and analyze them via Azure Monitor and Log Analytics makes it easier to audit and assess network traffic.

4. Network Topology:

   • The Network Topology feature offers a graphical representation of network resources and their interconnections. This feature is helpful in visualizing complex network architectures and aids in troubleshooting network issues by showing how resources are connected.

5. Integration with Azure Automation:

   • Integration with Azure Automation allows users to automate network-related diagnostics, reporting, and remediation actions, saving time and improving the consistency of network management.

---

Areas for Improvement:

1. Granularity in Monitoring and Alerts:

   • While Network Watcher provides valuable data, there could be more granularity when it comes to setting custom alerts based on specific network conditions (e.g., latency thresholds, specific types of traffic). More fine-grained alerting options, such as the ability to create customized alerts for specific network performance metrics, would improve the tool's utility.
   • Some users report that alerts for specific NSG flow logs or network anomalies can be broad and may require more fine-tuning to prevent unnecessary alerts from being triggered.

2. Additional Diagnostic Tools for Layer 7 Traffic:

   • Azure Network Watcher primarily focuses on lower-layer network diagnostics (Layer 3/4). More advanced diagnostics for Layer 7 (application layer) traffic would enhance the ability to monitor and troubleshoot web applications, APIs, and other application-level services. This could include insights into HTTP/HTTPS request/response analysis, detailed traffic tracing, and error codes.
   • Implementing more granular application-level performance insights would enable network and application teams to jointly troubleshoot issues that may involve both network and application layers.

3. Improved User Interface for Traffic Analytics:

   • While Traffic Analytics provides valuable insights, the user interface can sometimes feel cluttered or difficult to navigate. Simplifying the experience or introducing more customizable dashboards would allow users to focus on key metrics and improve the overall experience.
   • Adding more interactive features for data exploration, such as drilldowns on specific traffic patterns or timelines, would allow users to better understand their network’s performance and security.

4. Integration with Other Security Services:

   • While Network Watcher integrates with NSG flow logs, expanding its integration with other security services like Azure Firewall, DDoS Protection, or Azure Sentinel would provide a more complete view of network security and traffic analytics. This would streamline security monitoring and provide a unified dashboard for threat detection, network performance, and diagnostics.
   • Real-time security event tracking (such as blocked traffic or detected threats) integrated into Network Watcher’s diagnostic tools would also enhance its role in a comprehensive security operations strategy.

5. Troubleshooting for Multi-Region and Hybrid Networks:

   • Hybrid environments (on-premises and Azure) and multi-region configurations can complicate troubleshooting, especially when diagnosing issues involving ExpressRoute or VPN gateways. More comprehensive support and better diagnostics for hybrid and multi-region scenarios, with clearer visibility across on-premises and cloud resources, would make it easier to troubleshoot and optimize network performance.
   • Cloud-to-cloud troubleshooting (e.g., between regions or between Azure and other cloud providers) is a growing need, and enhancing the tooling for these scenarios would help businesses with geographically dispersed infrastructure.

6. Network Performance Metrics:

   • Currently, Network Watcher focuses more on connectivity and security issues. Including detailed network performance metrics (e.g., latency, bandwidth, jitter, packet loss) directly within the service would help users monitor and troubleshoot performance issues in addition to connectivity.
   • Adding more detailed insights into network latency and congestion, especially across larger network topologies, would help teams identify and mitigate bottlenecks in high-traffic environments.

---

Feature Requests:

1. Layer 7 Traffic Monitoring:

   • Introduce capabilities for deeper inspection of Layer 7 traffic, including application-level metrics and error handling for HTTP, HTTPS, DNS, and other protocols. This would help diagnose application performance issues in real-time.

2. Customizable Alerts and Metrics:

   • Provide the ability to set custom alerts for network-related metrics, such as specific latency thresholds, packet loss rates, or traffic patterns. A more granular, flexible alerting system would help users manage their networks more proactively.

3. Advanced Hybrid and Multi-Region Support:

   • Expand Network Watcher’s troubleshooting capabilities for hybrid network configurations (Azure and on-premises) and multi-region deployments. Offering better diagnostics for VPN connections, ExpressRoute, and cross-region traffic would be beneficial for complex enterprise architectures.

4. Security Integration with Azure Sentinel:

   • Direct integration with Azure Sentinel for real-time security monitoring and correlation of network traffic with potential threats would enhance security visibility. Providing immediate alerts for network anomalies and malicious activity would create a more unified security management experience.

5. Better Visualization for Traffic Analytics:

   • Enhance the visualization tools in Traffic Analytics with more customizable dashboards, drill-down capabilities, and interactive graphs that make it easier to spot network performance issues and trends.

6. Network Performance Monitoring:

• Expand Network Watcher’s capabilities to include detailed performance monitoring for traffic flows, latency, packet loss, jitter, and throughput. This would be especially useful in high-performance or latency-sensitive applications like VoIP or real-time video streaming.

Thant Zin Phyo@Cracky (MCT, MCE, MVP)

Security Alerting with AWS Security Hub

    With so many security tools available in both AWS and from third-party providers, those that are responsible for managing the alerts need a single pane of glass to centralize all the alerts and notifications coming in. AWS Security Hub helps you consolidate many of your security findings, alerts, and compliance reports from AWS services, including the following:

• AWS Identity and Access Management (IAM)

• Amazon Macie

• Amazon GuardDuty

• Amazon Inspector

• AWS Firewall Manager

In addition to these native AWS services, AWS Security Hub can be incorporated into any third-party partner solutions, such as Sumo Logic, Splunk, and other vendors you might use in your organization. A complete list of these partners can be found at https://aws.amazon.com/securityhub/partners/.

The Security Hub service allows you to categorize and prioritize all the events coming in from various sources. This single-pane-of-glass view gives you a more comprehensive picture and a deeper understanding of any threats and vulnerabilities forewarning your account. Security Hub, by default, is a regional service, but as with Amazon GuardDuty, it can be configured as an administration/member configuration from within a security account. Along with AWS Organizations, the organization management account specifies the Security Hub administrator account using the following configuration:

Figure : Security Hub administrator/member configuration

    Suppose you have already configured your accounts, as shown in Figure, for the GuardDuty service to consume all the account data in a segmented security account. In that case, the Security Hub service can utilize this same setup without additional configuration.

One of the most compelling use cases for Security Hub is automated compliance and configuration checks. Keeping every account compliant becomes burdensome and time-consuming as your organization grows and expands to multiple accounts. Trying to understand whether a developer has configured something in one of the child accounts that goes against the compliance standards of the whole organization can be automatically displayed and reported on, via the Security Hub service, so that you can take appropriate action.

AWS Security Hub, out of the box, performs 43 fully automated, nearly continuous checks against your accounts based on the CIS Foundations benchmark. It then takes the findings from the checks and displays them on the main dashboard of Security Hub, allowing your security and audit team to have quick access to these findings. Security Hub mainly looks at configuration choices and usage patterns at the account level. This is in contrast to AWS Config, which reviews items at instance and resource levels. By focusing on CIS checks, AWS Security Hub helps ensure that your AWS infrastructure adheres to the recommended security best practices set forth by CIS.

Having AWS Config enabled is a prerequisite to ensure that compliance standards are met while using Security Hub. This is because Security Hub uses the information from Config to determine when there is a change in an account. This allows the Security Hub service to be refreshed in near real time. Having covered how Security Hub works with multiple accounts and how it differs in its findings from the preceding AWS Config service, the following section will help you understand how Security Hub groups its findings to make things easier for security professionals.

Thant Zin Phyo@Cracky (MCT, MCE, MVP)

Enabling Amazon GuardDuty

    Amazon GuardDuty is a regional service. You must first select the region where you will enable the service; once that is done, it is effortless to enable it. Just complete the following steps:

1. From the Amazon Management Console, find the GuardDuty service by going to Services > Security, Identity, & Compliance > GuardDuty, or search for GuardDuty in the top search bar.

2. Once on the GuardDuty page, click the orange Get Started button to enable the GuardDuty service.

3. This will bring you to the Enable GuardDuty page. GuardDuty needs to create a service role to monitor and protect your account. Since data is involved, click on the orange Enable GuardDuty button at the bottom of the page to allow GuardDuty to be enabled.

After enabling GuardDuty, you will be brought to the main GuardDuty page (that is, the Findings page) by default. Since you have just enabled the service, three zeros in the page’s top right-hand corner should indicate no high-, medium-, or low-severity alerts.

Figure : The severity alert count for GuardDuty

Customizing GuardDuty

With GuardDuty enabled, you can customize the service to meet your organization’s needs. Combining GuardDuty with the CloudWatch Events service allows you to match specific severity levels with either automated remediations and/or alerts sent out to a particular SNS topic. GuardDuty rates the items that it finds on a numerical scale, broken down into the following three categories:

• High-severity items are valued between 7.0–8.9

• Medium-severity items are valued between 4.0–6.9

• Low-severity items are valued between 1.0–3.9

If the severity is not at a high level (say, for instance, if it is a 4 or 5), then you may only want to have the security team notified via the messaging channel (e.g., MS Teams or Slack) that they are constantly monitoring. Conversely, if the severity level is high (such as 7 or 8), you may want the security team and an executive stakeholder notified so that a response plan can be prepared, just in case it is needed.

Triggering GuardDuty

You can try modifying something in your account to see how a finding would look in the GuardDuty service.

AWS has created a lab to simulate malware in a contained environment and generate findings in the

GuardDuty service. You can find the files for this exercise at https://github.com/awslabs/amazon-guardduty-tester:

1. Download the repo by going to your computer and running the following command:

git clone https://github.com/awslabs/amazon-guarddutytester.git

2. Now that you have the CloudFormation templates and testing scripts available locally, go to your AWS Management Console and run the template from the CloudFormation service. You can get there quickly by going to the following URL: https://console.aws.amazon. com/cloudformation/.

3. Once in the CloudFormation service, under Stacks (if you have not already been taken there by default), click the Create Stack button so that additional menu items appear. Choose the option labeled With new resources (standard) to create a new CloudFormation stack.

Figure : The screen to create a CloudFormation stack

4. On the Create Stack page, move to the section labeled Specify template. Choose the option labeled Upload a template file. Click the Upload File button and find the folder on your local drive named amazon-guardduty-tester. Inside that folder will be a file named guardduty-tester.template.

Figure : Upload a template file screen for GuardDuty

5. Once you have selected the guardduty-tester.template file, press the orange Next button at the bottom of the page.

6. You should now be on the Specify stack details page. Enter guardduty-tester as the name of the stack.

Figure : The stack name screen for GuardDuty

7. In the Parameters section, under Availability Zones, as the prompt says, just select the first AZ to keep things simple.

Figure : The Availability Zones selection screen for GuardDuty

8. The next item you should customize is Allowed Bastion External Access CIDR. Instead of keeping this open globally, enter your IP address followed by /32. This means that only your singular IP address will be allowed to access the bastion host.

Figure : The Bastion CIDR details screen for GuardDuty

9. The final item that needs customization in the template is the key pair name. Use the dropdown to select one of your current EC2 key pairs.

10. After customizing the template, move to the bottom of the screen and press the orange Next button.

11. This brings you to the Configure stack options screen; scroll down to the bottom of the page and click the orange Next button.

12. You should now be on the Review guardduty-test page. Once again, scroll down to the bottom of the page. Acknowledge that this template may create IAM resources by clicking in the blue box. Once done, press the orange Submit button to bring up the template’s resources. After around 10 minutes, the template should have completed creating all the resources. You can now click on the Outputs tab and obtain the IP address of the Bastion host.

Figure : The GuardDuty Outputs screen

13. Next, open up a terminal window so that you can ssh into the Bastion IP address shown in the previous step. Use the IP provided with the username ec2-user to log in to the instance. You will also need the ssh key you designated when creating the template:

ssh -i ~/.ssh/key-name ec2-user@3.141.240.161

14. You will be greeted with a terminal window once you have entered the Bastion host. With the terminal window open, go to the EC2 service in the AWS Management Console and look for an instance named RedTeam. Click on this instance and copy (or write) its private IP address (you will need it in a moment).

15. You must place your SSH key on the Bastion host inside the ssh folder. On your local machine, print the key out, and then copy and paste it into the new file.

16. Returning to your terminal, create a file on your Bastion host at the path of ~/.ssh/{yourssh-key}. You can do this via the command line with the vim command:

vim ~/.ssh/{your-ssh-key.pem}

17. Once the Vim editor is opened, paste the value from your private key file.

18. Close and save the file by pressing the esc button at the top of the keyboard, and then type :wq.

This is the write/quit sequence in the Vim editor. To make sure that the file has been saved correctly, run the following command:

cat ~/.ssh/{your-ssh-key}

19. After you have created the key file, change the permissions of the file using the following command:

chmod 0600 ~/.ssh/{your-ssh-key.pem}

20. With the key in the ~/.ssh folder located on the Bastion host, you can now log in to a tester instance simply with the command:

ssh -i ~/.ssh/{your-ssh-key.pem} ec2-user@{RedTeam EC2 IP}

21. At this point in the exercise, you should be on the Red Team EC2 instance. Here, you can run the guardduty_tester.sh script. Run the following command to generate the findings:

./guardduty_tester.sh

22. Once you have generated the findings, you should be able to clean up all instances and resources quickly and easily by terminating the CloudFormation template.

After learning how to enable Amazon GuardDuty and trigger it using a few scripts, the next step is to review the findings and see how the GuardDuty service detects unusual activity.

Reviewing the Findings in GuardDuty

If you went through the exercise in the previous section, you will see the findings appear inside the GuardDuty console after about 8 to 10 minutes. At the top of the screen, the colored ovals that previously contained all zeros will now have one in the medium-severity category and two in the high-severity category.

Figure : Findings from the GuardDuty Test exercise

From the dashboard setting in GuardDuty, you can sort the findings by different values, including severity, finding type, resource (in this case, it would be an instance ID to allow grouping of all the EC2 instances), the time when the incident last occurred, and finally, the numerical count of the type

of attack.

If you click on the instance, it opens up in the EC2 service page. It also shows more information on the screen, with details of the finding and some remediation recommendations.

Figure : More details revealed about the GuardDuty findings

Thant Zin Phyo@Cracky (MCT, MCE, MVP)