Backgroound Image

Network Field Day 28 – Recap and Review

Posted on by Dan Kelcher

  I had the opportunity to participate as a delegate at Network Field Day 28, and I wanted to share my experience.

What is Network Field Day?

Network Field Day is one of the Tech Field Day events put on by Gestalt IT where sponsoring vendors present to a panel of delegates.  Network Field Day is specifically focused on networking solutions, and there are other events including Security Field Day and Storage Field Day with content that aligns to the respective categories.
There are usually about twelve delegates per event, with each one being invite-only.  Each delegate is independent (not employed by a vendor, or an industry analyst), is active in the community through things like blogs, podcasts, social media, etc, and could be considered a subject matter expert on the event topic.
There’s a lot of information on how TFD works at their “About” page – https://techfieldday.com/about/ I recommend checking out the infographic, and reading through the FAQ to get a better understanding of what the event is about.
Want to find out more about the presenters or delegates?  Want to watch the recorded sessions?  Go to the NFD28 page to get all that and more!

Vendor Presentations

The event spanned three days, with 9 presenters, with 13.5 hours of presentations and 4.5 hours of off-camera conversations.  Plus there were plenty of conversations with other delegates throughout the event.  That said, this isn’t an exhaustive review of everything.  I’ll be working on putting more detailed posts together soon.

Day 1

Juniper https://www.juniper.net/

Juniper had two 1.5 hour sessions, so there was a lot of information to cover.  There were a couple specific areas that they talked about extensively – Marvis and Apstra.
Marvis is Juniper’s AI that is used to help improve network operations.  One of the use cases would be streamlined anomaly detection, even to the point of potentially predicting issues before they occur.  There was a lot of discussion around Full Stack AIOps, along with a demo.  Another use case they presented was around wireless performance.  By collecting wireless performance data Marvis can recommend adding or moving APs to improve coverage.
Apstra is a solution that allows network teams to build out templates for data center deployments.  The cool thing that Apstra does is it disassociates the template from the underlying devices.  A single template could be deployed against Juniper, Cisco, and Arista hardware (among others) without needing to make any changes.  It takes the concept of intent-based networking and applies it in a mostly vendor agnostic way.  One of the use cases that was easy to see was environments that are being forced to look to different hardware vendors due to supply chain shortages. Continue reading “Network Field Day 28 – Recap and Review”

ThousandEyes Walkthrough Part 4.1 – SNMP Monitoring

Posted on by Dan Kelcher

This post will go over enabling and using SNMP monitoring in ThousandEyes. To see past posts in this series expand the box below.

ThousandEyes Walkthrough Table of Contents

There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:

With the lab environment built, and the agents installed and online, now it’s time to start actually getting monitoring data through ThousandEyes!

If you haven’t followed along with the previous posts in this series you can find the lab build here: https://www.mytechgnome.com/2022/04/thousandeyes-walkthrough-part-2-lab.html and the agent installation here: https://www.mytechgnome.com/2022/04/thousandeyes-walkthrough-part-3.html

This lab requires version 1.1 of the lab build.  Verify the lab you are using is 1.1 or newer.  If it’s not, look at the CHANGELOG section near the bottom of the lab build post: https://www.mytechgnome.com/2022/04/thousandeyes-walkthrough-part-2-lab.html

SNMP Configuration

The SNMP configuration will allow basic SNMP monitoring but is not intended to replace existing SNMP monitoring solutions.  Within ThousandEyes the value of SNMP monitoring is to provide more contextual data and visibility, and some capabilities to alert on different conditions.

  1. Open a web browser and navigate to https://www.thousandeyes.com/
  2. Log into your account
  3. Click the Hamburger icon in the top left
  4. Expand Devices
  5. Click on Device Settings
  6. There might be a Get Started with Devices splash screen, or it will take you directly to the Devices page.
    1. Splash screen –
      1. Click Start Discovery
    2. Devices Page
      1. Click Find New Devices
  7. On the right in the Basic Configuration enter the scan details
    2. In Targets enter the following subnet: 10.255.255.0/24
    3. In the Monitoring Agent drop-down select CS1-1
    4. Under Credentials click “Create new credentials”
      2. In the Add New, Credentials pane enter a name, and for the community, string enter: TE
    5. If the Credentials don’t auto-populate then click the dropdown and select the TE-SNMP that was just created
    6. Occasionally devices may not be picked up on the first discovery, but if the box is checked to “Save as a scheduled discovery” it will retry every hour
    7. Click Start Discovery
  8. Wait for the discovery process to complete – this might take a few minutes
    1. NOTE: There seems to be a bug in the UI where it displays a “No devices found” error, even though all the devices were discovered.
  9. Click back to the main section of the page and the Add Devices panel will disappear
  10. Click the Select All checkbox on the top left of the device list, then click Monitor at the bottom of the page.
  11. Wait a few minutes for the devices to show Green under the Last Contact column

SNMP Toplolgy

ThousandEyes includes a cool toplogy builder based on the data collected from the SNMP monitors.  It’s able to determine device adjacency, but not necissarily the best placement for our interpretation.  The good news is the devices can be moved to better align to what we’d like to see.
  1. Hover over the menu icon in the top left, then under Devices click on Views
  2. The Device Views will show some metric data on the top, and the topology on the bottom
  3. Click on Edit Topology Layout
  4. Devices in the topology view can be moved (drag-and-drop) to better represent the actual topology. Click Done Editing when the device positions match the lab topology.
As usual, if there were any issues you can add a comment to this post, or reach me on Twitter @Ipswitch. 

Conclusion

The SNMP monitoring in ThousandEyes is now configured.  One important note here is that this is a lab build.  In a production environment steps should be taken to secure SNMP access.  Restricting access to SNMP via ACL is always a good idea, as well as using SNMP v3 for authentication and encryption.
Later in this series the SNMP configuration will be revisited.  When data is flowing on the lab network the SNMP views will be useful in getting more information on traffic flows.  The SNMP data can also be used to help troubleshoot issues and to create alarms depending on network conditions.

What’s next

The next task is to define some scenarios to identify what needs to be monitored.  The scenarios will be generic but should be relatable for any IT professional out there.  After the scenarios are defined then the ThousandEyes tests can be built for each unique scenario.

ThousandEyes Walkthrough Part 3 – Enterprise and Endpoint Agent Installs

Posted on by Dan Kelcher

This post will go over installing the ThousandEyes agents in the lab. To see all the posts in this series expand the box below.

ThousandEyes Walkthrough Table of Contents

There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:

There are going to be a number of agent deployments in the lab that was covered in the previous post:

  • 4x Linux Enterprise Agent installs on the CML Ubuntu instances
    • CS1-1, CS1-2, CS2-1, and CS2-2
  • 2x Docker Enterprise Agent container deployments on the Ubuntu Docker host
    • These two agents will be added to a cluster
  • 1x Raspberry Pi Enterprise agent (optional)
  • 1x Windows Endpoint Agent install on the Windows VM

Prerequisites

The lab needs to be built out.  Details on that process can be found here: https://www.mytechgnome.com/2022/03/thousandeyes-walkthrough-part-2-lab.html
Before we can start with the agent installs some ThousandEyes licenses are required.  It’s possible you already have some ThousandEyes licenses.  Cisco has bundled Enterprise Agents with the purchase of DNA Advantage or Premier licensing on the Catalyst 9300 and 9400 switches.

If existing licenses are unavailable a 15-day trial license can be requested here: https://www.thousandeyes.com/signup/

Additional hardware and software

As a side note – if you plan to work a lot with the Raspberry Pi I strongly recommend getting the USB 3 adapter.  It has a significant improvement in performance over the USB 2 adapters that are typically bundled with Raspberry Pi kits.  The SD cards recommended by ThousandEyes are because of the card performance.  Other cards can be used, but there may be a negative impact on performance.

Installs

Account Group Token

Before getting started with the installs it is important to get your Account Group Token.  This is an ID that is used to associate the agents to the correct account.  When deploying agents it will often require the token to be specified.
There’s multiple ways to find the token, but I think the easiest is to just pull it from the Enterprise Agent deployment panel
  1. Open a web browser and navigate to https://www.thousandeyes.com/
  2. Log into your account
  3. Click the Hamburger icon in the top left
  4. Expand Cloud & Enterprise Agents
  5. Click Agent Settings
  6. Click the Add New Enterprise Agent button
  7. Click the eye button to show the token, or the copy button to store it on the clipboard
    2. In a production environment you would want to keep this token safe.  It provides devices access to your ThousandEyes account, so it should not be made public
  8. Store the token in a safe, convenient location.  It will be used to add agents to the ThousandEyes account throughout this process.

Linux Enterprise Agent install

  1. Open a web browser and navigate to https://www.thousandeyes.com/
  2. Log into your account
  3. Click the Hamburger icon in the top left
  4. Expand Cloud & Enterprise Agents
  5. Click Agent Settings
  6. Click the Add New Enterprise Agent button
  7. Click the option for Linux Package
  8. Copy the commands displayed 
    2. curl -Os https://downloads.thousandeyes.com/agent/install_thousandeyes.sh
chmod +x install_thousandeyes.sh
sudo ./install_thousandeyes.sh -b <--Your Token goes here-->
  9. Perform the following steps for CS1-1. CS1-2, CS2-1, and CS2-2 in CML
    1. In CLM open the terminal session and log in
    2. Paste the commands into the terminal and press Enter
    3. It may take some time, but eventually there will be a prompt that say:

      The default log path is /var/log. Do you want to change it [y/N]?

    4. Press Enter to accept the default log location
    5. It might take 10 minutes or it could be over an hour for the process to complete and the agent to come online.  When it returns to the user prompt the service should be started.
  10. When the installs are complete they should be listed in the ThousandEyes portal under Enterprise Agents
    2. If the agent status is yellow it likely means an agent update is required, and it should automatically update within a few minutes

Docker Enterprise Agent install

    1. Open a web browser and navigate to https://www.thousandeyes.com/
    2. Log into your account
    3. Click the Hamburger icon in the top left
    4. Expand Cloud & Enterprise Agents
    5. Click Agent Settings
    6. Click the Add New Enterprise Agent button
    7. Click the option for Docker
    8. Scroll down to the sections with the commands
    9. Copy the section to configure seccomp and apparmor profile 
      2. curl -Os https://downloads.thousandeyes.com/bbot/configure_docker.sh
chmod +x configure_docker.sh
sudo ./configure_docker.sh
    10. Log in to the Ubuntu node that is the Docker host and paste in the commands:
      1. Add listening IPs for the Docker containers 
        1. sudo ip add add 192.168.1.51 dev ens33
sudo ip add add 192.168.1.52 dev ens33
      2. Pull the TE Docker image 
        1. docker pull thousandeyes/enterprise-agent > /dev/null 2>&1
      3. Update these commands by putting in your ThousandEyes token and changing the IPs if needed, then run them to create two ThousandEyes agents.

NOTE: These commands have been updated to include DNS and IP settings that aren’t available on the ThousandEyes Enterprise Agent page. If you use the commands from ThousandEyes the DNS and Published ports will need to be updated.

      1. docker run
  --hostname='TE-Docker1' 
  --memory=2g 
  --memory-swap=2g 
  --detach=true 
  --tty=true 
  --shm-size=512M 
  -e TEAGENT_ACCOUNT_TOKEN=<--Your Token goes here--> 
  -e TEAGENT_INET=4 
  -v '/etc/thousandeyes/TE-Docker1/te-agent':/var/lib/te-agent 
  -v '/etc/thousandeyes/TE-Docker1/te-browserbot':/var/lib/te-browserbot 
  -v '/etc/thousandeyes/TE-Docker1/log/':/var/log/agent 
  --cap-add=NET_ADMIN 
  --cap-add=SYS_ADMIN 
  --name 'TE-Docker1' 
  --restart=unless-stopped 
  --security-opt apparmor=docker_sandbox 
  --security-opt seccomp=/var/docker/configs/te-seccomp.json 
  --dns=10.133.100.10 
  --dns-search=cml.lab 
  --publish=192.168.1.51:49152:49152/udp 
  --publish=192.168.1.51:49153:49153/udp 
  --publish=192.168.1.51:49153:49153/tcp 
  thousandeyes/enterprise-agent /sbin/my_init
      2. docker run
  --hostname='TE-Docker2' 
  --memory=2g 
  --memory-swap=2g 
  --detach=true 
  --tty=true 
  --shm-size=512M 
  -e TEAGENT_ACCOUNT_TOKEN=<--Your Token goes here--> 
  -e TEAGENT_INET=4 
  -v '/etc/thousandeyes/TE-Docker2/te-agent':/var/lib/te-agent 
  -v '/etc/thousandeyes/TE-Docker2/te-browserbot':/var/lib/te-browserbot 
  -v '/etc/thousandeyes/TE-Docker2/log/':/var/log/agent 
  --cap-add=NET_ADMIN 
  --cap-add=SYS_ADMIN 
  --name 'TE-Docker2' 
  --restart=unless-stopped 
  --security-opt apparmor=docker_sandbox 
  --security-opt seccomp=/var/docker/configs/te-seccomp.json 
  --dns=10.133.100.10 
  --dns-search=cml.lab 
  --publish=192.168.1.52:49152:49152/udp 
  --publish=192.168.1.52:49153:49153/udp 
  --publish=192.168.1.52:49153:49153/tcp 
  thousandeyes/enterprise-agent /sbin/my_init
  2. When the installs are complete they should be listed in the ThousandEyes portal under Enterprise Agents
    2. If the agent status is yellow it likely means an agent update is required, and it should automatically update within a few minutes

Docker Enterprise Agent configuration

There are two configuration tasks that will be performed on the Docker agents.  The IP setting in ThousandEyes will be updated to use the host IPs that are tied to the Docker agents instead of the private Docker IPs, and the two agents will be added to a ThousandEyes Cluster.
  1. Open a web browser and navigate to https://www.thousandeyes.com/
  2. Log into your account
  3. Click the Hamburger icon in the top left
  4. Expand Cloud & Enterprise Agents
  5. Click Agent Settings
  6. Click on the Agent
  7. In the right panel click on Advanced Settings
  8. Updated the IP address with the address assigned to that instance
  9. Click the Save Changes button on the bottom right
  10. Repeat this process for the other container agent
  11. At the Enterprise Agents page select both Docker agents
  12. Click the Edit button
  13. Select Edit Cluster
  14. On the right select Add to a new cluster
    2. In the name field type Docker
  15. Click Save Changes
    2. It will give a confirmation screen, click Save Changes again
  16. The agent icon will be updated to include the cluster icon, and under the Cluster tab it will display the new cluster
Wondering why those changes were made?
The first change to the IP address was because ThousandEyes learns the IP address of the agent from its local configuration.  Docker, by default, creates a bridged network that uses NAT to communicate with the rest of the network.  That means the addresses Docker assigns to containers aren’t accessible on the network.  The additional IPs were added to the Ubuntu host to allow static NAT entries to be created in Docker (the Publish lines), which redirect traffic to sent to those IPs to the correct agent.  Since there are two containers using the same ports, we need two IP addresses to uniquely address each instance.  The change that was made to the agent settings in ThousandEyes forces other agents to use the routed 192.168.1.0/24 LAN network instead of the unrouted 172.17.0.0/16 Docker network.  This is only needed because we are going to build inbound tests into those agents.  If this was only outbound then it wouldn’t matter.
As for the creation of the cluster, this was done for high availability.  Granted, in this scenario both instances are running on the same Docker host which defeats the purpose.  However, it still illustrates how to configure the cluster.  The purpose of the cluster is exactly what would be expected.  Both agents share a name, and are treated as a single agent.  If a test is assigned to a cluster then either instance could run it.  In addition to high availability, this also can provide some load balancing between the agents, and it can simplify test creation.  Instead of managing tests to multiple instances in one location we can use the cluster agent to distribute those tests.

Raspberry Pi Enterprise Agent install

I have an automated configuration process for the Raspberry Pi image: https://www.mytechgnome.com/2023/06/15/automated-thousandeyes-raspberry-pi-image-customization/

  1. Open a web browser and navigate to https://www.thousandeyes.com/
  2. Log into your account
  3. Click the Hamburger icon in the top left
  4. Expand Cloud & Enterprise Agents
  5. Click Agent Settings
  6. Click the Add New Enterprise Agent button
  7. The pane on the right should open the the Appliance tab, under Physical Appliance Installer find the Raspberry Pi 4, and to the right of that click Download – IMG
  8. Wait for the download to complete.  It’s nearly a 1GB file, so it might take a few minutes.
  9. Connect the SD card to the computer that will be doing the imaging
    1. This process erases the entire card.  Make sure you are using a blank card, or you have any valuable data on the card backed up elsewhere.
  10. Launch the Raspberry Pi Imager
  11. Under Operating System click Choose OS
  12. Scroll down to the bottom of the list and click Use custom
  13. Browse to the location of the downloaded image, select it, and click Open
  14. Under Storage click on Choose Storage (or Choose Stor…)
  15. Select the SD card in the window that pops up
    2. If the SD card does not show up try reseating the card
  16. Click Write
  17. Continuing this process will erase all data on the SD card, if that’s acceptable click Yes
  18. A progress bar will be displayed, and after a few minutes the image copy should complete successfully.  Click continue and close the Raspberry Pi Imager software
  19. Remove the SD card from the imaging PC and insert it in the Raspberry Pi.
  20. Boot the Raspberry Pi
    1. You’ll want a monitor connected to find the IP assigned, though this could also be done by looking at DHCP leases, scanning the network, or trying name resolution for the default hostname: tepi
    2. Make sure there’s a network cable plugged in and connected to the LAN (the ThousandEyes agent doesn’t support wireless connections)
  21. When the Pi finishes booting find the IP address displayed on the screen
  22. Use a web browser to connect to the IP of the Pi agent (using the name might work – https://tepi/)
  23. Likely the browser will display a security warning because the certificate is untrusted.  Go through the steps required to accept the security risk and access the site.
  24. At the login page enter the default credentials: admin / welcome
    2. After logging in there may be an error message that briefly appears in the lower right stating the Account Group Token needs to be set.  This will be resolved shortly, and the error can be ignored for now.
  25. The first page will prompt to change the password.  Enter the current password and create a new one, then click Change Password
    2. After the password change is saved click the Continue button at the bottom of the page
  26. The next page prompts for the Account Group Token.  Enter the token value that was collected earlier in this post and then click Continue
    2. Even though there is a button to enable Browserbot here, the Raspberry Pi agent does not support it.  Leave that field set to No.  You can decide if you want to leave the crash reports enabled.
  27. The agent will go through a check-in process and provide diagnostic data.  If everything looks good you can click Complete
  28. That completes the required agent set up.  It will then bring you to the network configuration page.  Scroll down to the DNS section, switch the Current DNS Resolver to Override and enter the IP 10.133.100.10 in the Primary DNS box
    2. For the purposes of this lab none of the other settings need to be changed.  A static IP can be configured and/or the hostname could be changed if desired
  29. The agent should now be listed in the ThousandEyes portal under Enterprise Agent
    2. If the agent status is yellow it likely means an agent update is required, and it should automatically update within a few minutes
That completes the Enterprise Agent installations for the lab.

Windows Endpoint Agent install

  1. Start the Windows VM and log in
  2. Open a web browser and navigate to https://www.thousandeyes.com/
  3. Log into your account
  4. Click the Hamburger icon in the top left
  5. Expand the Endpoint Agents section
  6. Click on Agent Settings
  7. Either a splash screen with a Download button will appear, or there will be a button to Add New Endpoint Agent.  Click the button that shows up – both bring up the same pane
    1. Splash screen – 
    2. Add Endpoint Agent Button
  8. Leave the Endpoint Agent radio button selected and click the button Download – Windows MSI
    2. The Mac installation isn’t being covered here, but there’s instructions on how to install it here: https://docs.thousandeyes.com/product-documentation/global-vantage-points/endpoint-agents/installing
  9. There will be two options for the processor architecture, select the x64 Windows MSI
  10. When the download completes run the MSI
  11. The installation is a typical MSI package, so I’m not going to include screenshots for every step
    1. Click Next to start the install
    2. Read the EULA and if you agree to the terms check the box to accept and click Next
    3. Click on the TCP Network Tests Support and select “Will be installed on local hard drive”
    4. Do the same for at least one browser extension.  Edge is the default browser on Windows 10, but if you want to install and use Chrome then get Chrome installed before continuing the Endpoint Agent installation.  Click Next when you have the browser selected.
    5. Click Install
    6. If there us a UAC prompt for the install, click yes to continue
    7. Click Finish
  12. It might take a few minutes for the agent to check in, but eventually you should see the agent listed under Endpoint Agents in the portal

Conclusion

This was the first post actually working with ThousandEyes, and hopefully it illustrates how powerful this tool is.  As part of the lab there are four different types of agents installed, but there’s many more available:
  • Bare metal install (Intel NUC or other hardware)
  • OVA (VMware ESX, Workstation, and Player, Microsoft Hyper-V Oracle VirtualBox)
  • Application hosting on Cisco platforms (Catalyst 9300 and 9400, Nexus 9300 and 9500, Catalyst 8000, ISR, ASR)
  • AWS CloudFormation Template
  • Mac OS Endpoint Agents
  • Pulse Endpoint Agents for external entities
In addition to the breadth of agents available, the deployment can easily be automated.  I’ve written a script that wrote the Raspberry Pi image to an SD card, then mounted it and applied customizations.  The MSI package can be used with the plethora of Windows software deployment tools, or a link can be given to end users to install on their own.  With DNA Center the image can be pushed to Catalyst switches in bulk.  The Docker images can be build with Docker files.  If that’s not enough, there’s also all the automation tools – Ansible, Terraform…
Getting ThousandEyes deployed throughout an environment can be done with ease.

What’s next?

That completes the agent installation.  The next installment in this series will cover some test scenarios, and walk through getting monitoring configured and tests created.

ThousandEyes Walkthrough Behind the Scenes – The Lab Build

Posted on by Dan Kelcher

This post will go over the planning of the ThousandEyes lab used in this series. To see past posts in this series expand the box below.

ThousandEyes Walkthrough Table of Contents

There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:

If you are following the series, this post is strictly informational.  It won’t contain any steps that need to be performed in the lab.  The goal is to provide insight into why I made the design choices I did with the lab.

The details on the lab build can be found here: https://www.mytechgnome.com/2022/04/thousandeyes-walkthrough-part-2-lab.html

And here’s an overview of the objective of this series: https://www.mytechgnome.com/2022/03/thousandeyes-walkthrough-part-1-what.html

CML

  • There are plenty of similar tools (GNS3, EveNG, etc) that are available, so why did I pick the paid tool?  The simple answer is licensing.  My understanding is CML is the only way to run virtual Cisco instances without running afoul of the EULA.  Yes, I could have used non-Cisco routers, but since Cisco is a major vendor it seemed reasonable to go with it.
  • The Personal version of CML has two flavors, Personal which allows 20 active nodes, and Personal Plus which allows 40 active nodes.  I built the lab using 20 nodes because the Personal Plus is an extra $150, and because the additional nodes would increase the resource requirements.  I wanted the lab to be as accessible as possible.  It could easily be extended to 40 nodes or higher, but 20 is enough to get basic testing done.
  • Even though the TE agents could be deployed to VMs, I wanted to use CML as a way to easily simulate scenarios where an engineer would need to do some troubleshooting.  Within CML links can be configured with bandwidth limits, latency, jitter, and loss.  The theory is that ThousandEyes should be able to detect and even alert on those conditions.
  • I am using version 2.2.3, even though version 2.3 is available.  The simple reason is that Cisco is still recommending version 2.2.3.  There are some known issues with 2.3, which is why I’m not running that.

IOSv Routers

  • Even though CML can run CSR 1000V and IOS-XR instances I decided to go with IOSv instances.  This was because of resource requirements.  The CSR 1000v and IOS-XR instances each require 3GB RAM, and with 14 routers that would consume an additional 35GB RAM over what the IOSv routers use.  For the purposes of the lab, the IOSv can do everything needed without the overhead.

Ubuntu

  • I wanted to keep as much of the lab in CML as possible, and running Ubuntu in CML aligns with that goal.  Of the Linux flavors that are available out of the box in CML, Ubuntu is the only one supported by ThousandEyes.
  • With Ubuntu being used in the CML lab it seemed reasonable to use Ubuntu for the Docker host as well.

Topology

  • I’ll admit I spent a lot of time working through different topology options.  At one point I had switches and HSRP in the design, but I decided to back away from layer 2 technologies to focus on layer 3.  The primary use case for ThousandEyes is looking at WAN links, and with the node limit in CML, it made sense to drop the L2 configurations to make room for more L3 devices.
  • I wanted to maximize the number of BGP AS configurations while maintaining multiple links, which is why there are 7 BGP AS configurations.  By simply shutting down specific links traffic could hit 6 of the 7 AS networks.  With some BGP reconfiguration that could be extended.
  • The two “Client” networks are intended to be what a network engineer would have in their environment.  Likely they’d have a lot more, but with the node limits having two networks is enough to test with.  Each of the client networks has two Ubuntu nodes that are running the TE Enterprise agent.  One of the Ubuntu nodes is also running Apache.  (more on Apache shortly)
  • In the “Public” network I wanted to add another BGP path outside the redundant ISP paths, and I wanted a service that was accessible.  With this being treated as public I opted to not run a TE agent there.
  • Access outside of the CML environment is done via the “External” network.  ThousandEyes is a SaaS service, which means the agents all need to be able to connect to the TE portal.
  • Even though the entire network is built using RFC 1918 addresses, the design is effectively using public addresses throughout the entire lab.  The “Client” addresses are propagated through the ISP and public networks, which isn’t typical in IPv4 deployments.  This was mainly choosing simplicity and efficiency.  If the client networks were masked then something like a VPN would be required to link the two client networks.  Though that better aligns with the real world, for the functional purposes of the lab it makes no difference.  Both ends need IP reachability and adding more NAT and VPN configuration work doesn’t provide a significant improvement in how the lab operates.

External Routing and NAT

  • On the external router, NAT is configured, which should allow internet access from the lab with no additional configuration needed.  The 192.168.1.0/24 network is excluded from translation with the intent that devices on the LAN (Docker, Windows, and Raspberry Pi agents) would be able to connect directly to devices in the CML lab.
  • For the LAN devices to reach the CML lab routes need to be added either to the LAN router or as static routes to each of the devices.  Using the LAN router requires the fewest changes, and is the most extensible.
  • Unfortunately not every environment is identical.  I suspect that there may be some issues with getting the routing working properly.  I spent a lot of time trying to decide if this routing solution was better than just using DHCP on the external router and doing full outbound NAT.  I decided that having the external agents able to have full connectivity to the internal agents was worth the added complexity.

Services

  • The Apache instances were set up just to create a simple webserver to establish HTTP connections.  For transaction tests, I will be using external websites.
  • Bind is deployed primarily for easy name resolution of the lab devices, and to have another service running inside the lab.  Since ThousandEyes can do DNS tests it made sense to include.

External Resources

  • The Docker, Windows, and Raspberry Pi agents are primarily just to provide the ability to test with those platforms.  The Docker and Pi agents are functionally similar to the Ubuntu agents running in the CML lab.  The Windows agent is an Endpoint agent, which brings a different set of functionality.  
  • I do expect that there will be improvements in test performance with these agents versus the ones in CML because there are fewer layers of abstraction.  I can’t imagine an Ubuntu agent running on a minimum spec VM inside KVM, that is running on the CML VM inside Workstation is going to be the most efficient.  Add in the software layers for the routers connecting those agents, and that only adds more potential performance impact.
  • As mentioned previously, internet access is required for ThousandEyes agents to reach the SaaS platform.  With that requirement in mind, it made sense to just use external websites for most of the testing instead of building elaborate web servers inside the lab.

Misc. Notes

  • Everyone has their preferred numbering scheme.  For this lab, I tried to come up with something that I could easily build on in a programmatic sense.  Yes, for the router links I could have used /30 or /31, but in a lab, I’m not worried about address consumption.  I built addresses based on the nodes being connected.
  • I’m sure someone somewhere will be upset that I don’t have passwords on the routers.  It’s a lab that I tear down frequently, and it’s inside a trusted network.  The risk of an attack is minimal, and worth it to not need to log in to each device.
  • The Ubuntu server version was the latest at the time of writing, and I went with Windows 10 to avoid some of the issues with getting Windows 11 deployed.
  • With the complexity of the build in CML, I decided it was easiest to just publish the YAML code.  Initially, I had intended to write up exactly how to build the lab, and provide configs for each device, but as I built it out it became clear that doing so would be quite cumbersome.  Using the YAML file should give more consistent deployments, with less manual work to get the lab running.
  • I’ve had several requests to incorporate AWS into this lab.  Currently, that’s outside the scope of the roadmap I have for this series.  The primary reason for that is because of the cost associated with AWS.  Once I get through the posts I have planned for this series I plan to investigate if I can leverage the AWS free tier to get useful data.
  • Despite most of the routers being in provider networks, each router has SNMP running.  The reason I did this was to show how ThousandEyes can use SNMP to add additional context to data, and in some cases, it can be used to trigger alarms.  In a real-world scenario you likely can’t get SNMP from provider networks, but you also likely have more than two network devices at a location.  The decrease in realism is more than made up for by not having to build out a complete LAN environment.
I’m sure there are plenty of things that I forgot to include here, and likely some good ideas that I didn’t even think about.  If you have any questions on the lab design please leave a comment below, or you can reach me on Twitter – @Ipswitch

ThousandEyes Walkthrough Part 2 – Lab build

Posted on by Dan Kelcher

This post will go over getting a ThousandEyes lab built out. To see all the posts in this series expand the box below.

ThousandEyes Walkthrough Table of Contents

There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:

Lab Build

Topology

I’ve built out this lab using VMware Workstation and Cisco Modeling Labs to simulate a network for ThousandEyes to monitor.

The lab is broken down into four types of sites, Client, ISP, Public, and External.  There are two Client sites, each consisting of two routers and two Ubuntu instances.  The routers are running BGP, and have SNMP enabled.  The first Ubuntu instance is only running a ThousandEyes agent, and the other instance is running a ThousandEyes agent and an Apache webserver.

The ISP networks are routers running BGP interconnecting all the other sites.  I have SNMP enabled on them just to show what ThousandEyes can do with SNMP monitoring.  Normally isn’t going to be accessible on ISP devices.

The Public zone is also running BGP and has an Ubuntu instance that is running DNS for the entire CML.LAB network.

The External site is used to bridge the lab environment to the network outside CML.  It has a static route out to the LAN gateway that is redistributed into BGP, and a static IP assigned on the LAN.  For traffic leaving the LAN, it has NAT configured.  This should reduce the configuration needs on the LAN side.  A static route can be added to the LAN gateway to send traffic to the External router, or static routes can be added to the individual devices that will connect to the lab network.

In addition to the CML lab, three additional devices will be deployed, an Ubuntu Server running Docker for ThousandEyes Enterprise Agents, a Windows 10 VM running the ThousandEyes Endpoint Agent, and a Raspberry Pi running the ThousandEyes Enterprise Agent.

This table breaks down the resources assigned to each node and the total amount of resources.  The CML VM will need to have enough assigned to it to allow the nodes inside it to run.


The Windows and Ubuntu Docker nodes will sit outside CML, as VMs in VMware Workstation.  There will also be a Raspberry Pi added to the environment.

Installation Prerequisites

Installation Process

The easiest way to get the lab up and running is to import a YAML file.  This file contains everything you need to get started, but some updates may be required.  The lab is configured for internet access, and there is a static IP and gateway assigned.  The LAN addressing might need to be updated to match your environment.

If you choose not to use the YAML import you can find the relevant node configurations in the YAML and then create and configure the nodes accordingly.
Expand each of the following sections for steps on how to build out the lab.

The YAML file can be downloaded from GitHub here: https://github.com/mytechgnome/CML-TE-Lab/blob/main/TE.yaml

Create a YAML file with this – Click to expand



Import YAML into CML – Click to expand

To import this into CML follow these steps:

  1. Copy the above YAML data into a new file
  2. Save the file as TE-Lab.yaml
  3. Log in to CML
  4. From the Dashboard Click Import
  5. Click in the File(s) to import area
  6. Browse to the location the YAML file was saved and select it
  7. Click Import
  8. It should import the lab successfully. Click the Go To Lab button

The entire simulation can be started at once, or the individual nodes can be started.  If they are being started manually start with the external connection, and work through all the routers first.  Then move on to PS3-1.  This node will take a few minutes to complete.  The remaining CS nodes can be started after PS3-1 completes its startup.

Credentials

Routers do not have a username or password to log in.  There is no enable password.
Ubuntu nodes: cisco/cisco

Verification tasks


Routers

  • show ip route
    • The route table should be populated, including a default route
  • ping 8.8.8.8
    • Should receive replies
    • If this fails verify the configuration of Gi0/0 matches the LAN requirements, and the CML VM NIC is configured for bridged access

PS3-1

  • systemctl status bind9
    • Should display active.  If this fails verify internet connectivity and then run these commands:
    • sudo apt-get update
    • sudo apt install install -o Dpkg::Options::=”–force-confold” bind9 -y
  • ping cisco.com
    • Should receive replies

CS1-2 and CS2-2

  • systemctl status apache2
    • Should display active.  If this fails verify internet connectivity and then run these commands:
    • sudo apt-get update
    • sudo apt install apache2
  • ping cisco.com
    • Should receive replies


Ubuntu Docker Host Deployment – Click to expand

  1. Open VMware Workstation and create a new VM by pressing Ctrl + N
  2. When the New Virtual Machine Wizard opens click Next
  3. Select the option for Installer disk image file (iso)
    1. Browse to the location of the Ubuntu Server ISO and click Next
  4. Enter a username and password, then click Next
  5. Enter a name for the VM, and verify the path, then click Next
  6. Set the virtual hard drive to 30GB, then click Next
  7. Click Customize Hardware
  8. Select the Network Adapter, change the Network Connection to Bridged, then click Close
  9. Click Finish
  10. Wait for the OS installation process to start
  11. Select your language and press Enter twice to select and confirm
  12. Use the arrow keys to select the NIC and press Enter
  13. Highlight Edit IPv4 and press Enter
  14. Press Enter to change the address assignment method and select Manual
  15. Use the arrow keys to move between fields filling out the IP address info, and then go to Save and press Enter when complete
    2. The default DNS server for the lab is 10.133.100.10, and the search domain is cml.lab
  16. Highlight Done, and press Enter
  17. Press Enter again to skip the Proxy config
  18. Press Enter again to use the default mirror location
  19. Use the arrow keys to highlight Done and press Enter to accept the default storage config
  20. Press Enter again to accept the file system config
  21. Highlight Continue and press Enter to confirm the storage settings
  22. Use the arrow keys to move between fields, fill out the Profile info, and then go to Done and press Enter when complete
  23. Press Enter again to skip Ubuntu Advantage
  24. Press Enter to enable SSH access, then highlight Done and press Enter
  25. Use the arrow keys to go down to highlight Done, and press Enter
  26. Wait for the installation to complete
  27. When the installation finishes highlight Reboot Now and press Enter
  28. When the server is back up log in
  29. Run the following commands to install Docker

sudo apt-get update
sudo apt-get install
ca-certificates
curl
gnupg
lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io -y


Windows 10 Endpoint Deployment – Click to expand

Create the Windows VM

  1. In VMware Workstation press CTRL+N to open the New Virtual Machine Wizard, and make sure Typical is selected, then click Next
  2. Select the option for Installer Disc Image File, and browse to the location you downloaded the Windows 10 ISO to then click Next
  3. Enter the name for the client and select the location
  4. Use the default hard drive size of 60GB (another drive will be added later for the iSCSI target storage), and click Next
  5. Click Customize Hardware
  6. Adjust the CPU and RAM as needed for your environment (2 vCPUs 4-8GB RAM would be recommended), and change the Network Adapter from NAT to Bridged
  7. Click close, verify the box is checked for “Power on this virtual machine after creation” and click finish.

Deploy the Windows OS


NOTE: While in the VM you will need to press Ctrl+Alt to release the cursor to get to your desktop

  1. >While the VM is booting you might see a prompt to press a key to boot from CD.  If that happens click into the window and press a key.
  2. Select the language, and keyboard settings
  3. Click Install Now
  4. On the Activate Windows screen click “I don’t have a product key”
  5. Select Windows 10 Pro and click Next
  6. Read through all of the licenses terms, and if you accept the terms check the box to accept them and click Next
  7. Select the Custom install option
  8. By default, it should already select Drive 0, which is the 60GB drive initially created.  Click next.  The OS install will start, so just let that process run.

OS Initial Config


Windows 10 has several steps to go through to get the OS configured before actually loading to a desktop.

  1. Select your regions and Click Yes
  2. Select your keyboard layout
  3. Skip adding the additional keyboard
  4. Wait a moment for it to progress to the account creation screen, then select “Set up for personal use” and click Next
  5. Microsoft is going to try to link to an online account, but since this is for a temporary lab PC click on “Offline account” in the bottom left.
  6. Microsoft really tries to push the online account, so again look in the bottom left corner and select “Limited experience”
  7. Enter a username and click Next
  8. Create a password and click Next
    2. The next screen will ask to confirm the password.  Reenter the password and click Next
  9. When prompted for the three security questions I just select the first three options and enter random characters.  This is a lab, and if I happen to forget the password I can easily recreate the VM.  Click Next
    2. Repeat the process for the other two questions.
  10. For the privacy settings, this really doesn’t matter, as it’s a lab machine that won’t exist for long.  Everything can be left enabled by default, or it can be disabled.  After applying the settings click Accept.
  11. On the Customize Experience page just click Skip
  12. Cortana… Microsoft really wants people to enable all their stuff.  Click “Not now” to move on.
  13. Success! The post-install prompts are done.  Now, wait for the configuration to complete.

Client OS config


To configure the OS there are only two tasks that are going to be performed.

  • Install VMware Tools
  • Configure DNS

Install VMware Tools

  1. Log into the VM using the password set previously
  2. Right-click on the VM in the Library and select Install VMware Tools
  3. Autorun should prompt to run, but if not then navigate to the D: drive and double click it.  That should kick off the Autorun for the installer.
  4. Follow the defaults for the installation.  Next > Next > Install > Finish and then click Yes when prompted for a reboot.

Configure DNS

  1. Open Powershell as admin
    1. Press the Windows key and type powershell
    2. Press Crtl+Shift+Enter to run as admin
  2. Run these commands:

Set-DnsClientServerAddress -InterfaceAlias Ethernet0 -ServerAddresses 10.133.100.10

Set-DnsClientGlobalSetting -SuffixSearchList cml.lab

There’s a lot to the lab build, but hopefully, it went smoothly.  If there were any issues you can add a comment to this post, or reach me on Twitter @Ipswitch.
As the lab build-out continues I may need to come back and edit the configuration here.  

CHANGELOG

v1.1
  • CML Lab YAML data
    • Corrected IR2-2 Gi0/0 IP configuration and BGP peering
    • Corrected IP assignment on PR3-2 – config was moved from Gi0/3 to Gi0/4
    • Added loopback interfaces to all routers (will be used for SNMP connections)
    • Updated DNS records to use loopback addresses
  • Lab config

What’s Next?

The next entry in this series will cover getting the ThousandEyes agent deployed into the lab, and getting things ready to start building tests and collecting data.

ThousandEyes Walkthrough Part 1 – The What and the Why

Posted on by Dan Kelcher

This post will go over what ThousandEyes is, and why you should be interested in learning how to use it. To see all the posts in this series expand the box below.

ThousandEyes Walkthrough Table of Contents

There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:

 What is ThousandEyes?

I’m not in marketing, so I’m going to avoid all the “founded in” type stuff (if you want to read that stuff check out the ThousandEyes site: https://www.thousandeyes.com/about/) Instead, let’s talk about what it means to IT professionals, and more specifically network engineers.  ThousandEyes is a monitoring tool (I know, one of many, but hear me out) that takes a different approach to monitoring.  We’re all familiar with SNMP monitoring.  Links go up, links go down.  The problem with this sort of monitoring is … well, it sucks for actual performance monitoring.  Sure, I can see the packet rate of a port.  I can use Netflow to look at what type of traffic it is.  None of this actually tells me how that link, or more importantly the service that uses that link, is performing.  More importantly still, what the end-user experience is of that service using that link.

I’ll get more into how ThousandEyes operates shortly, but before that let’s take a look at why we care about it.

Why ThousandEyes? 

“It’s slow” 

I think it’s safe to say those two words are possibly the most annoying words to hear as an engineer.  They are subjective, and often backed with little data.  I can’t look for “slow” in SNMP logs.  These types of issues typically result in spending hours looking at different interfaces, running tests, and often end with a shrug of the shoulders and either saying it’s a transient issue, or it’s on the other side.

“It’s a network problem”

There’s a phrase that can instantly raise the blood pressure of any network engineer.  Again, this statement is often followed with no useful information.  After that phrase is uttered the full weight of a Priority 1 outage is squarely focused on the network team, and now they shoulder the burden of proof before anything else happens.  I’ve had issues drag on for months because people believed, without evidence, there was a network problem, and no matter what I provided, it wasn’t enough.

I’ve often referred to the Internet as the Wild West.  Once traffic leaves the network I manage I lose visibility over it.  Tools like Netflow and SNMP no longer help.  I can’t leverage things like QoS to prioritize my traffic.  Instead, I leave it to the magic of TCP to make sure the traffic gets to the destination.  I’ve lost count of the number of calls where I’ve said “I see the traffic egress our perimeter, and it looked fine.” and similar statements.

I could go on, and on, and on.  I’d wager most network engineers have had similar experiences.

Enter ThousandEyes

With ThousandEyes we have a tool that helps quickly determine if something is slow, and where that might be occurring.  This moves the conversation from the realm of subjective user experience and wild accusations to objective, proactive detection of potential issues.  This is done through the use of Agents and Tests (more on those in a future post).  By running tests we can see hop-by-hop what is happening with that traffic, and most importantly, we can see it through networks that we don’t own.  

What’s the objective of this blog series?   

 The target audience is primarily network engineers, but application developers, server administrators, and countless other people in the IT field would benefit from knowing what this tool can do.

I’ll be building out a virtual lab topology and running ThousandEyes inside it to show what the tool is capable of.

My goal is to show that the tool is incredibly easy to use and powerful.  Over the years I’ve had plenty of vendors talk about how great their product is.  Every vendor thinks whatever their product is will be the greatest product ever.  I’ve watched sales reps move from vendor to vendor, and each new place happens to have the best widgets and gizmos.  No sales pitch here.  Just an IT guy that actually thinks this is an awesome tool, and it would be a great addition to most environments.

What’s Next?

In the next installment of this ThousandEyes Walkthrough series I’ll be detailing the lab environment that I will be using for testing.  Everything will be done using, VMware Workstation, CML, Windows and Ubuntu guests, and a Raspberry Pi for fun.  I’ll provide full configs so you can build out a similar environment.  The lab will include BGP, DNS, and web servers to allow different types of ThousandEyes tests to be configured.
-Spoiler Alert-
Here’s what I’m working on for the lab build:

Windows 11 VM deployment in VMware Workstation with TPM enabled

Posted on by Dan Kelcher

Windows 11 requires TPM, which adds some complexity to deploying it in VMware Workstation.  The good news is adding TPM to a VM is pretty simple.  Here’s how to get it working:

  1. Download the ISO from Microsoft: https://www.microsoft.com/software-download/windows11
    1. Go through the steps under the “Download Windows 11 Disk Image (ISO)” section
  2. In VMware Workstation create a new VM
    1. Select the Typical option
      1.  
      2. Click Next
    2. Select “Installer disc image file (iso)” and then browse to the Windows 11 ISO file
      2. Click Next
    3. Select the OS
      1. Select Microsoft Windows, and in the Version select Windows 10 and later x64
      3. Click Next
    4. Enter the VM Name and specify the location if not the default
      2. Click Next
    5. Set the hard drive configuration
      2. Windows 11 requires at least 64GB.  Make sure the minimum is set to at least 64GB
      3. Click Next
    6. Click Finish
    8. Enable TPM
      1. Right click on the VM and select Settings
      2. Click the Options tab at the top of the Settings window
      3. Click the “Access Control” option, and then click Encrypt
        2. In the window that pops up enter and confirm an encryption password
          2. Click Encrypt
        3. Click the Hardware tab to return to the hardware settings
        4. Adjust the CPU and NIC settings if needed
        5. The memory must be set to at least 4GB RAM
        6. Click the Add button
        8. Click “Trusted Platform Module”
          2. Click Finish
          3. Click OK
        9. Power up the VM
  3. Follow the on-screen instructions to complete the OS installation.
That’s it.  Windows 11 is now running as a VM with TPM enabled.

CCNP Data Center – DCIT 300-615 Exam Experience

Posted on by Dan Kelcher

 To finish off my CCNP: Data Center certification I successfully passed the DCIT exam.  While preparing for the exam I found there is very little preparation material outside of the official Cisco course.

I recently posted about my experience with the DCCOR exam, and in that covered my background and some of the general information on this set of exams.  You can read more about it here: https://www.mytechgnome.com/2022/02/ccnp-data-center-dccor-350-601-exam.html

About the Exam

The exam topics can be found here: https://learningnetwork.cisco.com/s/dcit-exam-topics

I went over the specific technologies in the DCCOR exam review, but I’ll quickly touch on them here as well.  The exam is specifically focused on troubleshooting the various technologies in each of these areas

Network (25%)

This section covers L2/L3 technologies including vPC, LACP, STP, OSPF, BGP, PIM, and FHRP (mainly HSRP).  Overlay protocols are also included, primarily VXLAN, but OTV is in there as well.  Of course ACI is also covered here.

Compute (25%)

Under the Compute category it’s exactly what you’d expect to see.  Lots of UCS, and though it doesn’t specifically call out HyperFlex or Intersight I think it’s fair to assume that those would be in scope.  Remember, Cisco states in the exam objectives that “The following topics are general guidelines for the content likely to be included on the exam. However, other related topics may also appear on any specific delivery of the exam.” 

Storage (15%)

Again, nothing unexpected here.  There’s fibre channel and FCoE, and you need to be able to troubleshoot issues that can come up with those protocols.  One specific call out here though is storage is 15% of the exam, but only has one major subtopic.  That leads me to a presumption that there will be a significant number of FC questions compared to something like vPC.

Automation (15%)

Like the Storage section, there’s only two subtopics for 15% of the exam.  It seems sensible to be very familiar with EEM, scheduling, and automation tools.

Management and Operations (20%)

In the final section the topics include the firmware management and security topics that were covered in DCCOR under different parent topics.  This topic feels a little more like a grab bag, being worth 20% and having the most subtopics.

Exam Prep

The biggest resource used for this was the preparation work I did for the DCCOR exam.  Both exams cover the same technology areas, but the DCIT is strictly on troubleshooting, instead of understanding how to use each piece.
As mentioned earlier, the only resource I could find on the exam was the Cisco training course.  Well, that’s not entirely true.  Google was more that willing to provide links to brain dump sites, and I found a set of practice tests on Udemy.  Unfortunately the tests on Udemy appear to be from the previous DCIT v6 300-180 exam, with questions on topics that are no longer in the exam objectives.  I suspect the author just took the same exam they wrote for the 300-180 and renamed it for 300-615.
I ended up purchasing the e-learning bundle from Cisco Learning Network: https://learningnetworkstore.cisco.com/on-demand-e-learning/troubleshooting-cisco-data-center-infrastructure-dcit-v7.0/ELT-DCIT-V7-023907.html mainly for the labs offered.  There are 30 guided labs included, with access to ACI, UCS, MDS, and Nexus hardware.  I found this much more useful than working with the limitations from the UCS Emulator, ACI Simulator, CML, dCloud, and DevNet.  Plus it was less expensive than looking at rack rentals, and it was cheaper and easier than searching eBay for all the hardware to build my own lab.
The video training in the course was pretty good.  Much of it was review from the DCCOR material, but that’s not a bad thing.  It helps to reinforce that learning and to use that understanding to help in the troubleshooting process.
The labs were decent.  I appreciated that they were broken out with your objective, and had a button to see the steps to work through getting the correct solution.  Often I found I took a different path to get to the answer, but I was usually able to figure it out on my own.  I would have liked to have seen more potential scenarios though.  Usually each topic area only had a few misconfigurations.  With some topics there wasn’t a lab at all.  There were no labs for anything under the Automation or Management and Operations topics.  
Within the lab you have full access to the hardware, which allows you to create your own scenarios to some extent.  One example would be EEM, a topic clearly listed in the exam topics, but there’s not a lab for it.  Since there’s not a scenario for it I tried to create my own.  It’s very difficult to build a troubleshooting problem for yourself because you likely know where the issue is when you set it up.  What I did instead was to create a scenario that I knew would not work, but then I would look at different show commands to see how it worked.  Sticking with the EEM example, here some of the questions I worked off of:
  • What happens if you create multiple events, but make a mistake in the tagging?
  • What happens if you don’t add event-default when you need it, or do add it when you don’t?
  • What happens if the event is based on CLI entries, but a user doesn’t type the full command, or uses an alias?
The idea was to work in reverse.  I know the problem, but I am learning how that problem presents.  This can be fun coming up with all the different scenarios of how something could break, and then seeing how that shows up.
The same process can be used for other areas too.  Here’s a few questions to work with on the storage side:
  • What’s the output of show flogi database or show fcns database if
    • a zoneset isn’t active?
    • a zone isn’t part of the active zoneset?
    • a port is assigned to the wrong VSAN?
    • a port mode is incorrect?
    • a WWN is incorrect in a zone config?
    • the FCoE VSAN to VLAN mapping is incorrect?
  • Using the above examples, are there other show commands that would be useful?
The more you know about how problems present themselves the easier the exam will be.  Of course, these are only a few examples what you could encounter on the exam.  The more you build out the better off you’ll be.

Exam Experience

Unfortunately Cisco has removed the simulation questions from the exams.  The TSHOOT exam was one of my favorite tests because it was heavy in the simulation side.  The sims had their share of issues, but I prefer them over the rote memorization of command syntax.  That said, the DCIT exam is the typical multiple choice type exam.  There might be some multi-select or drag-and-drop for some added flair, but no more sims.
Overall I thought the exam wasn’t bad.  Most of the questions were clear, but there were some that had me scratching my head.  I had a few questions that waded into the rote memorization world, and some that seemed very unlikely scenarios.  There were a few questions that were fairly complex, and took some work to find the right answer.
Since the exam is focused on troubleshooting I found it to be easier to work with than the DCCOR exam.    The DCCOR topic list is longer, and it includes verbs like Describe, Apply, Analyze, Implement, Evaluate, and Explain.  The DCIT had one verb, Troubleshoot.

Final Thoughts

I wish the exam had sim questions, but even without, I prefer the scenario-based questions.  I’d like to see less rote memorization and edge case questions, and more focus on things that are more realistic to face in the real world.  Given that though, I think the exam was fair.  If you have a good understanding of how these technologies work, and what happens when they are misconfigured then this exam shouldn’t be too bad.

CCNP Data Center – DCCOR 350-601 Exam Experience

Posted on by Dan Kelcher

 I recently passed the DCCOR exam, and since it’s a difficult test I thought I’d share my experience with it.

My Background

Before getting into my preparation and exam experience I’m going to provide some background about the knowledge I had coming into this.  Everyone has different skills and experiences, and that’s going to have an impact on how they prepare.  The exam is broken down to five topic domains, so I’ll use that format.

Network

I’ve worked in the IT for about 20 years, and I already have my CCNP:EN (converted from the old CCNP:RS).  The L2 and L3 topics were mostly review, but the overlay technologies were new for me.  I also came in with very little practical ACI knowledge.  I’ve worked in environments where ACI was deployed, but I wasn’t responsible for maintaining it.

Compute

This is an area I felt fairly comfortable with initially.  I’ve worked with UCS off and on for over seven years.  I’ve deployed and managed blades and rack servers, as well as multiple HyperFlex environments.  I’ve also worked a bit with Intersight.

Storage

For a relatively brief period in my career I did a lot of SAN deployments, so I was somewhat familiar with the overall storage concepts.  I stopped working with fibre channel maybe five years ago and had worked mainly with iSCSI connectivity or virtual SAN environments since then.

Automation

I have the DevNet Associate certification, so I’m at least passably familiar with automation.  However, it’s not something I do day-to-day.

Security

In IT Security is part of everything we do.  I’m well versed in the overall security concepts, but I rarely go in and deal with RBAC or similar settings.  Usually that was something where the roles were configured when a platform was deployed, and I could go months without needing to make any changes aside from adding users to the correct groups.

About the Exam

The exam topics list can be found here: https://learningnetwork.cisco.com/s/dccor-exam-topics
First off, I want to say that this exam is massive.  If you compare it to what the CCNP:RS exams were I think that can illustrate my point.  In the old 300-101 ROUTE exam 40% of the exam was on routing.  The remaining 60% was split between five categories.  The 300-115 SWITCH exam had 65% of the exam focused on L2 technologies, with the remaining 35% split between two other categories.
In contrast, the 350-601 DCCOR has 25% of the overall exam on Networking.  In that 25% it includes OSPF and BGP (covered in the previous ROUTE exam), and L2 topics like LACP, and STP.  However, the DCCOR also includes PIM, vPC, overlay protocols (OTV and VXLAN), and ACI.  Just the breadth of topics in the Network topic makes this exam feel bigger, and harder than the previous CCNP:RS.
The remaining 75% of the exam is comparatively simple.  The Network portion is 25% of the exam, but contains 10 subtopics.  The remain 3/4 of the exam is a total of 16 topics.  By topic area, I would have expected the Network portion to be closer to 40% of the exam.
The Compute portion of the exam covers the UCS servers, HyperFlex, Intersight, and general management of a compute environment.  From a topic standpoint nothing unexpected.
Moving to the Storage section of the exam it is, like the Compute section, pretty straight forward.  I will admit that I was surprised the topics didn’t mention iSCSI at all.  Aside from that the Storage is what you’d expect.  It covers how fibre channel works, zoning, NPV/NPIV, VSAN, etc.
Automation is another topic that seems easy, but can get really broad quickly.  Though the topic list doesn’t go into the details I think it’s fair to assume that when it lists things like REST API that means you will need an understanding of how an API would be used to manage any of the hardware covered in the exam.
Lastly, the Security is focused heavily on the AAA and RBAC configurations for the different technologies covered in the exam.  Additionally, there are some more specific subtopics for each technology type.
One final note on the exam topics – Cisco has a line on that exam topics page that I think is important and often overlooked by test takers “The following topics are general guidelines for the content likely to be included on the exam.” Which means that things like iSCSI could actually appear on the exam.  Even though the topic isn’t specifically called out, it is a storage networking protocol that has seen wide adoption, and that means it could appear on the exam. 

Exam Prep

I started a blog post about the resources I used for the exam here: https://www.mytechgnome.com/2021/12/ccnp-datacenter-journey-dccor-350-601.html
I started with the Cisco Press Official Certification Guide, and I think it’s an awesome resource.  Unfortunately, I struggled with reading it and I literally found this book was putting me to sleep.  (That’s more a me problem than a book problem).  I found it worked better for me to use it as a reference for specific topics instead of trying to read it cover-to-cover.  One other thing I realized early on is the DCCOR exam replaced the CCIE written exam.  I found myself questioning if the depth covered in the book was for the benefit of the DCCOR exam, or as a resource for the CCIE lab (spoiler alert – it is for the DCCOR).
When I was having difficulty getting through the book I switched to the INE material.  Their video training is great, but incomplete.  At least at the time I went through it the HyperFlex and Automation sections are listed as “Coming Soon”.  It also seems like the Storage section was just duplicated from the previous CCIE:DC training.  I thought it was very hands-on like what I’d expect when preparing for a CCIE lab, but it seemed to focus more on the configuration requirements than I would have expected would be needed for the DCCOR.
I switched over to the CBT Nuggets training to compliment what INE covered, and to fill the gaps in what INE hadn’t published.  There were a number of things covered in the CBT Nuggets material that wasn’t covered in the INE material, so I was happy I went through it as well.

Labs

Books and video training are great, and they have their place, but the fun part of learning new stuff is to actually do it!  To prepare for the exam I used a few different tools provided by Cisco.
For the Networking topics CML is nearly perfect.  With CML you can create labs to cover most of the network topics.  I created maybe ten different lab environments and built out a bunch of different configurations:
  • Switching: STP, vPC, LACP, HSRP, VRRP
  • Routing: OSPF, BGP (iBGP and eBGP), PIM
  • Overlay: OTV (CML can’t do OTV on the data plane so you can’t actually pass traffic), VXLAN
I also did combination labs.  For example, create a BGP and OSPF “provider” network, and run VXLAN over it.  Then in the “provider” environment I would enable or disable multicast and adjust the VXLAN deployment accordingly.  I would also configure anycast gateways in each network.  It’s also worth pointing out that you can enable the NX-API in CML, and run automation tools against it.  That’s a great way to practice both the automation skills as well as the networking skills.
To get hands-on with ACI I primarily used the ACI simulator.  With the simulator I was able to do a large deployment with three APICs, as well as two spine and two leaf switches.  I was able to go through and build out EPGs, contracts, domains, etc.  One thing that I found particularly cool with the ACI simulator was that I could SSH to spine and leaf switches.  This allowed me to get hands-on with the CLI of the underlying hardware.
On the Compute side, I used the UCS emulator.  This allows you to run UCS manager and create virtual UCS hardware and policies.  Like the previous tools, the UCS emulator also allows CLI access to the virtual FIs.
The storage portion of the exam was the most difficult to actually get hands-on labs for.  There are some labs available through the DevNet Sandbox and dCloud that have MDS switches, but the labs are limited in what they can do.  The good news is that comparatively the commands for storage are far less than what you need to know for other topics.
With both Automation and Security all of the above resources can be used.  DevNet also has a lot of good resources for automation.  The security side is mainly just being familiar with how accounts and roles are created and the configuration of RADIUS/TACACS connections.

Exam Experience

I took the exam remotely, which generally has been a positive experience.  There are a few of tips that I can provide for the online exam.  First would be to go through the pre-check to make sure everything works as expected.  I used a laptop with an external monitor, and I found that it defaulted to the laptop webcam (with the laptop lid closed this didn’t work), but there’s a dropdown to select the webcam so it was an easy fix.  Make sure you have good audio.  The proctors often do PC-based calls, and I found them difficult to hear using my monitor speakers.  Switching to the laptop speakers solved that problem.  Also I recommend installing the Pearson app when checking in.  I had numerous issues using the web-based tool.  It seemed like when trying to upload images they were blurry in the web app, and when I switched to the mobile app the issues were resolved.
Cisco has an NDA all test takers have to accept, so I’m going to be very vague in what I say about the actual exam.  The first thing is I will reiterate that this exam is broad.  For each topic area (ACI, Nexus, UCS, MDS, etc.) be familiar with all of the different tools used to manage them.  That means GUI, CLI, and API.  Make sure you are paying close attention to the exam topics, and you have a firm grasp on each and every topic item.
I did get some questions that seemed to be poorly worded, and a couple left me confused on what specifically was being asked.  In some cases it was just be the way that I read the question, and taking a moment then rereading it and it made more sense.  In others, it didn’t matter how many times I read it, it just didn’t make sense.
One other trend that I’ve found with exams is the tendency to find the most obscure question possible.  As engineers we have access to context sensitive help.  Preparing for the exam I watched CCIEs extensively use the “?” to find the syntax that was needed, or look for a config option in the wrong area.  I find it frustrating when on an exam there are syntax questions that are needlessly specific.  An example would be something like if something is measured in bits, kilobits, bytes, kilobytes, etc.  Another example would be if a timer is configured in seconds or milliseconds.  If these situations ever came up, I’d have access to tools without needing to memorize the plethora of commands and options.  To make matters worse, often when these questions come up they are on commands that are rarely used.  This exam is no different.  Make sure you are committing the exact syntax of commands to memory.
On my first attempt I failed, and looking at the score report I’m guessing it was close.  The passing score and received score aren’t provided anymore, but the percentage per topic is displayed. Not surprisingly, the areas I performed the worst in were Storage, Automation, and Security.  Those areas were where I had much more general knowledge initially, and it showed.  However, after having actually taken the exam I knew where I was weak, and the level of depth on the exam.  I went back and reviewed the Network and Compute topics, and spent more time digging in to the areas I needed to improve.  When I retook the exam I was able to pass the exam.

Final Thoughts

This exam is tough.  I can confidently say it’s one of the most difficult exams I’ve ever taken.  It’s a mile wide, and it can also get quite deep.  There are plenty of challenging questions that really test your knowledge.  With enough preparation and practice it is something that can be accomplished.

CCNP Datacenter Journey – DCCOR 350-601 and DCIT 300-615 – Resources

Posted on by Dan Kelcher

 

I’m working towards attaining the CCNP: Data Center certification.  I’ll be compiling a list of resources and tips that I used along the way.

Software

VMware Workstation Pro – https://store-us.vmware.com/workstation_buy_dual_new

Cisco Modeling Labs – https://learningnetworkstore.cisco.com/cisco-modeling-labs-personal/cisco-modeling-labs-personal-plus/CML-PERSONAL-PLUS.html

UCS Emulator – https://software.cisco.com/download/beta/1850014776

ACI Simulator – https://software.cisco.com/download/home/286283149/type/286283168/release/5.2(1g) (If you don’t have access to the software you can work with a Cisco SE to grant you access)

Use this command in Linux to merge the downloaded files into a single OVA to import into VMware Workstation

cat acisim-5.2-1g_part1.ova acisim-5.2-1g_part2.ova acisim-5.2-1g_part3.ova acisim-5.2-1g_part4.ova acisim-5.2-1g_part5.ova acisim-5.2-1g_part6.ova > acisim-5.2-1g.ova

Material

Cisco Press Official Cert Guide
INE CCNP:DC training course
CBT Nuggets CCNP:DC training course