I gave this presentation at the ChiNOG 11 conference, May 11th, 2023. https://chinog.org/
Category: Uncategorized
Intro to Automation – Learning while having fun with Autonauts!
I have heard too many network engineers push back on learning how to automate things because they “don’t want to be a software developer.” There’s a slight problem with that argument, though. You don’t need to be a “software developer” to get into automation. Automation is surprisingly painless, and it can even be fun!
One recommendation I would give people looking for a fun and easy approach to learning automation would be to look at games like Autonauts (https://www.humblebundle.com/store/autonauts) or if you want to add in some combat Autonauts Vs. Piratebots (https://www.humblebundle.com/store/autonauts-vs-piratebots)
Both games are surprisingly fun to play. You start with some simple resources and need to craft and build your way to greatness. Luckily, you have bots at your disposal that you can train to do pretty much anything. Chop down trees. Collect materials. Craft tools. Build. Autonauts vs. Piratebots adds in attack and defense. The best thing that the bots can do… build more bots!
This is an example of a relatively simple script for a bot. It uses a drag-and-drop interface and can learn tasks by recording the player’s character’s actions.
As can be guessed by its name, this bot is a stone miner. To mine stone in both Autonauts games, you need a pick. During a game, hundreds, if not thousands, of units of stone are required. Why mine all of that manually when you can have a bot do it?
This script uses a repeat loop, with a condition of “forever!” meaning the bot will always be running this task.
Inside that first Repeat loop is another Repeat loop, and this one has an exit condition when the bot’s hands are full. When the bot isn’t holding anything, it will perform the action of moving to the Crude Pick Storage location and then the action of removing a pick from storage.
Now that the bot is holding a pick, it exits the repeat loop and moves on to the next step in the script. The next Repeat loop will run continuously until the tool breaks, and the bot’s hands are empty. During this loop, the bot will look for a stone deposit, move to it, and then use the pick to mine the stone.
The final script results in a bot that will mine until its pick breaks, and when that happens, it will get a replacement pick and return to mining. The bot will continue producing stone as long as there are picks in the storage location and open stone deposits.
Of course, the work doesn’t stop with just this one bot. The crafting and storing of picks will require automation. All that stone that was mined will need to be moved to a storage location.
This script could also be improved upon. The bot can move tools to a backpack and retrieve them. The mining location could be tied to a spot indicated by a sign. The script can be saved and then linked to other bots.
As the player progresses through the game, the tasks become far more complex. Each bot has limited memory in its “bot brain,” which limits each script’s length. A difficult task requires efficient scripting or multiple bots doing parts of the script.
Sure, the game doesn’t specifically cover anything about network automation. However, that’s where I think the value is. The best part of this game is how it makes you think of different ways to build out the various tasks. There are several mechanisms beyond the repeat loops that can be used. It shifts from a linear focus for each task to a more systematic approach.
Traditional CLI configuration is linear. Log into a device, move through the different config layers, and apply the config change. Network automation can use that same linear methodology, but the real power of it is when logic is layered on. Using if/then/else statements, while and for loops, variables, functions, etc., can all increase the power of a script significantly.
The beautiful part about network automation is much like a script in Autonauts; it can be improved iteratively. As new techniques are learned, they can be applied to increase the efficiency of the scripts. Those scripts can be reused later or modified to suit changing needs.
ThousandEyes Walkthrough Part 4.3.2 Scenario 2 – Enterprise DNS test configuration
This post will go over the second scenario for the ThousandEyes lab. To see all the posts in this series expand the box below.
- Part 1 – The What and the Why
- Part 2 – Lab build
- Part 3 – Enterprise and Endpoint Agent Installs
- Part 4.1 – SNMP Monitoring
- Part 4.2 – Scenarios and Test Types
- Part 4.3.1 – Scenario 1 – Enterprise agent to agent test configuration
- Part 4.3.2 – Scenario 2 – Enterprise DNS test configuration <–You Are Here
- Part 4.3.3 – Scenario 3 – Enterprise and Endpoint HTTP test configuration (Coming soon)
- Part 4.3.4 – Scenario 4 – Enterprise Page Load test configuration (Coming less soon)
- Part 4.3.5 – Scenario 5 – Enterprise Transaction test configuration and Endpoint Agent Browser Settings (Coming more less soon)
- Part 4.4+ – Details TBD
There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:
- Behind the Scenes – The Lab Build
- Ok, there’s only one so far, but I plan to add more where it makes sense.
Scenario 2
Create an Enterprise DNS Server test
- Log in to ThousandEyes (I presume this skill has been mastered by now)
- On the left side, expand the menu, then click on Cloud and Enterprise Agents to expand that list, and then click Test Settings
- Click Add New Test.
- This will be a DNS Server test. Click DNS for the Layer, and then DNS Server for the Test Type
- Under Basic Configuration, in the Domain field enter cml.lab
- As before, set the interval to 30 minutes to reduce the test load
- In the Agents field, select all the enterprise agents deployed
- Enter the lab DNS server IP in the DNS Servers field: 10.133.100.10
- Uncheck the Enable box for alerts
- When complete it should look like this:
- Click Create New Test
ThousandEyes Walkthrough 4.3.1 Scenario 1 – Enterprise agent to agent test configuration
This post will go over the first scenario for the ThousandEyes lab. To see past posts in this series expand the box below.
- Part 1 – The What and the Why
- Part 2 – Lab build
- Part 3 – Enterprise and Endpoint Agent Installs
- Part 4.1 – SNMP Monitoring
- Part 4.2 – Scenarios and Test Types
- Part – 4.3.1 Scenario 1 – Enterprise agent to agent test configuration <–You Are Here
- Part 4.3.2 – Scenario 2 – Enterprise DNS test configuration
- Part 4.3.3 – Scenario 3 – Enterprise and Endpoint HTTP test configuration (Coming soon)
- Part 4.3.4 – Scenario 4 – Enterprise Page Load test configuration (Coming less soon)
- Part 4.3.5 – Scenario 5 – Enterprise Transaction test configuration and Endpoint Agent Browser Settings (Coming more less soon)
- Part 4.4+ – Details TBD
There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:
- Behind the Scenes – The Lab Build
- Ok, there’s only one so far, but I plan to add more where it makes sense.
Scenario 1
Create an Enterprise agent-to-agent test
- Log in to ThousandEyes (I presume this skill has been mastered by now)
- On the left side, expand the menu, then click on Cloud and Enterprise Agents to expand that list, and then click Test Settings
- Click Add New Test.
- This will be an agent-to-agent test. For the layer select Network, and then under Test Type select Agent to Agent.
- Under Basic Configuration there are a few things to set here.
- Click in the Target Agent field
- On the right side, click Enterprise to filter the list down to only our Enterprise agents
- NOTE: the agents listed here are Cloud agents, and those can be used to test public services from locations all over the world
- Select the CS1-2 agent
- The interval is how often these tests are performed. Since this is a lab we’ll back this off to a 30-minute interval to reduce the number of tests being run.
- In the Agents field, we’ll select the source agents. Again, filter based on Enterprise agents, and then select all agents except CS1-2
- NOTE: Selecting the North America group (or your local region) will include the CS1-2 agent, and that won’t allow the test to be created because a source and target are the same.
- Under Direction, select Both Directions
- Leave the Protocol option set to TCP
- Check the box to Enable Throughput monitoring, then leave the duration with the default 10s time.
- Leave Path Trace Mode unchecked
- Uncheck the Enabled box next to Alerts. We’ll cover alerts later on in this series
- When everything is completed it should look like this:
- Click Create New Test
- Click the ellipsis (…) to the right of the test, and then in the menu click Duplicate
- Correct the test name
- Change the Target Agent to CS2-2
- In the Agents field uncheck CS2-2 and check CS1-2
- NOTE: If you can’t uncheck CS2-2 then click the box next to the region, and that should unselect all agents. Then reselect all agents except CS2-2
- When complete it should look like this:
- Click Create New Test again
ThousandEyes Walkthrough 4.2 Scenarios and Test Types
This post will go over the scenarios that will be used for the ThousandEyes lab, as well as the different test types available in ThousandEyes. To see past posts in this series expand the box below.
- Part 1 – The What and the Why
- Part 2 – Lab build
- Part 3 – Enterprise and Endpoint Agent Installs
- Part 4.1 – SNMP Monitoring
- Part 4.2 Scenarios and Test Types <–You Are Here
- Part 4.3.1 Scenario 1 – Enterprise agent to agent test configuration
- Part 4.3.2 Scenario 2 – Enterprise DNS test configuration
- Part 4.3.3 Scenario 3 – Enterprise and Endpoint HTTP test configuration (Coming soon)
- Part 4.3.4 Scenario 4 – Enterprise Page Load test configuration (Coming less soon)
- Part 4.3.5 Scenario 5 – Enterprise Transaction test configuration and Endpoint Agent Browser Settings (Coming more less soon)
- Part 4.4+ – Details TBD
Scenarios
Business Use Cases
- Both of the corporate campuses (client site 1 and client site 2) house resources critical to business processes. It is important that these resources are accessible and performance meets the business requirements.
- It’s known that critical applications are dependent on other network services, but there is a concern that the underlying services aren’t able to support the applications.
- An externally managed web application is used frequently used by employees, and if it were unavailable or slow that would negatively impact the employees’ productivity.
- Two internal web applications are used by customer-facing staff, and if these applications are unavailable or poorly performing for any location that would negatively impact the customer experience.
- A critical business process relies on a third-party web service. If there are delays in any stage of this process it can cause a significant impact.
Translation into Technical Requirements
- Network performance from all enterprise locations should be monitored.
- DNS has been identified as a critical service that other applications are dependent on. The CML.LAB domain must be monitored for availability and performance.
- Connection performance and availability to these HTTP servers must be monitored. The application vendor should be monitoring the application performance
- NOTE: There’s potential value in monitoring application performance both to keep track of SLA metrics, and also to speed the identification of an issue, but for this example, we’ll take the simplest approach.
- In addition to monitoring the HTTP connections, the time required to access these applications should be monitored
- Multiple web pages need to be monitored, and it needs to happen in a sequence similar to how a user would interact with the application.
Test Types
ThousandEyes has a total of 12 Enterprise test types split across 4 categories.
- Routing
- BGP
- This test looks at BGP peering (the lab environment isn’t peered out to the internet, and setting up private peering is going to be out of scope for the lab)
- Network
- Agent-to-Server
- This creates either a TCP or ICMP connection to the target address and monitors loss, latency, and jitter
- Agent-to-Agent
- Creates a connection between two ThousandEyes agents, and allows bidirectional TCP or UDP testing, monitoring loss, latency, jitter, and optionally throughput.
- DNS
- DNS Server
- This is pretty straightforward, a DNS record and DNS server are entered, and the test checks for resolution of that DNS record.
- DNS Trace
- A DNS trace queries top-level DNS servers and then works down through name servers to show the path used to resolve a DNS query the fastest.
- DNSSec
- With this test, the validity of a DNSSEC entry can be verified
- Web
- HTTP Server
- An HTTP(S) connection is made to a web server, effectively this is like using cURL to check if a connection can be established.
- Page Load
- Building on the HTTP test, the page load actually loads the HTML and related objects and tracks the time for each step in the process.
- Transaction
- Continuing to build on the Page Load test, a transaction test uses a Selenium browser and a script to simulate user interaction with a web application.
- FTP
- Establishes an FTP (or SFTP/FTPS) connection, and attempts to download, upload, or list files.
- Voice
- SIP Server
- This test checks SIP access to a server (by default TCP 5060), and under the advanced options, it can be configured to attempt to register a device with the SIP server.
- RTP Stream
- This is similar to an agent-to-agent test, but since it tests specifically focuses on RTP it includes MOS and PDV metrics in addition to the loss, latency, and jitter provided by normal agent tests.
There’s a lot more detail around tests and some of the advanced options available for each. More info can be found here: https://docs.thousandeyes.com/product-documentation/internet-and-wan-monitoring/tests
The Endpoint tests have two different methods to choose from. The first is a scheduled test, which functions in a similar way to the Enterprise tests. Endpoints can either do agent-to-server network tests or HTTP web tests. The other option, which is unique to Endpoint agents, is the Browser Session test. A Browser Session test uses a browser plugin (installed as part of the Endpoint agent installation) that collects data from the browser based on real-time interactions with web applications.
- Routing
- BGP
- This test looks at BGP peering (the lab environment isn’t peered out to the internet, and setting up private peering is going to be out of scope for the lab)
- Network
- Agent-to-Server
- This creates either a TCP or ICMP connection to the target address and monitors loss, latency, and jitter
- Agent-to-Agent
- Creates a connection between two ThousandEyes agents, and allows bidirectional TCP or UDP testing, monitoring loss, latency, jitter, and optionally throughput.
- DNS
- DNS Server
- This is pretty straightforward, a DNS record and DNS server are entered, and the test checks for resolution of that DNS record.
- DNS Trace
- A DNS trace queries top-level DNS servers and then works down through name servers to show the path used to resolve a DNS query the fastest.
- DNSSec
- With this test, the validity of a DNSSEC entry can be verified
- Web
- HTTP Server
- An HTTP(S) connection is made to a web server, effectively this is like using cURL to check if a connection can be established.
- Page Load
- Building on the HTTP test, the page load actually loads the HTML and related objects and tracks the time for each step in the process.
- Transaction
- Continuing to build on the Page Load test, a transaction test uses a Selenium browser and a script to simulate user interaction with a web application.
- FTP
- Establishes an FTP (or SFTP/FTPS) connection, and attempts to download, upload, or list files.
- Voice
- SIP Server
- This test checks SIP access to a server (by default TCP 5060), and under the advanced options, it can be configured to attempt to register a device with the SIP server.
- RTP Stream
- This is similar to an agent-to-agent test, but since it tests specifically focuses on RTP it includes MOS and PDV metrics in addition to the loss, latency, and jitter provided by normal agent tests.
What’s next?
Network Field Day 28 – Recap and Review
I had the opportunity to participate as a delegate at Network Field Day 28, and I wanted to share my experience.
What is Network Field Day?
Vendor Presentations
Day 1
Juniper https://www.juniper.net/
ThousandEyes Walkthrough Part 4.1 – SNMP Monitoring
This post will go over enabling and using SNMP monitoring in ThousandEyes. To see past posts in this series expand the box below.
- Part 1 – The What and the Why
- Part 2 – Lab build
- Part 3 – Enterprise and Endpoint Agent Installs
- Part 4.1 – SNMP Monitoring <–You Are Here
- Part 4.2 – Scenarios and Test Types
- Part 4.3.1 – Scenario 1 – Enterprise agent to agent test configuration
- Part 4.3.2 – Scenario 2 – Enterprise DNS test configuration
- Part 4.3.3 – Scenario 3 – Enterprise and Endpoint HTTP test configuration (Coming soon)
- Part 4.3.4 – Scenario 4 – Enterprise Page Load test configuration (Coming less soon)
- Part 4.3.5 – Scenario 5 – Enterprise Transaction test configuration and Endpoint Agent Browser Settings (Coming more less soon)
- Part 4.4+ – Details TBD
There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:
- Behind the Scenes – The Lab Build
- Ok, there’s only one so far, but I plan to add more where it makes sense.
With the lab environment built, and the agents installed and online, now it’s time to start actually getting monitoring data through ThousandEyes!
If you haven’t followed along with the previous posts in this series you can find the lab build here: https://www.mytechgnome.com/2022/04/thousandeyes-walkthrough-part-2-lab.html and the agent installation here: https://www.mytechgnome.com/2022/04/thousandeyes-walkthrough-part-3.html
This lab requires version 1.1 of the lab build. Verify the lab you are using is 1.1 or newer. If it’s not, look at the CHANGELOG section near the bottom of the lab build post: https://www.mytechgnome.com/2022/04/thousandeyes-walkthrough-part-2-lab.html
SNMP Configuration
- Open a web browser and navigate to https://www.thousandeyes.com/
- Log into your account
- Click the Hamburger icon
in the top left
- Expand Devices
- Click on Device Settings
- There might be a Get Started with Devices splash screen, or it will take you directly to the Devices page.
- On the right in the Basic Configuration enter the scan details
- In Targets enter the following subnet: 10.255.255.0/24
- In the Monitoring Agent drop-down select CS1-1
- Under Credentials click “Create new credentials”
- If the Credentials don’t auto-populate then click the dropdown and select the TE-SNMP that was just created
- Occasionally devices may not be picked up on the first discovery, but if the box is checked to “Save as a scheduled discovery” it will retry every hour
- Click Start Discovery
- Wait for the discovery process to complete – this might take a few minutes
- NOTE: There seems to be a bug in the UI where it displays a “No devices found” error, even though all the devices were discovered.
- Click back to the main section of the page and the Add Devices panel will disappear
- Click the Select All checkbox on the top left of the device list, then click Monitor at the bottom of the page.
- Wait a few minutes for the devices to show Green under the Last Contact column
SNMP Toplolgy
- Hover over the menu icon in the top left, then under Devices click on Views
- The Device Views will show some metric data on the top, and the topology on the bottom
- Click on Edit Topology Layout
- Devices in the topology view can be moved (drag-and-drop) to better represent the actual topology. Click Done Editing when the device positions match the lab topology.
Conclusion
What’s next
ThousandEyes Walkthrough Part 3 – Enterprise and Endpoint Agent Installs
This post will go over installing the ThousandEyes agents in the lab. To see all the posts in this series expand the box below.
- Part 1 – The What and the Why
- Part 2 – Lab build
- Part 3 – Enterprise and Endpoint Agent Installs <–You Are Here
- Part 4.1 – SNMP Monitoring
- Part 4.2 – Scenarios and Test Types
- Part 4.3.1 – Scenario 1 – Enterprise agent to agent test configuration
- Part 4.3.2 – Scenario 2 – Enterprise DNS test configuration
- Part 4.3.3 – Scenario 3 – Enterprise and Endpoint HTTP test configuration (Coming soon)
- Part 4.3.4 – Scenario 4 – Enterprise Page Load test configuration (Coming less soon)
- Part 4.3.5 – Scenario 5 – Enterprise Transaction test configuration and Endpoint Agent Browser Settings (Coming more less soon)
- Part 4.4+ – Details TBD
There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:
- Behind the Scenes – The Lab Build
- Ok, there’s only one so far, but I plan to add more where it makes sense.
- 4x Linux Enterprise Agent installs on the CML Ubuntu instances
- CS1-1, CS1-2, CS2-1, and CS2-2
- 2x Docker Enterprise Agent container deployments on the Ubuntu Docker host
- These two agents will be added to a cluster
- 1x Raspberry Pi Enterprise agent (optional)
- 1x Windows Endpoint Agent install on the Windows VM
Prerequisites
- More details on the ThousandEyes/Catalyst licensing can be found here:Â https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9300-series-switches/nb-06-thousand-campus-assurance-so-cte-en.html
- There’s detailed information on how to activate the Licenses here:Â https://www.cisco.com/c/dam/m/en_us/customer-experience/collateral/thousandeyes-activation-guide.pdf
If existing licenses are unavailable a 15-day trial license can be requested here:Â https://www.thousandeyes.com/signup/
Additional hardware and software
- Raspberry Pi 4 Model B with at least 4GB RAM (Very hard to find in stock right now
- Computer with SD card slot or adapter
- USB 3.0 SD card adapter –Â https://amzn.to/3NNerVg
- Blank 32GB SD card – These two are recommended by ThousandEyes
- Samsung EVO Plus 32GB SD card –Â https://amzn.to/3DHu1gu
- SanDisk Extreme 32GB SD card –Â https://amzn.to/3x85PCW
- SD card writing program installed –Â https://www.raspberrypi.com/software/
Installs
Account Group Token
- Open a web browser and navigate to https://www.thousandeyes.com/
- Log into your account
- Click the Hamburger iconÂ
in the top left
- Expand Cloud & Enterprise Agents
- Click Agent Settings
- Click the Add New Enterprise Agent button
- Click the eye button to show the token, or the copy button to store it on the clipboard
- Store the token in a safe, convenient location. It will be used to add agents to the ThousandEyes account throughout this process.
Linux Enterprise Agent install
- Open a web browser and navigate to https://www.thousandeyes.com/
- Log into your account
- Click the Hamburger icon
in the top left
- Expand Cloud & Enterprise Agents
- Click Agent Settings
- Click the Add New Enterprise Agent button
- Click the option for Linux Package
- Copy the commands displayed
- Perform the following steps for CS1-1. CS1-2, CS2-1, and CS2-2 in CML
- In CLM open the terminal session and log in
- Paste the commands into the terminal and press Enter
- It may take some time, but eventually there will be a prompt that say:
The default log path is /var/log. Do you want to change it [y/N]?
- Press Enter to accept the default log location
- It might take 10 minutes or it could be over an hour for the process to complete and the agent to come online. When it returns to the user prompt the service should be started.
- When the installs are complete they should be listed in the ThousandEyes portal under Enterprise Agents
Docker Enterprise Agent install
-
- Open a web browser and navigate to https://www.thousandeyes.com/
- Log into your account
- Click the Hamburger iconÂ
in the top left
- Expand Cloud & Enterprise Agents
- Click Agent Settings
- Click the Add New Enterprise Agent button
- Click the option for Docker
- Scroll down to the sections with the commands
- Copy the section to configure seccomp and apparmor profile
- Log in to the Ubuntu node that is the Docker host and paste in the commands:
- Add listening IPs for the Docker containers
-
sudo ip add add 192.168.1.51 dev ens33 sudo ip add add 192.168.1.52 dev ens33
-
- Pull the TE Docker image
-
docker pull thousandeyes/enterprise-agent > /dev/null 2>&1
-
- Update these commands by putting in your ThousandEyes token and changing the IPs if needed, then run them to create two ThousandEyes agents.
- Add listening IPs for the Docker containers
NOTE: These commands have been updated to include DNS and IP settings that aren’t available on the ThousandEyes Enterprise Agent page. If you use the commands from ThousandEyes the DNS and Published ports will need to be updated.
-
-
-
docker run --hostname='TE-Docker1' --memory=2g --memory-swap=2g --detach=true --tty=true --shm-size=512M -e TEAGENT_ACCOUNT_TOKEN=<--Your Token goes here--> -e TEAGENT_INET=4 -v '/etc/thousandeyes/TE-Docker1/te-agent':/var/lib/te-agent -v '/etc/thousandeyes/TE-Docker1/te-browserbot':/var/lib/te-browserbot -v '/etc/thousandeyes/TE-Docker1/log/':/var/log/agent --cap-add=NET_ADMIN --cap-add=SYS_ADMIN --name 'TE-Docker1' --restart=unless-stopped --security-opt apparmor=docker_sandbox --security-opt seccomp=/var/docker/configs/te-seccomp.json --dns=10.133.100.10 --dns-search=cml.lab --publish=192.168.1.51:49152:49152/udp --publish=192.168.1.51:49153:49153/udp --publish=192.168.1.51:49153:49153/tcp thousandeyes/enterprise-agent /sbin/my_init
-
docker run --hostname='TE-Docker2' --memory=2g --memory-swap=2g --detach=true --tty=true --shm-size=512M -e TEAGENT_ACCOUNT_TOKEN=<--Your Token goes here--> -e TEAGENT_INET=4 -v '/etc/thousandeyes/TE-Docker2/te-agent':/var/lib/te-agent -v '/etc/thousandeyes/TE-Docker2/te-browserbot':/var/lib/te-browserbot -v '/etc/thousandeyes/TE-Docker2/log/':/var/log/agent --cap-add=NET_ADMIN --cap-add=SYS_ADMIN --name 'TE-Docker2' --restart=unless-stopped --security-opt apparmor=docker_sandbox --security-opt seccomp=/var/docker/configs/te-seccomp.json --dns=10.133.100.10 --dns-search=cml.lab --publish=192.168.1.52:49152:49152/udp --publish=192.168.1.52:49153:49153/udp --publish=192.168.1.52:49153:49153/tcp thousandeyes/enterprise-agent /sbin/my_init
-
-
- When the installs are complete they should be listed in the ThousandEyes portal under Enterprise Agents
Docker Enterprise Agent configuration
- Open a web browser and navigate to https://www.thousandeyes.com/
- Log into your account
- Click the Hamburger iconÂ
in the top left
- Expand Cloud & Enterprise Agents
- Click Agent Settings
- Click on the Agent
- In the right panel click on Advanced Settings
- Updated the IP address with the address assigned to that instance
- Click the Save Changes button on the bottom right
- Repeat this process for the other container agent
- At the Enterprise Agents page select both Docker agents
- Click the Edit button
- Select Edit Cluster
- On the right select Add to a new cluster
- Click Save Changes
- The agent icon will be updated to include the cluster icon, and under the Cluster tab it will display the new cluster
Raspberry Pi Enterprise Agent install
I have an automated configuration process for the Raspberry Pi image: https://www.mytechgnome.com/2023/06/15/automated-thousandeyes-raspberry-pi-image-customization/
- Open a web browser and navigate to https://www.thousandeyes.com/
- Log into your account
- Click the Hamburger iconÂ
in the top left
- Expand Cloud & Enterprise Agents
- Click Agent Settings
- Click the Add New Enterprise Agent button
- The pane on the right should open the the Appliance tab, under Physical Appliance Installer find the Raspberry Pi 4, and to the right of that click Download – IMG
- Wait for the download to complete. It’s nearly a 1GB file, so it might take a few minutes.
- Connect the SD card to the computer that will be doing the imaging
- This process erases the entire card. Make sure you are using a blank card, or you have any valuable data on the card backed up elsewhere.
- Launch the Raspberry Pi Imager
- Under Operating System click Choose OS
- Scroll down to the bottom of the list and click Use custom
- Browse to the location of the downloaded image, select it, and click Open
- Under Storage click on Choose Storage (or Choose Stor…)
- Select the SD card in the window that pops up
- Click Write
- Continuing this process will erase all data on the SD card, if that’s acceptable click Yes
- A progress bar will be displayed, and after a few minutes the image copy should complete successfully. Click continue and close the Raspberry Pi Imager software
- Remove the SD card from the imaging PC and insert it in the Raspberry Pi.
- Boot the Raspberry Pi
- You’ll want a monitor connected to find the IP assigned, though this could also be done by looking at DHCP leases, scanning the network, or trying name resolution for the default hostname: tepi
- Make sure there’s a network cable plugged in and connected to the LAN (the ThousandEyes agent doesn’t support wireless connections)
- When the Pi finishes booting find the IP address displayed on the screen
- Use a web browser to connect to the IP of the Pi agent (using the name might work –Â https://tepi/)
- Likely the browser will display a security warning because the certificate is untrusted. Go through the steps required to accept the security risk and access the site.
- At the login page enter the default credentials: admin / welcome
- The first page will prompt to change the password. Enter the current password and create a new one, then click Change Password
- The next page prompts for the Account Group Token. Enter the token value that was collected earlier in this post and then click Continue
- The agent will go through a check-in process and provide diagnostic data. If everything looks good you can click Complete
- That completes the required agent set up. It will then bring you to the network configuration page. Scroll down to the DNS section, switch the Current DNS Resolver to Override and enter the IP 10.133.100.10 in the Primary DNS box
- The agent should now be listed in the ThousandEyes portal under Enterprise Agent
Windows Endpoint Agent install
- Start the Windows VM and log in
- Open a web browser and navigate to https://www.thousandeyes.com/
- Log into your account
- Click the Hamburger iconÂ
in the top left
- Expand the Endpoint Agents section
- Click on Agent Settings
- Either a splash screen with a Download button will appear, or there will be a button to Add New Endpoint Agent. Click the button that shows up – both bring up the same pane
- Leave the Endpoint Agent radio button selected and click the button Download – Windows MSI
- The Mac installation isn’t being covered here, but there’s instructions on how to install it here:Â https://docs.thousandeyes.com/product-documentation/global-vantage-points/endpoint-agents/installing
- There will be two options for the processor architecture, select the x64 Windows MSI
- When the download completes run the MSI
- The installation is a typical MSI package, so I’m not going to include screenshots for every step
- Click Next to start the install
- Read the EULA and if you agree to the terms check the box to accept and click Next
- Click on the TCP Network Tests Support and select “Will be installed on local hard drive”
- Do the same for at least one browser extension. Edge is the default browser on Windows 10, but if you want to install and use Chrome then get Chrome installed before continuing the Endpoint Agent installation. Click Next when you have the browser selected.
- Click Install
- If there us a UAC prompt for the install, click yes to continue
- Click Finish
- It might take a few minutes for the agent to check in, but eventually you should see the agent listed under Endpoint Agents in the portal
Conclusion
- Bare metal install (Intel NUC or other hardware)
- OVA (VMware ESX, Workstation, and Player, Microsoft Hyper-V Oracle VirtualBox)
- Application hosting on Cisco platforms (Catalyst 9300 and 9400, Nexus 9300 and 9500, Catalyst 8000, ISR, ASR)
- AWS CloudFormation Template
- Mac OS Endpoint Agents
- Pulse Endpoint Agents for external entities
What’s next?
ThousandEyes Walkthrough Behind the Scenes – The Lab Build
This post will go over the planning of the ThousandEyes lab used in this series. To see past posts in this series expand the box below.
- Part 1 – The What and the Why
- Part 2 – Lab build
- Part 3 – Enterprise and Endpoint Agent Installs
- Part 4.1 – SNMP Monitoring
- Part 4.2 – Scenarios and Test Types
- Part 4.3.1 – Scenario 1 – Enterprise agent to agent test configuration
- Part 4.3.2 – Scenario 2 – Enterprise DNS test configuration
- Part – 4.3.3 Scenario 3 – Enterprise and Endpoint HTTP test configuration (Coming soon)
- Part – 4.3.4 Scenario 4 – Enterprise Page Load test configuration (Coming less soon)
- Part – 4.3.5 Scenario 5 – Enterprise Transaction test configuration and Endpoint Agent Browser Settings (Coming more less soon)
- Part – 4.4+ – Details TBD
There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:
- Behind the Scenes – The Lab Build <–You Are Here
- Ok, there’s only one so far, but I plan to add more where it makes sense.
If you are following the series, this post is strictly informational. It won’t contain any steps that need to be performed in the lab. The goal is to provide insight into why I made the design choices I did with the lab.
The details on the lab build can be found here: https://www.mytechgnome.com/2022/04/thousandeyes-walkthrough-part-2-lab.html
CML
- There are plenty of similar tools (GNS3, EveNG, etc) that are available, so why did I pick the paid tool? The simple answer is licensing. My understanding is CML is the only way to run virtual Cisco instances without running afoul of the EULA. Yes, I could have used non-Cisco routers, but since Cisco is a major vendor it seemed reasonable to go with it.
- The Personal version of CML has two flavors, Personal which allows 20 active nodes, and Personal Plus which allows 40 active nodes. I built the lab using 20 nodes because the Personal Plus is an extra $150, and because the additional nodes would increase the resource requirements. I wanted the lab to be as accessible as possible. It could easily be extended to 40 nodes or higher, but 20 is enough to get basic testing done.
- Even though the TE agents could be deployed to VMs, I wanted to use CML as a way to easily simulate scenarios where an engineer would need to do some troubleshooting. Within CML links can be configured with bandwidth limits, latency, jitter, and loss. The theory is that ThousandEyes should be able to detect and even alert on those conditions.
- I am using version 2.2.3, even though version 2.3 is available. The simple reason is that Cisco is still recommending version 2.2.3. There are some known issues with 2.3, which is why I’m not running that.
IOSv Routers
- Even though CML can run CSR 1000V and IOS-XR instances I decided to go with IOSv instances. This was because of resource requirements. The CSR 1000v and IOS-XR instances each require 3GB RAM, and with 14 routers that would consume an additional 35GB RAM over what the IOSv routers use. For the purposes of the lab, the IOSv can do everything needed without the overhead.
Ubuntu
- I wanted to keep as much of the lab in CML as possible, and running Ubuntu in CML aligns with that goal. Of the Linux flavors that are available out of the box in CML, Ubuntu is the only one supported by ThousandEyes.
- With Ubuntu being used in the CML lab it seemed reasonable to use Ubuntu for the Docker host as well.
Topology
- I’ll admit I spent a lot of time working through different topology options. At one point I had switches and HSRP in the design, but I decided to back away from layer 2 technologies to focus on layer 3. The primary use case for ThousandEyes is looking at WAN links, and with the node limit in CML, it made sense to drop the L2 configurations to make room for more L3 devices.
- I wanted to maximize the number of BGP AS configurations while maintaining multiple links, which is why there are 7 BGP AS configurations. By simply shutting down specific links traffic could hit 6 of the 7 AS networks. With some BGP reconfiguration that could be extended.
- The two “Client” networks are intended to be what a network engineer would have in their environment. Likely they’d have a lot more, but with the node limits having two networks is enough to test with. Each of the client networks has two Ubuntu nodes that are running the TE Enterprise agent. One of the Ubuntu nodes is also running Apache. (more on Apache shortly)
- In the “Public” network I wanted to add another BGP path outside the redundant ISP paths, and I wanted a service that was accessible. With this being treated as public I opted to not run a TE agent there.
- Access outside of the CML environment is done via the “External” network. ThousandEyes is a SaaS service, which means the agents all need to be able to connect to the TE portal.
- Even though the entire network is built using RFC 1918 addresses, the design is effectively using public addresses throughout the entire lab. The “Client” addresses are propagated through the ISP and public networks, which isn’t typical in IPv4 deployments. This was mainly choosing simplicity and efficiency. If the client networks were masked then something like a VPN would be required to link the two client networks. Though that better aligns with the real world, for the functional purposes of the lab it makes no difference. Both ends need IP reachability and adding more NAT and VPN configuration work doesn’t provide a significant improvement in how the lab operates.
External Routing and NAT
- On the external router, NAT is configured, which should allow internet access from the lab with no additional configuration needed. The 192.168.1.0/24 network is excluded from translation with the intent that devices on the LAN (Docker, Windows, and Raspberry Pi agents) would be able to connect directly to devices in the CML lab.
- For the LAN devices to reach the CML lab routes need to be added either to the LAN router or as static routes to each of the devices. Using the LAN router requires the fewest changes, and is the most extensible.
- Unfortunately not every environment is identical. I suspect that there may be some issues with getting the routing working properly. I spent a lot of time trying to decide if this routing solution was better than just using DHCP on the external router and doing full outbound NAT. I decided that having the external agents able to have full connectivity to the internal agents was worth the added complexity.
Services
- The Apache instances were set up just to create a simple webserver to establish HTTP connections. For transaction tests, I will be using external websites.
- Bind is deployed primarily for easy name resolution of the lab devices, and to have another service running inside the lab. Since ThousandEyes can do DNS tests it made sense to include.
External Resources
- The Docker, Windows, and Raspberry Pi agents are primarily just to provide the ability to test with those platforms. The Docker and Pi agents are functionally similar to the Ubuntu agents running in the CML lab. The Windows agent is an Endpoint agent, which brings a different set of functionality.
- I do expect that there will be improvements in test performance with these agents versus the ones in CML because there are fewer layers of abstraction. I can’t imagine an Ubuntu agent running on a minimum spec VM inside KVM, that is running on the CML VM inside Workstation is going to be the most efficient. Add in the software layers for the routers connecting those agents, and that only adds more potential performance impact.
- As mentioned previously, internet access is required for ThousandEyes agents to reach the SaaS platform. With that requirement in mind, it made sense to just use external websites for most of the testing instead of building elaborate web servers inside the lab.
Misc. Notes
- Everyone has their preferred numbering scheme. For this lab, I tried to come up with something that I could easily build on in a programmatic sense. Yes, for the router links I could have used /30 or /31, but in a lab, I’m not worried about address consumption. I built addresses based on the nodes being connected.
- I’m sure someone somewhere will be upset that I don’t have passwords on the routers. It’s a lab that I tear down frequently, and it’s inside a trusted network. The risk of an attack is minimal, and worth it to not need to log in to each device.
- The Ubuntu server version was the latest at the time of writing, and I went with Windows 10 to avoid some of the issues with getting Windows 11 deployed.
- With the complexity of the build in CML, I decided it was easiest to just publish the YAML code. Initially, I had intended to write up exactly how to build the lab, and provide configs for each device, but as I built it out it became clear that doing so would be quite cumbersome. Using the YAML file should give more consistent deployments, with less manual work to get the lab running.
- I’ve had several requests to incorporate AWS into this lab. Currently, that’s outside the scope of the roadmap I have for this series. The primary reason for that is because of the cost associated with AWS. Once I get through the posts I have planned for this series I plan to investigate if I can leverage the AWS free tier to get useful data.
- Despite most of the routers being in provider networks, each router has SNMP running. The reason I did this was to show how ThousandEyes can use SNMP to add additional context to data, and in some cases, it can be used to trigger alarms. In a real-world scenario you likely can’t get SNMP from provider networks, but you also likely have more than two network devices at a location. The decrease in realism is more than made up for by not having to build out a complete LAN environment.
ThousandEyes Walkthrough Part 2 – Lab build
This post will go over getting a ThousandEyes lab built out. To see all the posts in this series expand the box below.
- Part 1 – The What and the Why
- Part 2 – Lab build <–You Are Here
- Part 3 – Enterprise and Endpoint Agent Installs
- Part 4.1 – SNMP Monitoring
- Part 4.2 Scenarios and Test Types
- Part 4.3.1 Scenario 1 – Enterprise agent to agent test configuration
- Part 4.3.2 Scenario 2 – Enterprise DNS test configuration
- Part 4.3.3 Scenario 3 – Enterprise and Endpoint HTTP test configuration (Coming soon)
- Part 4.3.4 Scenario 4 – Enterprise Page Load test configuration (Coming less soon)
- Part 4.3.5 Scenario 5 – Enterprise Transaction test configuration and Endpoint Agent Browser Settings (Coming more less soon)
- Part 4.4+ – Details TBD
There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:
- Behind the Scenes – The Lab Build
- Ok, there’s only one so far, but I plan to add more where it makes sense.
Lab Build
Topology
I’ve built out this lab using VMware Workstation and Cisco Modeling Labs to simulate a network for ThousandEyes to monitor.
The lab is broken down into four types of sites, Client, ISP, Public, and External. There are two Client sites, each consisting of two routers and two Ubuntu instances. The routers are running BGP, and have SNMP enabled. The first Ubuntu instance is only running a ThousandEyes agent, and the other instance is running a ThousandEyes agent and an Apache webserver.
The ISP networks are routers running BGP interconnecting all the other sites. I have SNMP enabled on them just to show what ThousandEyes can do with SNMP monitoring. Normally isn’t going to be accessible on ISP devices.
The Public zone is also running BGP and has an Ubuntu instance that is running DNS for the entire CML.LAB network.
The External site is used to bridge the lab environment to the network outside CML. It has a static route out to the LAN gateway that is redistributed into BGP, and a static IP assigned on the LAN. For traffic leaving the LAN, it has NAT configured. This should reduce the configuration needs on the LAN side. A static route can be added to the LAN gateway to send traffic to the External router, or static routes can be added to the individual devices that will connect to the lab network.
In addition to the CML lab, three additional devices will be deployed, an Ubuntu Server running Docker for ThousandEyes Enterprise Agents, a Windows 10 VM running the ThousandEyes Endpoint Agent, and a Raspberry Pi running the ThousandEyes Enterprise Agent.
This table breaks down the resources assigned to each node and the total amount of resources. The CML VM will need to have enough assigned to it to allow the nodes inside it to run.
The Windows and Ubuntu Docker nodes will sit outside CML, as VMs in VMware Workstation. There will also be a Raspberry Pi added to the environment.
Installation Prerequisites
- Management workstation with Python 3 installed and Paramiko installed (pip install paramiko)
- CML (I am using 2.2.3) must be installed, configured, and have a bridged network adapter. https://learningnetworkstore.cisco.com/cisco-modeling-labs-personal/cisco-modeling-labs-personal/CML-PERSONAL.html
- An ISO image for Ubuntu Server (I am using 20.04.4) https://ubuntu.com/download/server
- An ISO image for Windows 10 (I am using 21H2) https://www.microsoft.com/en-us/software-download/windows10
- VMware Workstation Pro (I am using v16) https://www.vmware.com/products/workstation-pro/workstation-pro-evaluation.html
- (optional) Raspberry Pi 4 Model B with at least 4GB RAM and a blank 32GB SD card https://www.raspberrypi.com/products/raspberry-pi-4-model-b/ – The steps for the Pi deployment will be covered in the next post.
- (optional) Configure a static route on the LAN for the 10.0.0.0/8 network that points to the IP of the edge router (the default IP is 192.168.1.42). This can also be accomplished by adding static routes on the devices that will need to communicate into the lab.
Installation Process
The easiest way to get the lab up and running is to import a YAML file. This file contains everything you need to get started, but some updates may be required. The lab is configured for internet access, and there is a static IP and gateway assigned. The LAN addressing might need to be updated to match your environment.
If you choose not to use the YAML import you can find the relevant node configurations in the YAML and then create and configure the nodes accordingly.
Expand each of the following sections for steps on how to build out the lab.
The YAML file can be downloaded from GitHub here: https://github.com/mytechgnome/CML-TE-Lab/blob/main/TE.yaml
- Copy the above YAML data into a new file
- Save the file as TE-Lab.yaml
- Log in to CML
- From the Dashboard Click Import
- Click in the File(s) to import area
- Browse to the location the YAML file was saved and select it
- Click Import
- It should import the lab successfully. Click the Go To Lab button
The entire simulation can be started at once, or the individual nodes can be started. If they are being started manually start with the external connection, and work through all the routers first. Then move on to PS3-1. This node will take a few minutes to complete. The remaining CS nodes can be started after PS3-1 completes its startup.
Credentials
Verification tasks
Routers
- show ip route
- The route table should be populated, including a default route
- ping 8.8.8.8
- Should receive replies
- If this fails verify the configuration of Gi0/0 matches the LAN requirements, and the CML VM NIC is configured for bridged access
PS3-1
- systemctl status bind9
- Should display active. If this fails verify internet connectivity and then run these commands:
- sudo apt-get update
- sudo apt install install -o Dpkg::Options::=”–force-confold” bind9 -y
- ping cisco.com
- Should receive replies
CS1-2 and CS2-2
- systemctl status apache2
- Should display active. If this fails verify internet connectivity and then run these commands:
- sudo apt-get update
- sudo apt install apache2
- ping cisco.com
- Should receive replies
- Open VMware Workstation and create a new VM by pressing Ctrl + N
- When the New Virtual Machine Wizard opens click Next
- Select the option for Installer disk image file (iso)
- Enter a username and password, then click Next
- Enter a name for the VM, and verify the path, then click Next
- Set the virtual hard drive to 30GB, then click Next
- Click Customize Hardware
- Select the Network Adapter, change the Network Connection to Bridged, then click Close
- Click Finish
- Wait for the OS installation process to start
- Select your language and press Enter twice to select and confirm
- Use the arrow keys to select the NIC and press Enter
- Highlight Edit IPv4 and press Enter
- Press Enter to change the address assignment method and select Manual
- Use the arrow keys to move between fields filling out the IP address info, and then go to Save and press Enter when complete
- Highlight Done, and press Enter
- Press Enter again to skip the Proxy config
- Press Enter again to use the default mirror location
- Use the arrow keys to highlight Done and press Enter to accept the default storage config
- Press Enter again to accept the file system config
- Highlight Continue and press Enter to confirm the storage settings
- Use the arrow keys to move between fields, fill out the Profile info, and then go to Done and press Enter when complete
- Press Enter again to skip Ubuntu Advantage
- Press Enter to enable SSH access, then highlight Done and press Enter
- Use the arrow keys to go down to highlight Done, and press Enter
- Wait for the installation to complete
- When the installation finishes highlight Reboot Now and press Enter
- When the server is back up log in
- Run the following commands to install Docker
sudo apt-get update
sudo apt-get install
ca-certificates
curl
gnupg
lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io -y
Create the Windows VM
- In VMware Workstation press CTRL+N to open the New Virtual Machine Wizard, and make sure Typical is selected, then click Next
- Select the option for Installer Disc Image File, and browse to the location you downloaded the Windows 10 ISO to then click Next
- Enter the name for the client and select the location
- Use the default hard drive size of 60GB (another drive will be added later for the iSCSI target storage), and click Next
- Click Customize Hardware
- Adjust the CPU and RAM as needed for your environment (2 vCPUs 4-8GB RAM would be recommended), and change the Network Adapter from NAT to Bridged
- Click close, verify the box is checked for “Power on this virtual machine after creation” and click finish.
Deploy the Windows OS
NOTE: While in the VM you will need to press Ctrl+Alt to release the cursor to get to your desktop
- >While the VM is booting you might see a prompt to press a key to boot from CD. If that happens click into the window and press a key.
- Select the language, and keyboard settings
- Click Install Now
- On the Activate Windows screen click “I don’t have a product key”
- Select Windows 10 Pro and click Next
- Read through all of the licenses terms, and if you accept the terms check the box to accept them and click Next
- Select the Custom install option
- By default, it should already select Drive 0, which is the 60GB drive initially created. Click next. The OS install will start, so just let that process run.
OS Initial Config
Windows 10 has several steps to go through to get the OS configured before actually loading to a desktop.
- Select your regions and Click Yes
- Select your keyboard layout
- Skip adding the additional keyboard
- Wait a moment for it to progress to the account creation screen, then select “Set up for personal use” and click Next
- Microsoft is going to try to link to an online account, but since this is for a temporary lab PC click on “Offline account” in the bottom left.
- Microsoft really tries to push the online account, so again look in the bottom left corner and select “Limited experience”
- Enter a username and click Next
- Create a password and click Next
- When prompted for the three security questions I just select the first three options and enter random characters. This is a lab, and if I happen to forget the password I can easily recreate the VM. Click Next
- For the privacy settings, this really doesn’t matter, as it’s a lab machine that won’t exist for long. Everything can be left enabled by default, or it can be disabled. After applying the settings click Accept.
- On the Customize Experience page just click Skip
- Cortana… Microsoft really wants people to enable all their stuff. Click “Not now” to move on.
- Success! The post-install prompts are done. Now, wait for the configuration to complete.
Client OS config
To configure the OS there are only two tasks that are going to be performed.
- Install VMware Tools
- Configure DNS
Install VMware Tools
- Log into the VM using the password set previously
- Right-click on the VM in the Library and select Install VMware Tools
- Autorun should prompt to run, but if not then navigate to the D: drive and double click it. That should kick off the Autorun for the installer.
- Follow the defaults for the installation. Next > Next > Install > Finish and then click Yes when prompted for a reboot.
Configure DNS
- Open Powershell as admin
- Press the Windows key and type powershell
- Press Crtl+Shift+Enter to run as admin
- Run these commands:
Set-DnsClientServerAddress -InterfaceAlias Ethernet0 -ServerAddresses 10.133.100.10
Set-DnsClientGlobalSetting -SuffixSearchList cml.lab
There’s a lot to the lab build, but hopefully, it went smoothly. If there were any issues you can add a comment to this post, or reach me on Twitter @Ipswitch.
As the lab build-out continues I may need to come back and edit the configuration here.
CHANGELOG
- CML Lab YAML data
- Corrected IR2-2 Gi0/0 IP configuration and BGP peering
- Corrected IP assignment on PR3-2 – config was moved from Gi0/3 to Gi0/4
- Added loopback interfaces to all routers (will be used for SNMP connections)
- Updated DNS records to use loopback addresses
- Lab config
- Added lab prerequisite to have a management workstation with Python and paramiko
- Added link to Github Repo with YAML file
- Created python script to update routers with v1.1 changes
- https://raw.githubusercontent.com/mytechgnome/CML-TE-Lab/main/v1.1-update.py
- Run the script from a machine with Python and Paramiko installed, and network connectivity to the lab networks.
What’s Next?
The next entry in this series will cover getting the ThousandEyes agent deployed into the lab, and getting things ready to start building tests and collecting data.