Backgroound Image

Tag: Raspberry Pi

Automated ThousandEyes Raspberry Pi image customization

Posted on by mytechgnome

If you haven’t caught this yet, I am a huge fan of ThousandEyes! I was working on a project where we were planning to deploy ThousandEyes agents to hundreds of sites, and to keep costs down we were going to use the Raspberry Pi 4 Model B. Having to image hundreds of SD cards was going to be a nightmare, so I wanted to automate the ThousandEyes Raspberry Pi agent image customization process to simplify the deployment.

Full disclosure: I started this project just as the supply chain issues were hitting the Pi market, and I left that job a few months later. I was never able to run this at scale, but I was able to use this for the few units I was able to get my hands on.

I also have a series on getting ThousandEyes deployed in a lab environment if you want to read more on getting ThousandEyes running: https://www.mytechgnome.com/2022/03/28/thousandeyes-walkthrough-part-1-what/

Hardware/Software required

Configure Pi for imaging

I used a Raspberry Pi as the system for building the images. I found it makes the process much easier, but this isn’t a hard requirement. The main thing is you will need to be able to mount a Linux file system, and it seemed to work much better using a native Ubuntu instance instead of trying to map an SD card to a VM or WSL instance.

  1. Using your preferred tool, flash the Ubuntu image onto one of the microSD cards – these are both good options:
  2. Boot a Pi using the newly flashed Ubuntu image and complete the setup

Automated and Manual Processes

I have an automated process available at my GitHub page: https://github.com/mytechgnome/TE-Imaging
The automation uses a JSON file with the device-specific information. You could automate the process of filling out the hostname and IP info if you have that info in an IPAM or similar platform.

How to use the vars.json file:

  • Token Value – Group token from ThousandEyes portal
  • Image name – Shouldn’t need to do anything with this, unless ThousandEyes changes the image file name
  • SSH key – Leave blank if you aren’t adding a key, otherwise paste in the contents of the public key file
  • Devices – Duplicate the device objects as needed
    • Hostname – The name of the specific agent
    • IP – Assign static IP or for DHCP, leave it blank- “”
    • Subnet mask and gateway – self explanatory
    • DNS – Also self explanatory. If you’re only using one DNS server, leave DNS2 blank- “”
vars.json

Below is the manual process. The overall process is the same as the automated script, but with less automated goodness. If you want to understand how the automated process works, you can read through the manual process to get a better understanding.

Mount the ThousandEyes image

Before we can customize the image we need to mount it. Run the following steps from a terminal on the imaging Pi created previously.

  1. Download the ThousandEyes image
    wget https://app.thousandeyes.com/install/downloads/appliance/thousandeyes-appliance.rpi4.img.xz
  2. Decompress the image
    unxz -k thousandeyes-appliance.rpi4.img.xz
  3. Create a mount directory
    mkdir /tmp/temount
  4. Mount the image
    sudo mount -o loop,offset=269484032 thousandeyes-appliance.rpi4.img /tmp/temount
    • Note: Because there are two partitions, we need to mount the second partition
      • If this doesn’t work it could be caused by a change in the partition layout, use these steps to get the correct offset
        fdisk -l thousandeyes-appliance.rpi4.img
      • Here’s an example output:
        Disk thousandeyes-appliance.rpi4.img: 4.07 GiB, 4367262720 bytes, 8529810 sectors
        Units: sectors of 1 * 512 = 512 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes
        Disklabel type: dos
        Disk identifier: 0x354db354

        Device Boot Start End Sectors Size Id Type
        thousandeyes-appliance.rpi4.img1 * 2048 526335 524288 256M c W95 FAT32 (LBA)
        thousandeyes-appliance.rpi4.img2 526336 8529809 8003474 3.8G 83 Linux
      • Multiply the sector size (512) by the start block (526336) and use that value in the offset.
  5. Verify the image mounted
    ls /tmp/temount/
    • There should be several folders listed. If not, check the offset mentioned in step 4.

Customize the image

There are three files that need to be modified to get the image ready.

  1. Apply the ThousandEyes Account Token
    sudo sed -i 's/<account-token>/$TOKEN/g' /tmp/temount/etc/te-agent.cfg
    • Note: replace $TOKEN with your token value.
    • Follow these steps to get your account token:
      • Open a web browser and navigate to https://www.thousandeyes.com/
      • Log into your account
      • Click the Hamburger icon in the top left
      • Expand Cloud & Enterprise Agents
      • Click Agent Settings
      • Click the Add New Enterprise Agent button
      • Click the eye button to show the token, or the copy button to store it on the clipboard
  2. Set the hostname
    sudo sed -i 's/tepi/$HOSTNAME/g' /tmp/temount/etc/hostname
    • Replace $HOSTNAME with your desired hostname
  3. Configure a static IP (Optional – by default the appliance will use DHCP)
    sudo sed -i 's/dhcp/static/g' /tmp/temount/etc/network/interfaces
    sudo echo address $IP >> /tmp/temount/etc/network/interfaces
    sudo echo netmask $MASK >> /tmp/temount/etc/network/interfaces
    sudo echo broadcast $BROADCAST >> /tmp/temount/etc/network/interfaces
    sudo echo gateway $GW >> /tmp/temount/etc/network/interfaces
    sudo sed -i 's/#DNS=/DNS= $DNS/g' /tmp/temount/etc/systemd/resolved.conf
    • These values will need to be updated accordingly: $IP $MASK $BROADCAST $GW $DNS
    • A second DNS server can be added by simply adding both addresses with a space between them
  4. Add an SSH key (Optional)
    sudo echo $SSH >> /tmp/temount/etc/ssh/keys/thousandeyes/authorized_keys
    • You can generate an SSH key with the following command: ssh-keygen -b 2048 -t rsa
    • Copy the contents of the key file and put it in place of $SSH
  5. Unmount the image
    sudo umount /tmp/temount

Flash the image to a microSD card

This is the easiest part, but also the most time consuming. After plugging in the SD card, find the device path for it by running this:

sudo fdisk -l | grep sd

If the SD card is already has partitions on it the output will show each partition, so if you see /dev/sda1 and /dev/sda2 that’s showing the partitions. The image file contains its own partitions, so it needs to be written directly to the card, not into an existing partition. To do that just ignore the numbers shown, so the destination would be /dev/sda.


To start writing the SD card run this command with the correct destination location.

dd if=thousandeyes-appliance.rpi4.img of=/dev/sda status=progress

Make sure the correct destination is selected, otherwise you might overwrite something you don’t want to lose. It usually takes about 10 minutes to write the SD card. When it’s finished, try booting a Pi using the SD card and it should come up with the correct hostname, IP, and account token.

All set!

After booting the Pi you’ll want to log in and change the local admin username and password

Default username: admin
Default password: welcome

After that, you can double check the setup wizard to make sure everything is good to go. The ThousandEyes agent will reach out to their registration servers and it will use the account token to get assigned to your account. The agent should be online in the portal within a few minutes.

BONUS Fun!

I’m not going to go through all the possible code examples of what you can do with this, but I thought I’d throw out a few things to think about.

  • Make API calls into an IPAM to get the site name and IP info, then populate the vars.json file automatically with that data.
  • Flip the script – this process builds the site-specific data into the image. Another option would be to use a generic image (still embedding the token and SSH key) and use DHCP addressing. Then, query the ThousandEyes API to find new agents named “tepi” and then remotely update the IP and hostname that way
  • A unique SSH key can be added to each image by moving the SSH key value under the devices in the vars.json and moving the SSH section of the script under the per device section, along with changing the variable to use the correct location from vars.json.
  • Similarly, you could also create new, unique SSH keys by including the keygen process in the script. Just make sure you keep a copy of the keys somewhere safe!

That’s it! Overall, the process is pretty easy, and far easier than finding Raspberry Pi 4s in stock anywhere… If you use this process to automate the ThousandEyes Raspberry Pi agent image customization, I’d love to hear how it worked for you and what, if any, changes you made to the process. You can add a comment here,

ThousandEyes Walkthrough Part 3 – Enterprise and Endpoint Agent Installs

Posted on by Dan Kelcher

This post will go over installing the ThousandEyes agents in the lab. To see all the posts in this series expand the box below.

ThousandEyes Walkthrough Table of Contents

There are some behind-the-scenes posts that go into more detail on how and why I took the approach that I did. Those can be found here:

There are going to be a number of agent deployments in the lab that was covered in the previous post:

  • 4x Linux Enterprise Agent installs on the CML Ubuntu instances
    • CS1-1, CS1-2, CS2-1, and CS2-2
  • 2x Docker Enterprise Agent container deployments on the Ubuntu Docker host
    • These two agents will be added to a cluster
  • 1x Raspberry Pi Enterprise agent (optional)
  • 1x Windows Endpoint Agent install on the Windows VM

Prerequisites

The lab needs to be built out.  Details on that process can be found here: https://www.mytechgnome.com/2022/03/thousandeyes-walkthrough-part-2-lab.html
Before we can start with the agent installs some ThousandEyes licenses are required.  It’s possible you already have some ThousandEyes licenses.  Cisco has bundled Enterprise Agents with the purchase of DNA Advantage or Premier licensing on the Catalyst 9300 and 9400 switches.

If existing licenses are unavailable a 15-day trial license can be requested here: https://www.thousandeyes.com/signup/

Additional hardware and software

As a side note – if you plan to work a lot with the Raspberry Pi I strongly recommend getting the USB 3 adapter.  It has a significant improvement in performance over the USB 2 adapters that are typically bundled with Raspberry Pi kits.  The SD cards recommended by ThousandEyes are because of the card performance.  Other cards can be used, but there may be a negative impact on performance.

Installs

Account Group Token

Before getting started with the installs it is important to get your Account Group Token.  This is an ID that is used to associate the agents to the correct account.  When deploying agents it will often require the token to be specified.
There’s multiple ways to find the token, but I think the easiest is to just pull it from the Enterprise Agent deployment panel
  1. Open a web browser and navigate to https://www.thousandeyes.com/
  2. Log into your account
  3. Click the Hamburger icon in the top left
  4. Expand Cloud & Enterprise Agents
  5. Click Agent Settings
  6. Click the Add New Enterprise Agent button
  7. Click the eye button to show the token, or the copy button to store it on the clipboard
    2. In a production environment you would want to keep this token safe.  It provides devices access to your ThousandEyes account, so it should not be made public
  8. Store the token in a safe, convenient location.  It will be used to add agents to the ThousandEyes account throughout this process.

Linux Enterprise Agent install

  1. Open a web browser and navigate to https://www.thousandeyes.com/
  2. Log into your account
  3. Click the Hamburger icon in the top left
  4. Expand Cloud & Enterprise Agents
  5. Click Agent Settings
  6. Click the Add New Enterprise Agent button
  7. Click the option for Linux Package
  8. Copy the commands displayed 
    2. curl -Os https://downloads.thousandeyes.com/agent/install_thousandeyes.sh
chmod +x install_thousandeyes.sh
sudo ./install_thousandeyes.sh -b <--Your Token goes here-->
  9. Perform the following steps for CS1-1. CS1-2, CS2-1, and CS2-2 in CML
    1. In CLM open the terminal session and log in
    2. Paste the commands into the terminal and press Enter
    3. It may take some time, but eventually there will be a prompt that say:

      The default log path is /var/log. Do you want to change it [y/N]?

    4. Press Enter to accept the default log location
    5. It might take 10 minutes or it could be over an hour for the process to complete and the agent to come online.  When it returns to the user prompt the service should be started.
  10. When the installs are complete they should be listed in the ThousandEyes portal under Enterprise Agents
    2. If the agent status is yellow it likely means an agent update is required, and it should automatically update within a few minutes

Docker Enterprise Agent install

    1. Open a web browser and navigate to https://www.thousandeyes.com/
    2. Log into your account
    3. Click the Hamburger icon in the top left
    4. Expand Cloud & Enterprise Agents
    5. Click Agent Settings
    6. Click the Add New Enterprise Agent button
    7. Click the option for Docker
    8. Scroll down to the sections with the commands
    9. Copy the section to configure seccomp and apparmor profile 
      2. curl -Os https://downloads.thousandeyes.com/bbot/configure_docker.sh
chmod +x configure_docker.sh
sudo ./configure_docker.sh
    10. Log in to the Ubuntu node that is the Docker host and paste in the commands:
      1. Add listening IPs for the Docker containers 
        1. sudo ip add add 192.168.1.51 dev ens33
sudo ip add add 192.168.1.52 dev ens33
      2. Pull the TE Docker image 
        1. docker pull thousandeyes/enterprise-agent > /dev/null 2>&1
      3. Update these commands by putting in your ThousandEyes token and changing the IPs if needed, then run them to create two ThousandEyes agents.

NOTE: These commands have been updated to include DNS and IP settings that aren’t available on the ThousandEyes Enterprise Agent page. If you use the commands from ThousandEyes the DNS and Published ports will need to be updated.

      1. docker run
  --hostname='TE-Docker1' 
  --memory=2g 
  --memory-swap=2g 
  --detach=true 
  --tty=true 
  --shm-size=512M 
  -e TEAGENT_ACCOUNT_TOKEN=<--Your Token goes here--> 
  -e TEAGENT_INET=4 
  -v '/etc/thousandeyes/TE-Docker1/te-agent':/var/lib/te-agent 
  -v '/etc/thousandeyes/TE-Docker1/te-browserbot':/var/lib/te-browserbot 
  -v '/etc/thousandeyes/TE-Docker1/log/':/var/log/agent 
  --cap-add=NET_ADMIN 
  --cap-add=SYS_ADMIN 
  --name 'TE-Docker1' 
  --restart=unless-stopped 
  --security-opt apparmor=docker_sandbox 
  --security-opt seccomp=/var/docker/configs/te-seccomp.json 
  --dns=10.133.100.10 
  --dns-search=cml.lab 
  --publish=192.168.1.51:49152:49152/udp 
  --publish=192.168.1.51:49153:49153/udp 
  --publish=192.168.1.51:49153:49153/tcp 
  thousandeyes/enterprise-agent /sbin/my_init
      2. docker run
  --hostname='TE-Docker2' 
  --memory=2g 
  --memory-swap=2g 
  --detach=true 
  --tty=true 
  --shm-size=512M 
  -e TEAGENT_ACCOUNT_TOKEN=<--Your Token goes here--> 
  -e TEAGENT_INET=4 
  -v '/etc/thousandeyes/TE-Docker2/te-agent':/var/lib/te-agent 
  -v '/etc/thousandeyes/TE-Docker2/te-browserbot':/var/lib/te-browserbot 
  -v '/etc/thousandeyes/TE-Docker2/log/':/var/log/agent 
  --cap-add=NET_ADMIN 
  --cap-add=SYS_ADMIN 
  --name 'TE-Docker2' 
  --restart=unless-stopped 
  --security-opt apparmor=docker_sandbox 
  --security-opt seccomp=/var/docker/configs/te-seccomp.json 
  --dns=10.133.100.10 
  --dns-search=cml.lab 
  --publish=192.168.1.52:49152:49152/udp 
  --publish=192.168.1.52:49153:49153/udp 
  --publish=192.168.1.52:49153:49153/tcp 
  thousandeyes/enterprise-agent /sbin/my_init
  2. When the installs are complete they should be listed in the ThousandEyes portal under Enterprise Agents
    2. If the agent status is yellow it likely means an agent update is required, and it should automatically update within a few minutes

Docker Enterprise Agent configuration

There are two configuration tasks that will be performed on the Docker agents.  The IP setting in ThousandEyes will be updated to use the host IPs that are tied to the Docker agents instead of the private Docker IPs, and the two agents will be added to a ThousandEyes Cluster.
  1. Open a web browser and navigate to https://www.thousandeyes.com/
  2. Log into your account
  3. Click the Hamburger icon in the top left
  4. Expand Cloud & Enterprise Agents
  5. Click Agent Settings
  6. Click on the Agent
  7. In the right panel click on Advanced Settings
  8. Updated the IP address with the address assigned to that instance
  9. Click the Save Changes button on the bottom right
  10. Repeat this process for the other container agent
  11. At the Enterprise Agents page select both Docker agents
  12. Click the Edit button
  13. Select Edit Cluster
  14. On the right select Add to a new cluster
    2. In the name field type Docker
  15. Click Save Changes
    2. It will give a confirmation screen, click Save Changes again
  16. The agent icon will be updated to include the cluster icon, and under the Cluster tab it will display the new cluster
Wondering why those changes were made?
The first change to the IP address was because ThousandEyes learns the IP address of the agent from its local configuration.  Docker, by default, creates a bridged network that uses NAT to communicate with the rest of the network.  That means the addresses Docker assigns to containers aren’t accessible on the network.  The additional IPs were added to the Ubuntu host to allow static NAT entries to be created in Docker (the Publish lines), which redirect traffic to sent to those IPs to the correct agent.  Since there are two containers using the same ports, we need two IP addresses to uniquely address each instance.  The change that was made to the agent settings in ThousandEyes forces other agents to use the routed 192.168.1.0/24 LAN network instead of the unrouted 172.17.0.0/16 Docker network.  This is only needed because we are going to build inbound tests into those agents.  If this was only outbound then it wouldn’t matter.
As for the creation of the cluster, this was done for high availability.  Granted, in this scenario both instances are running on the same Docker host which defeats the purpose.  However, it still illustrates how to configure the cluster.  The purpose of the cluster is exactly what would be expected.  Both agents share a name, and are treated as a single agent.  If a test is assigned to a cluster then either instance could run it.  In addition to high availability, this also can provide some load balancing between the agents, and it can simplify test creation.  Instead of managing tests to multiple instances in one location we can use the cluster agent to distribute those tests.

Raspberry Pi Enterprise Agent install

I have an automated configuration process for the Raspberry Pi image: https://www.mytechgnome.com/2023/06/15/automated-thousandeyes-raspberry-pi-image-customization/

  1. Open a web browser and navigate to https://www.thousandeyes.com/
  2. Log into your account
  3. Click the Hamburger icon in the top left
  4. Expand Cloud & Enterprise Agents
  5. Click Agent Settings
  6. Click the Add New Enterprise Agent button
  7. The pane on the right should open the the Appliance tab, under Physical Appliance Installer find the Raspberry Pi 4, and to the right of that click Download – IMG
  8. Wait for the download to complete.  It’s nearly a 1GB file, so it might take a few minutes.
  9. Connect the SD card to the computer that will be doing the imaging
    1. This process erases the entire card.  Make sure you are using a blank card, or you have any valuable data on the card backed up elsewhere.
  10. Launch the Raspberry Pi Imager
  11. Under Operating System click Choose OS
  12. Scroll down to the bottom of the list and click Use custom
  13. Browse to the location of the downloaded image, select it, and click Open
  14. Under Storage click on Choose Storage (or Choose Stor…)
  15. Select the SD card in the window that pops up
    2. If the SD card does not show up try reseating the card
  16. Click Write
  17. Continuing this process will erase all data on the SD card, if that’s acceptable click Yes
  18. A progress bar will be displayed, and after a few minutes the image copy should complete successfully.  Click continue and close the Raspberry Pi Imager software
  19. Remove the SD card from the imaging PC and insert it in the Raspberry Pi.
  20. Boot the Raspberry Pi
    1. You’ll want a monitor connected to find the IP assigned, though this could also be done by looking at DHCP leases, scanning the network, or trying name resolution for the default hostname: tepi
    2. Make sure there’s a network cable plugged in and connected to the LAN (the ThousandEyes agent doesn’t support wireless connections)
  21. When the Pi finishes booting find the IP address displayed on the screen
  22. Use a web browser to connect to the IP of the Pi agent (using the name might work – https://tepi/)
  23. Likely the browser will display a security warning because the certificate is untrusted.  Go through the steps required to accept the security risk and access the site.
  24. At the login page enter the default credentials: admin / welcome
    2. After logging in there may be an error message that briefly appears in the lower right stating the Account Group Token needs to be set.  This will be resolved shortly, and the error can be ignored for now.
  25. The first page will prompt to change the password.  Enter the current password and create a new one, then click Change Password
    2. After the password change is saved click the Continue button at the bottom of the page
  26. The next page prompts for the Account Group Token.  Enter the token value that was collected earlier in this post and then click Continue
    2. Even though there is a button to enable Browserbot here, the Raspberry Pi agent does not support it.  Leave that field set to No.  You can decide if you want to leave the crash reports enabled.
  27. The agent will go through a check-in process and provide diagnostic data.  If everything looks good you can click Complete
  28. That completes the required agent set up.  It will then bring you to the network configuration page.  Scroll down to the DNS section, switch the Current DNS Resolver to Override and enter the IP 10.133.100.10 in the Primary DNS box
    2. For the purposes of this lab none of the other settings need to be changed.  A static IP can be configured and/or the hostname could be changed if desired
  29. The agent should now be listed in the ThousandEyes portal under Enterprise Agent
    2. If the agent status is yellow it likely means an agent update is required, and it should automatically update within a few minutes
That completes the Enterprise Agent installations for the lab.

Windows Endpoint Agent install

  1. Start the Windows VM and log in
  2. Open a web browser and navigate to https://www.thousandeyes.com/
  3. Log into your account
  4. Click the Hamburger icon in the top left
  5. Expand the Endpoint Agents section
  6. Click on Agent Settings
  7. Either a splash screen with a Download button will appear, or there will be a button to Add New Endpoint Agent.  Click the button that shows up – both bring up the same pane
    1. Splash screen – 
    2. Add Endpoint Agent Button
  8. Leave the Endpoint Agent radio button selected and click the button Download – Windows MSI
    2. The Mac installation isn’t being covered here, but there’s instructions on how to install it here: https://docs.thousandeyes.com/product-documentation/global-vantage-points/endpoint-agents/installing
  9. There will be two options for the processor architecture, select the x64 Windows MSI
  10. When the download completes run the MSI
  11. The installation is a typical MSI package, so I’m not going to include screenshots for every step
    1. Click Next to start the install
    2. Read the EULA and if you agree to the terms check the box to accept and click Next
    3. Click on the TCP Network Tests Support and select “Will be installed on local hard drive”
    4. Do the same for at least one browser extension.  Edge is the default browser on Windows 10, but if you want to install and use Chrome then get Chrome installed before continuing the Endpoint Agent installation.  Click Next when you have the browser selected.
    5. Click Install
    6. If there us a UAC prompt for the install, click yes to continue
    7. Click Finish
  12. It might take a few minutes for the agent to check in, but eventually you should see the agent listed under Endpoint Agents in the portal

Conclusion

This was the first post actually working with ThousandEyes, and hopefully it illustrates how powerful this tool is.  As part of the lab there are four different types of agents installed, but there’s many more available:
  • Bare metal install (Intel NUC or other hardware)
  • OVA (VMware ESX, Workstation, and Player, Microsoft Hyper-V Oracle VirtualBox)
  • Application hosting on Cisco platforms (Catalyst 9300 and 9400, Nexus 9300 and 9500, Catalyst 8000, ISR, ASR)
  • AWS CloudFormation Template
  • Mac OS Endpoint Agents
  • Pulse Endpoint Agents for external entities
In addition to the breadth of agents available, the deployment can easily be automated.  I’ve written a script that wrote the Raspberry Pi image to an SD card, then mounted it and applied customizations.  The MSI package can be used with the plethora of Windows software deployment tools, or a link can be given to end users to install on their own.  With DNA Center the image can be pushed to Catalyst switches in bulk.  The Docker images can be build with Docker files.  If that’s not enough, there’s also all the automation tools – Ansible, Terraform…
Getting ThousandEyes deployed throughout an environment can be done with ease.

What’s next?

That completes the agent installation.  The next installment in this series will cover some test scenarios, and walk through getting monitoring configured and tests created.