Network Topology Visualization – Example of Using LLDP Neighborships, NETCONF and little Python/Javascript

Ok, this is a continuation of two streams of articles here, first my recent NETCONF tutorial here, and secondly my very old project (back then in Java) of visualization of network topologies using SNMP information called “HelloRoute”. So this is a resurrection of a very old ideas, just using newer methods and tools. But first a foreword on visualization.

Foreword – Visualization use in Network Infrastructure by Author’s experience

Well, as far as I would say, automated network visualization or documentation never really took of as primary source of documentation, everywhere I look we still maintain manually created maps with version control, trying to keep them up-to-date in change process and etc… , the reason why this is so is the context that human author can give the map, for example office networks mapped by purpose or parts of buildings, or by legal organizations. Have a look on the picture below, this is a difference between human and automated maps in most generic network modeling tools.

Now to not completely kill the point of you finishing this tutorial, I BELIEVE THE PROBLEM IS THAT VISUALIZATION TOOLS ON MARKET ARE MOSTLY GENERIC PRODUCTS, by this I mean they have generic algorithms, only follow what the vendor could “expect” in advance and there are not many ways how to add extra code/logic to them to fit YOUR visualization context without paying $$$ to the vendor. So from my perspective the for any network infrastructure maintenance organization (e.g. myself working in my job or freelance consultancy) the SOLUTION IS TO BUILD VISUALIZATION TOOLS IN-HOUSE using available components (libraries, algorithms) on the fly and not put forward generic vendor software if they cannot be heavily customized (and I am yet to find such vendor).

What we will build in this tutorial then?

Simple, we will build something like this, which is a middle-ground between the two extremes, or at least for my small Spine-Leaf network( LIVE DEMO link HERE ):

This is not the best visualization in the world (I do not claim that), the point here is to try to show you that building something yourself is not that hard as it seems with a little bit of scripting.

Part I. LAB topology and prerequisites

We will continue exactly where we left last time in the NETCONF tutorial and re-iterate the same LAB and requirements.

A. LAB Topology

Simple as last time, I am having my development laptop with SSH/IP access to two comware7 switches, those in turn have some LLDP neighbors on both uplinks and server downlinks.

LAB Topology is simply active two comware7 switches with IP management access

LAB Topology is simply active two comware7 switches with IP management access

NOTE: Yes, servers speak LLDP and from my perspective is a good practice to have that running, on both Windows and Linux this is a few clicks/commands work to enable and also hyper-visors in DCs can have it, for example on ESX/vmWare’s distributed switch settings, this is one checkbox to have enabled.

B. Installed python + libraries

I will be using python 2.7 interpreter, so download that for your system (preferred linux so that installation is simple) and download/install both ncclient library using pip and the HP Networkings PyHPEcw7 library using a few commands below:

C. (Optional) Web development tools plugin

When we get to developing the visualization, we will be using simple HTML/CSS + Javascript, you only need a notepad to work on this, but for troubleshooting, I really recommend that you googlesearch “Web development tools” plugin for your browser or at least learn where your particular browser have “console” view so you can see what the Javascript prints to the console as debug messages during problems.

Part II. Example python script to pull LLDP neighbors to a JSON map

Using the knowledge you gained in my previous NETCONF tutorial, you should be able to understand what is happening here, so I will just put the whole code here, and provide some explanation afterwards. But I will not go through this line-by-line as this is NOT a python tutorial (other pages for that). The code essentials:

INPUT: Needs “DEVICES.txt” file to exist nearby that has list of IPs/Hostnames to access, one host at each line, e.g. I have inside my DEVICES.txt:

OUTPUT: Then the script will product three JSON files, these are:

  • graph.json – Main topology file, that will hold in JSON format all the graph nodes (devices) and links (interfaces), we will alter use this as input to visualization
  • no_neighbor_interfaces.json – This is a bonus JSON file that I decided to create to have available also a quick list of interfaces that are in “UP” operational status, but there is no LLDP neighbor behind them, this is an “unknown factor” risk I want to be aware of in my visualization exercise
  • neighborships.json – This describes per-device a list of interfaces and LLDP neighbor behind each of them.

SCRIPT SOURCE:

Yes I realize this is a super-long script, but instead of describing all the script content using this article, I tried really hard to describe its operation using comments inside the code, your only real work to adjust for your lab is to change the configuration parameters at lines 15,16,17  and depending on your labs naming convention, you might want to change the regular expressions at lines 33-36 to match your logic.

Part III. Examining the output JSON

So after I run the python script from Part II in my lab topology, two JSON files were produced, the first one is simply the description of a topology using a simple list of nodes (a.k.a. devices) and links (a.k.a. interfaces), here is the graph.json:

Ok, then there are also two important JSONs called neighborship.json and no_neighbor_interfaces.json that in a very similar way hold information about neighborships, but to save space here, if you want to see their structure, have a look here.

Part III. Visualization of graph.json using D3 JavaScript library

My first inspiration for this came by checking the D3 library demo of Force-Directed Graph here.  It shows a simple, but powerful code to visualize character co-apperances in Victor Hugo’s Les Misérables, but using input also as JSON structure of nodes and links. (Actually to tell the truth my python in Part II produced exactly this type of JSON after I knew this D3 demo, so the python you seen was already once re-written to create exactly this type of JSON file structure).

The second and biggest change I did to that demo is that I needed some form of hierarchy in my graph to separate core/aggregation/access layer and endpoints, so I hijacked the D3’s Y-axis gravity settings to create multiple artificial gravity lines, have a look on lines 178-189 to see how I have done this.

Again just like in the previous Python script in Part II, here is a complete JavaScript, I tried to put together a good comments into the code as there is simply not enough space here to explain this line by line. But this code simply works.

Part IV. Final visualization of my LAB

When you put together the JSON files produced by python script in Part II and the D3 visuzalization from Part III, your result on a super-small topology like my LAB will be something like this (using screenshot right now):

Visualization of LAB topology

Boring, right? Click here for live JavaScript animated demo.

Right, but this is not limited to only my small lab, I actually tried running this in lab at work with a little bit larger device count, this is the result, and again click here for BIGGER live demo.

Example of using the shown python/JavaScript visualization on larger LAB

Click here for BIGGER live JavaScript animated demo.

Summary

Well, I shared here with you my quick experiment/example of how to do a quick visualization of network topology using first quick mapping with NETCONF+python+PyHPEcw7 and then visualizing it using Javascript+D3 library to create an interactive, self-organizing map of the topology and some small information pop-ups for each device after clicking on it. You can download the whole project as a package here.

About Peter Havrila

Author's Profile