MULTIHOMING - a complete step-by-step guide [Download eBook] Advertisment

[mini-update] Network Topology Visualization #2 – Using SNMP as data source and enhanced visuals

Peter Havrila
Network Visualization Networking Projects

After a longer break, I managed to get back to my small visualization experiment from last time, and improve it a little. Firstly, the NETCONF interface was switched for a more standard SNMP one, even if that is not as cutting edge anymore.  So without keeping you waiting, here is firs the visualization example, as you can see I was playing here a little and got inspired by the Mischief-makers map from Harry Potter. Of course if you do not like the visuals just re-do the CSS to get rid of it.

Also, this time I made the code public using the much more common github.com here. 

Contents

Usage/Tutorial

Once you download this project from github, you will find several files in the directory, your starting point are the two configuration files called:

– config.ini  – holds SNMP configuration like community string and a list of IPs for each network device to be SNMP queried
– pyconfig.py – holds low level configuration on how to visualize the data recieved and how to order the devices detected into layers

Lets start with the config.ini, which is much smaller. Here is an example.

[DEFAULT]
SnmpVersion = 2c
SnmpCommunityString = devpassword
LogFile = Logs/NetworkMapper.log
Debug = yes
[DEVICES]
devices = [
     "192.168.56.200",
     "192.168.56.201",
     "192.168.56.202"
     ]

As you can see, you need to list devices using their IPs and snmp community string.

Secondly, you have the pyconfig.py, which is a python data file that controls what happens with the data received from the SNMP. Practically speaking, we use naming recursive patterns to classify the devices to a group, which is practically a layer in where that devices should appear in the map.

We also have a few definitions, but the important part is the parameter called MAX_STATS_RECORDS, which holds information on how many traffic counter readings to hold.

# TABLE ROW DEFINITIONS
sysName_named_oid = [('SNMPv2-MIB', 'sysName', 0)]
interfaces_table_named_oid = [
    ('IF-MIB', 'ifDescr'),
    ('IF-MIB', 'ifType'),
    ('IF-MIB', 'ifMtu'),
    ('IF-MIB', 'ifSpeed'),
    ('IF-MIB', 'ifPhysAddress'),
    ('IF-MIB', 'ifAdminStatus'),
    ('IF-MIB', 'ifOperStatus'),
    ('IF-MIB', 'ifHCInOctets'),
    ('IF-MIB', 'ifHCOutOctets'),
    ('IF-MIB', 'ifHighSpeed')
]
lldp_table_named_oid = [
    ('LLDP-MIB', 'lldpRemSysName'),
    ('LLDP-MIB', 'lldpRemSysDesc'),
    ('LLDP-MIB', 'lldpRemPortId'),
    ('LLDP-MIB', 'lldpRemPortDesc')
]
lldp_local_port_name = [('LLDP-MIB', 'lldpLocPortId', 0)]
#######
#STATS
#######
MAX_STATS_RECORDS = 2016
#########################################################
# REGULAR EXPLRESSIONS FOR MATCHING PORT NAMES TO SPEEDS
# NOTE: This is used in visuzation later to color lines
#########################################################
LINK_SPEEDS = [("^TwentyGigE*", "20"),
               ("^FortyGig*", "40"),
               ("^Ten-GigabitEthernet*", "10"),
               ("^GigabitEthernet*", "1")]
#########################################################
# REGULAR EXPLRESSIONS FOR MATCHING DEVICES HIERARHY
# E.g. Access layer switches have "AC" in their name
# or aggregation layer devices have "AG" in their names
#########################################################
NODE_HIERARCHY = [
                  ('^[a-zA-Z]{5}L2.*', "4", "L2.png"),
                  ('^[a-zA-Z]{5}L3.*', "5", "L3.png"),
                  ('^[a-zA-Z]{5}DS.*', "3", "DS.png"),
                  ('^[a-zA-Z]{5}AC.*', "2", "AC.png")
                  ]
IGNORED_IFTYPES = [ "l3ipvlan", "softwareLoopback", "other"]

In summary these files are quite self explanatory, but of course some experimentation is expected on your side.

Running the program

When you have everything configured, your task is to execute the python file network_mapper.py periodically, this creates the map, but also gathers counters on all interfaces it detects. Depending on the size of the network and how much granularity you need on interface stats, I recommend somewhere between 5 and 30 minutes.

I recommend experimenting first using direct python execution, e.g.

$ python3 network_mapper.py

and afterwards put it to cron scheduled for the periodic execution.

The results and new GUI  features

Every run of the program creates a visualized topology in directory /html that is identical with the example shown above. You should move these results under some web servers documents root and you are finished. As you realize I have added the funny “Mischief-Makes Map” theme, but if you edit the css files inside the results directory, it should be simple to drop this in favor of a more serious theme.

In regards to the new features, there are several.

Navigation and zoom

It was really hard to do a SWG diagram zoon, so bear with me. To achieve something like this there is a giant javascript that is doing x,y coordinates translation calculations and do not get me started on the zoom at all :). In summary it works and it is now good enough for larger topologies, that have  a tendency to be out of a single screen.

Also check the vertical force slider, that is really fun to play with 🙂

Interface statistics

We have two types here, first is similar to good old MRTG, in that it aggregates interface stats to a detailed graph, see below.

You can get this by either clicking on a node and selecting interface stats from a drop down, or clicking on any link directly.

Additionally, in the graph itself, you can have links of three colors, green when utilization below 50%, orange when utilization between 50-75% and red if utilization above 75%. This is great to identify where in the topology you have a bottleneck.

Here is a quick example done manually, since I right now do not have any lab topology with fully burned links.

Summary

Not much to say really, this is an incremental update, which unfortunately took me a year to do because of the SWG manipulations and traffic statistics with adequate performance (both in collection and GUI), but it works!

You can get it yourself on my github.com here. Just remember, keep this open source and referencing me as original author and I am all fine with anyone using this. Hope you enjoy it.

---
Peter Havrila , published on

24 comments ...

  1. Interesting !
    But why this and the previous project only works in IE Edge (verison 42)browser ?
    I tried to run index.htm in firefox, chrome, opera and it doesn’t work (doesn’t display network diagram).
    All page (index.html) elements are displayed only in IE Edge browser (verison 42)
    When I tried to run index.html on IE EDGE version 25 it didn’t work (network diagram don’t display)
    I tried on several computers and its the same effects.

    1. Are you trying my online examples or opening index.html as you downloaded the project from github? Because if you are trying to open the index.html as a file from your PC’s hard-drive you are probably hitting a cross-reference protection of the browser that blocks local files loading parts of the project from the internet (the d3 library is internet based).

  2. Hi Peter,

    You work is very awasome.
    Please could you help me to show the lldp table also from Fortigate devices?
    unfortunately not use the LLDP-MIB.. but a private MIB.

    Best Regards

    1. Then you will need to rewrite the code itself. E.g. take the private MIB in ASN.1 format, compile it for the PySNMP using pysmi tool and you end with your private MIB as a python file that you can then try to retrieve using python code.

      But based on the content/logic of the private mib you will have to re-work the visualization logic.

  3. Hello , I am trying to use your code for a demo, but I cant. Can you tell me which version of python you are using?

    Let me know some tips , if you can. Thanks.

  4. Hi Peter,

    Thanks for such awesome work.
    Can we use snmp v3? and how? is it possible to poll 1000+ devices for visualization?

    Thanks

    1. Hi Anna,

      First SNMPv3 is possible of course since the code is using the PySNMP library that can support it, I just didn’t built this into my code as I was focusing on the visuals. So you have to google some pySNMP example for v3 login and change that part of the code.

      Secondly I never tried with more than 50+ devices, but should be possible if your system has enough RAM (you know since python is a wasteful interpreted language). If my code would hit problems with that high number of boxes the solution will be to store partial results into files and also some optimizations needed (e.g. I do some ugly array orderings) as I feel on that number of devices it will take looong time to complete.

      Peter

  5. Hi Peter,
    May I know, which files should I focus and edit to have my own devices. Is it only config.ini? or any other files too…

    The visualization is based on json files right…can you confirm which json file?
    Thanks for your attention and help.

    1. Hi DZ,

      Check the config.ini for list of devices (in a form of IP list) and SNMPv2 community password.

      Then in the pyconfig.py file there are details on how to visualize them like what device to put to what layer using regular expression and also translation matrix of interface names to numerical speeds.

      Hope that helps.

  6. Hi Peter,
    I edit config.ini and remain pyconfig.
    I run the script over Dell and 3Com..both return error below… what could be wrong? Appreciate you could advise…I really want it to work.
    Testing access to 10.100.1.133 … 2019 Dec 10-11:24:18: Testing access to 10.100.1.133
    DEBUG: len of result from fetch() 1
    2019 Dec 10-11:24:19: SNMPv2-MIB::sysName.0 = SWSVR01TS
    2019 Dec 10-11:24:19: This is how to get to numeric OID: (1, 3, 6, 1, 2, 1, 1, 5, 0)
    2019 Dec 10-11:24:19: This is how to get data only: SWSVR01TS
    2019 Dec 10-11:24:19: # Connection worked, will retrieve data next
    2019 Dec 10-11:24:19: INTERFACES TABLE:
    Traceback (most recent call last):
    File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/builder.py”, line 383, in loadModules
    self.loadModule(modName, **userCtx)
    File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/builder.py”, line 361, in loadModule
    modName and modName + “.py[co]”, ‘, ‘.join([str(x) for x in self.__mibSources]))
    pysnmp.smi.error.MibNotFoundError: MIB file “IF-MIB.py[co]” not found in search path (DirMibSource(‘/usr/local/lib/python3.6/site-packages/pysnmp/smi/mibs’), DirMibSource(‘/usr/local/lib/python3.6/site-packages/pysnmp/smi/mibs/instances’), DirMibSource(‘pysnmp_mibs’), DirMibSource(‘.’), DirMibSource(‘/home/dz/.pysnmp/mibs’))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “network_mapper.py”, line 257, in
    return_code = args.func(args)
    File “network_mapper.py”, line 66, in main_with_args
    hlapi.CommunityData(config[‘DEFAULT’][‘SnmpCommunityString’]))
    File “/home/dz/SnmpNetworkTopologyVisualizator/quicksnmp.py”, line 63, in get_table
    return cut_array_to_table(fetch(handler),len(oids))
    File “/home/dz/SnmpNetworkTopologyVisualizator/quicksnmp.py”, line 112, in fetch
    varBinds) in handler:
    File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/asyncore/sync/cmdgen.py”, line 349, in nextCmd
    initialVars = [x[0] for x in vbProcessor.makeVarBinds(snmpEngine, varBinds)]
    File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/varbinds.py”, line 39, in makeVarBinds
    __varBinds.append(varBind.resolveWithMib(mibViewController))
    File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/rfc1902.py”, line 847, in resolveWithMib
    self.__args[0].resolveWithMib(mibViewController)
    File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/rfc1902.py”, line 481, in resolveWithMib
    self.__modName, self.__symName
    File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/builder.py”, line 423, in importSymbols
    self.loadModules(modName, **userCtx)
    File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/builder.py”, line 392, in loadModules
    raise error.MibNotFoundError(‘%s compilation error(s): %s’ % (modName, errs))
    pysnmp.smi.error.MibNotFoundError: IF-MIB compilation error(s): missingcaused by : MIB file “IF-MIB.py[co]” not found in search path (DirMibSource(‘/usr/local/lib/python3.6/site-packages/pysnmp/smi/mibs’), DirMibSource(‘/usr/local/lib/python3.6/site-packages/pysnmp/smi/mibs/instances’), DirMibSource(‘pysnmp_mibs’), DirMibSource(‘.’), DirMibSource(‘/home/dz/.pysnmp/mibs’))

  7. After run the network mapper… I access to html folder and run server(python server simplehttp)… but the visualization is still the default data not the switch I scan… when i check graph.json it still having the default data… not the new one.. please advise.. Thanks

  8. This is the output when i executed networkmapper
    ##################

    2019 Dec 10-11:50:12: ############
    2019 Dec 10-11:50:12: # Starting #
    2019 Dec 10-11:50:12: ############
    2019 Dec 10-11:50:12: Initializing NetworkTopology class

    ############################################
    Testing access to 10.100.1.33 … 2019 Dec 10-11:50:12: Testing access to 10.100.1.33
    DEBUG: len of result from fetch() 1
    2019 Dec 10-11:50:12: SNMPv2-MIB::sysName.0 = SWSVR01TS
    2019 Dec 10-11:50:12: This is how to get to numeric OID: (1, 3, 6, 1, 2, 1, 1, 5, 0)
    2019 Dec 10-11:50:12: This is how to get data only: SWSVR01TS
    2019 Dec 10-11:50:12: # Connection worked, will retrieve data next
    2019 Dec 10-11:50:12: INTERFACES TABLE:
    Traceback (most recent call last):
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 32, in __call__
    self._testValue(value, idx)
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 549, in _testValue
    ‘all of %s failed for “%s”‘ % (self._values, value)
    pyasn1.type.error.ValueConstraintError: all of (, ) failed for “65534”

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 32, in __call__
    self._testValue(value, idx)
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 497, in _testValue
    constraint(value, idx)
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 36, in __call__
    ‘%s failed at: %r’ % (self, sys.exc_info()[1])
    pyasn1.type.error.ValueConstraintError: <ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(‘all of (, ) failed for “65534”‘,)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/base.py”, line 242, in __init__
    self.subtypeSpec(value)
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 36, in __call__
    ‘%s failed at: %r’ % (self, sys.exc_info()[1])
    pyasn1.type.error.ValueConstraintError: <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(\’all of (, ) failed for “65534”\’,)’,)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/rfc1902.py”, line 867, in resolveWithMib
    self.__args[1] = self.__args[0].getMibNode().getSyntax().clone(self.__args[1])
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/base.py”, line 349, in clone
    return self.__class__(value, **initilaizers)
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/univ.py”, line 103, in __init__
    base.AbstractSimpleAsn1Item.__init__(self, value, **kwargs)
    File “/usr/local/lib/python3.6/site-packages/pyasn1/type/base.py”, line 246, in __init__
    raise exType(‘%s at %s’ % (exValue, self.__class__.__name__))
    pyasn1.type.error.ValueConstraintError: <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(\’all of (, ) failed for “65534”\’,)’,) at IANAifType

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “/usr/local/lib/python3.6/site-packages/pysnmp/carrier/asyncore/dispatch.py”, line 46, in runDispatcher
    use_poll=True, map=self.__sockMap, count=1)
    File “/usr/local/lib/python3.6/asyncore.py”, line 207, in loop
    poll_fun(timeout, map)
    File “/usr/local/lib/python3.6/asyncore.py”, line 188, in poll2
    readwrite(obj, flags)
    File “/usr/local/lib/python3.6/asyncore.py”, line 123, in readwrite
    obj.handle_error()
    File “/usr/local/lib/python3.6/asyncore.py”, line 108, in readwrite
    obj.handle_read_event()
    File “/usr/local/lib/python3.6/asyncore.py”, line 423, in handle_read_event
    self.handle_read()
    File “/usr/local/lib/python3.6/site-packages/pysnmp/carrier/asyncore/dgram/base.py”, line 170, in handle_read
    self._cbFun(self, transportAddress, incomingMessage)
    File “/usr/local/lib/python3.6/site-packages/pysnmp/carrier/base.py”, line 70, in _cbFun
    self, transportDomain, transportAddress, incomingMessage
    File “/usr/local/lib/python3.6/site-packages/pysnmp/entity/engine.py”, line 152, in __receiveMessageCbFun
    self, transportDomain, transportAddress, wholeMsg
    File “/usr/local/lib/python3.6/site-packages/pysnmp/proto/rfc3412.py”, line 486, in receiveMessage
    cachedParams[‘cbCtx’])
    File “/usr/local/lib/python3.6/site-packages/pysnmp/entity/rfc3413/cmdgen.py”, line 152, in processResponsePdu
    cbFun(snmpEngine, origSendRequestHandle, None, PDU, cbCtx)
    File “/usr/local/lib/python3.6/site-packages/pysnmp/entity/rfc3413/cmdgen.py”, line 303, in processResponseVarBinds
    varBindTable, cbCtx):
    File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/asyncore/cmdgen.py”, line 345, in __cbFun
    varBindTable],
    File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/asyncore/cmdgen.py”, line 344, in
    [vbProcessor.unmakeVarBinds(snmpEngine, varBindTableRow, lookupMib) for varBindTableRow in
    File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/varbinds.py”, line 46, in unmakeVarBinds
    varBinds = [ObjectType(ObjectIdentity(x[0]), x[1]).resolveWithMib(mibViewController) for x in varBinds]
    File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/varbinds.py”, line 46, in
    varBinds = [ObjectType(ObjectIdentity(x[0]), x[1]).resolveWithMib(mibViewController) for x in varBinds]
    File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/rfc1902.py”, line 871, in resolveWithMib
    sys.exc_info()[1]))
    pysnmp.smi.error.SmiError: MIB object ‘IF-MIB::ifType.4228426’ having type ‘IANAifType’ failed to cast value <Integer value object at 0x7effc618c390 tagSet subtypeSpec <ConstraintsIntersection object at 0x7effcd315cc0 consts > payload [65534]>: <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(\’all of (, ) failed for “65534”\’,)’,) at IANAifTypecaused by : <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(\’all of (, ) failed for “65534”\’,)’,) at IANAifType

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “network_mapper.py”, line 257, in
    return_code = args.func(args)
    File “network_mapper.py”, line 66, in main_with_args
    hlapi.CommunityData(config[‘DEFAULT’][‘SnmpCommunityString’]))
    File “/home/joe/TOPOLOGY/SnmpNetworkTopologyVisualizator/quicksnmp.py”, line 63, in get_table
    return cut_array_to_table(fetch(handler),len(oids))
    File “/home/joe/TOPOLOGY/SnmpNetworkTopologyVisualizator/quicksnmp.py”, line 112, in fetch
    varBinds) in handler:
    File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/asyncore/sync/cmdgen.py”, line 362, in nextCmd
    snmpEngine.transportDispatcher.runDispatcher()
    File “/usr/local/lib/python3.6/site-packages/pysnmp/carrier/asyncore/dispatch.py”, line 50, in runDispatcher
    raise PySnmpError(‘poll error: %s’ % ‘;’.join(format_exception(*exc_info())))
    pysnmp.error.PySnmpError: poll error: Traceback (most recent call last):
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 32, in __call__
    self._testValue(value, idx)
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 549, in _testValue
    ‘all of %s failed for “%s”‘ % (self._values, value)
    ;pyasn1.type.error.ValueConstraintError: all of (, ) failed for “65534”
    ;
    During handling of the above exception, another exception occurred:

    ;Traceback (most recent call last):
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 32, in __call__
    self._testValue(value, idx)
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 497, in _testValue
    constraint(value, idx)
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 36, in __call__
    ‘%s failed at: %r’ % (self, sys.exc_info()[1])
    ;pyasn1.type.error.ValueConstraintError: <ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(‘all of (, ) failed for “65534”‘,)
    ;
    During handling of the above exception, another exception occurred:

    ;Traceback (most recent call last):
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/base.py”, line 242, in __init__
    self.subtypeSpec(value)
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/constraint.py”, line 36, in __call__
    ‘%s failed at: %r’ % (self, sys.exc_info()[1])
    ;pyasn1.type.error.ValueConstraintError: <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(\’all of (, ) failed for “65534”\’,)’,)
    ;
    During handling of the above exception, another exception occurred:

    ;Traceback (most recent call last):
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/rfc1902.py”, line 867, in resolveWithMib
    self.__args[1] = self.__args[0].getMibNode().getSyntax().clone(self.__args[1])
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/base.py”, line 349, in clone
    return self.__class__(value, **initilaizers)
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/univ.py”, line 103, in __init__
    base.AbstractSimpleAsn1Item.__init__(self, value, **kwargs)
    ; File “/usr/local/lib/python3.6/site-packages/pyasn1/type/base.py”, line 246, in __init__
    raise exType(‘%s at %s’ % (exValue, self.__class__.__name__))
    ;pyasn1.type.error.ValueConstraintError: <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(\’all of (, ) failed for “65534”\’,)’,) at IANAifType
    ;
    During handling of the above exception, another exception occurred:

    ;Traceback (most recent call last):
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/carrier/asyncore/dispatch.py”, line 46, in runDispatcher
    use_poll=True, map=self.__sockMap, count=1)
    ; File “/usr/local/lib/python3.6/asyncore.py”, line 207, in loop
    poll_fun(timeout, map)
    ; File “/usr/local/lib/python3.6/asyncore.py”, line 188, in poll2
    readwrite(obj, flags)
    ; File “/usr/local/lib/python3.6/asyncore.py”, line 123, in readwrite
    obj.handle_error()
    ; File “/usr/local/lib/python3.6/asyncore.py”, line 108, in readwrite
    obj.handle_read_event()
    ; File “/usr/local/lib/python3.6/asyncore.py”, line 423, in handle_read_event
    self.handle_read()
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/carrier/asyncore/dgram/base.py”, line 170, in handle_read
    self._cbFun(self, transportAddress, incomingMessage)
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/carrier/base.py”, line 70, in _cbFun
    self, transportDomain, transportAddress, incomingMessage
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/entity/engine.py”, line 152, in __receiveMessageCbFun
    self, transportDomain, transportAddress, wholeMsg
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/proto/rfc3412.py”, line 486, in receiveMessage
    cachedParams[‘cbCtx’])
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/entity/rfc3413/cmdgen.py”, line 152, in processResponsePdu
    cbFun(snmpEngine, origSendRequestHandle, None, PDU, cbCtx)
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/entity/rfc3413/cmdgen.py”, line 303, in processResponseVarBinds
    varBindTable, cbCtx):
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/asyncore/cmdgen.py”, line 345, in __cbFun
    varBindTable],
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/asyncore/cmdgen.py”, line 344, in
    [vbProcessor.unmakeVarBinds(snmpEngine, varBindTableRow, lookupMib) for varBindTableRow in
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/varbinds.py”, line 46, in unmakeVarBinds
    varBinds = [ObjectType(ObjectIdentity(x[0]), x[1]).resolveWithMib(mibViewController) for x in varBinds]
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/hlapi/varbinds.py”, line 46, in
    varBinds = [ObjectType(ObjectIdentity(x[0]), x[1]).resolveWithMib(mibViewController) for x in varBinds]
    ; File “/usr/local/lib/python3.6/site-packages/pysnmp/smi/rfc1902.py”, line 871, in resolveWithMib
    sys.exc_info()[1]))
    ;pysnmp.smi.error.SmiError: MIB object ‘IF-MIB::ifType.4228426’ having type ‘IANAifType’ failed to cast value <Integer value object at 0x7effc618c390 tagSet subtypeSpec <ConstraintsIntersection object at 0x7effcd315cc0 consts > payload [65534]>: <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(\’all of (, ) failed for “65534”\’,)’,) at IANAifTypecaused by : <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(\’all of (, ) failed for “65534”\’,)’,) at IANAifType
    caused by : MIB object ‘IF-MIB::ifType.4228426’ having type ‘IANAifType’ failed to cast value <Integer value object at 0x7effc618c390 tagSet subtypeSpec <ConstraintsIntersection object at 0x7effcd315cc0 consts > payload [65534]>: <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , > failed at: ValueConstraintError(\’all of (, ) failed for “65534”\’,)’,) at IANAifTypecaused by : <ConstraintsIntersection object at 0x7effc6244240 consts , <ConstraintsUnion object at 0x7effc62440b8 consts , >> failed at: ValueConstraintError(‘<ConstraintsUnion object at 0x7effc62440b8 consts , ) failed for “65534”\’,)’,) at IANAifType
    #################

    1. Heh, my script essentially only puts data to the html folders at the end and only if it runs without any major SNMP errors, and this output above looks like it is having quit a lot of troubles. Your problems are starting somewhere in my code on line 63 looking at this log:

      File “/home/joe/TOPOLOGY/SnmpNetworkTopologyVisualizator/quicksnmp.py”, line 63, in get_table
      return cut_array_to_table(fetch(handler),len(oids))

      Looks like your router responded to some of the SNMP questions with data that my script didn’t understood. I would recommend you doing step by step python debug and at the same time running wireshark to see what was the last SNMP querry/answer that the script asked before crashing. Usually different NW vendors use slightly diffent data types and OIDs and my script might simply be expecting something different to what your devices are sending.

      Peter

  9. Hi,
    The script starts well but an error occurs later.
    Have any ideas?

    #python3 network_mapper.py
    2019 Dec 29-16:48:48: ############
    2019 Dec 29-16:48:48: # Starting #
    2019 Dec 29-16:48:48: ############
    2019 Dec 29-16:48:48: Initializing NetworkTopology class

    ############################################
    Testing access to 192.168.1.1 … 2019 Dec 29-16:48:48: Testing access to 192.168.1.1
    DEBUG: len of result from fetch() 1
    2019 Dec 29-16:48:48: SNMPv2-MIB::sysName.0 = sw-central.scm
    2019 Dec 29-16:48:48: This is how to get to numeric OID: (1, 3, 6, 1, 2, 1, 1, 5, 0)
    2019 Dec 29-16:48:48: This is how to get data only: sw-central.scm
    2019 Dec 29-16:48:48: # Connection worked, will retrieve data next
    2019 Dec 29-16:48:48: INTERFACES TABLE:
    DEBUG: len of result from fetch() 100
    2019 Dec 29-16:48:49: IF-MIB::ifDescr.1 = Ethernet0/0
    2019 Dec 29-16:48:49: IF-MIB::ifType.1 = ethernetCsmacd
    2019 Dec 29-16:48:49: IF-MIB::ifMtu.1 = 1500
    2019 Dec 29-16:48:49: IF-MIB::ifSpeed.1 = 10000000
    2019 Dec 29-16:48:49: IF-MIB::ifPhysAddress.1 = aa:bb:cc:00:02:00
    2019 Dec 29-16:48:49: IF-MIB::ifAdminStatus.1 = up
    2019 Dec 29-16:48:49: IF-MIB::ifOperStatus.1 = up
    2019 Dec 29-16:48:49: IF-MIB::ifHCInOctets.1 = 0
    2019 Dec 29-16:48:49: IF-MIB::ifHCOutOctets.1 = 23339
    2019 Dec 29-16:48:49: IF-MIB::ifHighSpeed.1 = 10

    2019 Dec 29-16:48:49: LLDP TABLE:
    2019 Dec 29-16:48:49: ############
    Traceback (most recent call last):
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/builder.py”, line 401, in loadModules
    self.loadModule(modName, **userCtx)
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/builder.py”, line 379, in loadModule
    modName and modName + “.py[co]”, ‘, ‘.join([str(x) for x in self.__mibSources]))
    pysnmp.smi.error.MibNotFoundError: MIB file “RMON2-MIB.py[co]” not found in search path (DirMibSource(‘/usr/local/lib/python3.6/dist-packages/pysnmp/smi/mibs’), DirMibSource(‘/usr/local/lib/python3.6/dist-packages/pysnmp/smi/mibs/instances’), DirMibSource(‘pysnmp_mibs’), DirMibSource(‘.’), DirMibSource(‘/root/.pysnmp/mibs’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/builder.py”, line 362, in loadModule
    exec(codeObj, g)
    File “./LLDP-MIB.py”, line 12, in
    ZeroBasedCounter32, TimeFilter = mibBuilder.importSymbols(“RMON2-MIB”, “ZeroBasedCounter32”, “TimeFilter”)
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/builder.py”, line 441, in importSymbols
    self.loadModules(modName, **userCtx)
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/builder.py”, line 410, in loadModules
    raise error.MibNotFoundError(‘%s compilation error(s): %s’ % (modName, errs))
    pysnmp.smi.error.MibNotFoundError: RMON2-MIB compilation error(s): missing; missing; no module “RFC1271-MIB” in symbolTable at MIB RMON2-MIB; no module “RFC1271-MIB” in symbolTable at MIB TOKEN-RING-RMON-MIBcaused by : MIB file “RMON2-MIB.py[co]” not found in search path (DirMibSource(‘/usr/local/lib/python3.6/dist-packages/pysnmp/smi/mibs’), DirMibSource(‘/usr/local/lib/python3.6/dist-packages/pysnmp/smi/mibs/instances’), DirMibSource(‘pysnmp_mibs’), DirMibSource(‘.’), DirMibSource(‘/root/.pysnmp/mibs’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’), DirMibSource(‘.’))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “network_mapper.py”, line 257, in
    return_code = args.func(args)
    File “network_mapper.py”, line 86, in main_with_args
    hlapi.CommunityData(config[‘DEFAULT’][‘SnmpCommunityString’]))
    File “/var/www/html/SnmpNetworkTopologyVisualizator/quicksnmp.py”, line 63, in get_table
    return cut_array_to_table(fetch(handler),len(oids))
    File “/var/www/html/SnmpNetworkTopologyVisualizator/quicksnmp.py”, line 112, in fetch
    varBinds) in handler:
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/hlapi/asyncore/sync/cmdgen.py”, line 349, in nextCmd
    initialVars = [x[0] for x in vbProcessor.makeVarBinds(snmpEngine, varBinds)]
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/hlapi/varbinds.py”, line 39, in makeVarBinds
    __varBinds.append(varBind.resolveWithMib(mibViewController, ignoreErrors=False))
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/rfc1902.py”, line 853, in resolveWithMib
    self.__args[0].resolveWithMib(mibViewController)
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/rfc1902.py”, line 481, in resolveWithMib
    self.__modName, self.__symName
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/builder.py”, line 441, in importSymbols
    self.loadModules(modName, **userCtx)
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/builder.py”, line 401, in loadModules
    self.loadModule(modName, **userCtx)
    File “/usr/local/lib/python3.6/dist-packages/pysnmp/smi/builder.py”, line 367, in loadModule
    ‘MIB module \’%s\’ load error: %s’ % (modPath, traceback.format_exception(*sys.exc_info()))

  10. Hi Peter,
    You program is interesting. But when I try run this script I get this some output (error) like Bob20 in December 2019 posts.
    Generally, I encounter some errors when running (mainly python libraries and and other dependencies).
    The best for my opinion when you share this program in the form of a conteiner or virtual machine (ubuntu + python enviroment).

    Regards

    1. I was thinking about that approach, unfortunately I would have to recreate also the whole network somehow inside that virtual machine for that program to collect that in some form. Maybe as part of part #3.

  11. Unfortunately with Cisco devices not working

    pyasn1.type.error.ValueConstraintError: <ConstraintsIntersection object, consts , > failed at: ValueConstraintError(‘ failed at: ValueConstraintError(10101,)’,) at LldpPortNumber

    seems that the index about ports are not the same also other vendors.
    and the limit is 4096

    Someone could you help me?
    Best Regards

  12. Thanks a lot for sharing this, definitely my cuppa tea.

    I’ve nearly cracked it but having trouble seeing the html visualisation. For the time being I just have one HP switch connected, but hoping to see this in the html.

    I get all the correct snmp output from Python once network_mapper.py has completed. With this at the end:

    2020 Aug 06-23:55:55: links from LLDP
    2020 Aug 06-23:55:55: JSON printed into graph.json
    2020 Aug 06-23:55:55: JSON printed into interfaces.json
    2020 Aug 06-23:55:56: JSON printed into graph.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/_8.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/VSR1000_17.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/VSR1000_33.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/VSR1000_49.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/VSR1000-2_17.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/VSR1000-2_33.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/VSR1000-2_49.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/VSR1000-3_17.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/VSR1000-3_33.json
    2020 Aug 06-23:55:56: JSON printed into html/data/stats/VSR1000-3_49.json
    2020 Aug 06-23:55:56: JSON printed into neighborships.json
    2020 Aug 06-23:55:56: JSON printed into no_neighbor_interfaces.json
    return code:0

    graph.json shows this.

    {
    “links”: [],
    “nodes”: [
    {
    “group”: “1”,
    “id”: “”,
    “image”: “default.png”
    }
    ]
    }

    But nothing in the index.html, am I missing something?

    Cheers, George

  13. I run the software, but did not generate the topology, I only turned on the SNMP protocol, did not turn on LLDP, will it cause the generation failure

Comments are closed.