Network Automation Notes

It’s a good thing to automate the boring stuff. It also pays well to know what exactly there is to “automate”. Cross vendor support for network programmability is a bit behind and even though there are standards in several RFCs, core networks can be slower to respond to changes. Rightfully so. Yet if you would happen to be the brave one to do it, then I’m guessing a starting point will help.

Goals

Most network automation starts wrong in my opinion. AI will not save us from bad designs. Hard/soft goals must be set in place, based on the questions we ask, like: What are we trying to automate, what are we trying to know from the network, what information will be relevant, what will not. Do existing systems fulfill their role, how satisfied are we with existing tooling, can we do better, etc, etc.

A network is not useful only for network engineers but many other areas as well (operations, security, sysadmins, firewalls, load balancers, radio, value-added services). This is a chance to gather common interactions to help build the scaffolding before any attempt is made.

So, in essence, it would seem to treat this task as just a piece of software with its bugs, design specifications, RFCs, versions, etc. Implementation details are not helpful at this stage (i.e. what language do we use).

So some other ideas would be: provisioning tasks, operational tasks as verifying a configuration across multiple devices, servers in the network that report to a controller that some users are experiencing congestion on a link and request traffic redirection, traffic-engineering paths across the network using PCEP (RFC 5440), traffic trends and provisioning notifications of all links, label-switched path statistics, inventory systems (customer, vendor), route replays (network state replay), network graph based on rd/rt vrf data, geographical and vrf aware IPAM — are just some of the examples off the top of my head.

Some of these are resolved in part via data modeling from what a database can tell us, as long as data is populated and up-to-date.

Once the idea is written down it could be that an issue tracker (for bugs, proposals, wikis, storyboarding, etc) might come in handy (if you treat this task just like a piece of software), pick the feature and (set of) devices to first work on. Prioritize them based on what can be done with existing hardware.

Data mining and gathering

This does not come cheap. Untethered access to the network is paramount, even to test out ideas with a live environment. Announced beforehand of course. How will you get the data, in what format will you get it, how is the data represented and how will data be pushed on the device — all will depend on the devices you are speaking to, type of device and the communication protocol used between them. On Juniper you have several options to talk to a router, but when you have multi-vendor or not even IP nodes involved then it can get hairy. The most important is to have at least CLI access, but the best is an API. There are still devices that are text-driven interfaces only, which are the hardest to work with concerning automating them. So when there are 3 - 4 vendors in the network where each knows its own communication protocol then that’s not quite easy.

Whether using a standards-based API like NETCONF, or a vendor-specific API, you’ll eventually need to interact with one of these to retrieve data from a device or push configuration changes. An API is just like a contract from one piece of software to another piece of software to send and receive full or partial configuration data sets.

A) Using command-line interface (CLI), configuration

This is the most cumbersome way to get data as you will need to scrape the screen essentially, and each output differs across vendors, devices or software versions. Works well in simple setups.

Over time, a data modeling language was developed to address this, and network management protocols have these associated data modeling languages. For instance, SNMP has SMI language, SMIv2 is currently in use and defined in RFC 2578, RFC 2579 and RFC 2580. SNMP is most commonly used, but not real time, can be taxing on CPU — however not such an issue now — but to quote Juniper: “The so-called pull model used by SNMP and the CLI, which requires additional processing to periodically poll the network element, directly limits scaling.”

B) Through NETCONF

Defined in RFC 6241, NETCONF protocol uses (XML) based data encoding for the configuration data as well as the protocol messages. Support for encoding in JSON was also added later. Every NETCONF message is a well-formed XML document. Client applications access the NETCONF server using the SSH protocol and use the standard SSH authentication mechanism. The use of SSH is to establish connections between a configuration management server and a device running the NETCONF server. YANG is a data modeling language representing data structures in an XML tree format. YANG data models comprise of modules and submodules and can define configuration and state data, notifications, and Remote Procedure Calls for use by NETCONF-based operations. In distributed computing, a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction.

C) Through gRPC subscription function calls

You have these collectors that request data by initiating to juniper device a telemetry subscription. Data is requested only once and is streamed periodically. It uses HTTP/2 for transport, Protocol Buffers as the interface description language https://en.wikipedia.org/wiki/Interface_description_language, and provides features such as authentication, bidirectional streaming and flow control, blocking or nonblocking bindings, and cancellation and timeouts.

Key takeaways:

• gRPC is an open-source framework that provides secure and reliable transport of data.
• Data streamed through gRPC is formatted in OpenConfig key/value pairs in protocol buffers (gpb) messages and are UDP encapsulated.
• OpenConfig supports the use of vendor-neutral data models for configuring and managing multivendor networks, and Protocol Buffers is a method of serializing structured data. The messages are serialized into a binary wire format which is intended to be read by a machine rather than a human being.

• A software developer defines data structures (called messages) and services in a proto definition file (.proto) and compiles it with a compiler protoc. This compilation generates code that can be invoked by a sender or recipient of these data structures.

D) OpenDaylight

There’s the ODL controller. ODL provides a southbound Network Configuration Protocol (NETCONF) connector API, which uses NETCONF and YANG models, to interact with a network device. A southbound interface, an OpenFlow (or alternative) protocol specification, enables the communication between ODL and routers or switches. Once this feature is implemented on a router, you can use the ODL platform to carry out any configuration change in the routers, orchestrate and provision the router, and execute Remote Procedure Calls (RPCs) to the router to get state information.

F) Scraping the screen

(when no CLI is even available or when using for instance Citrix for remote device administration). This is the worst and makes close to impossible to automate this.

Special Mention is Juniper Extension Toolkit (JET)

In short, this allows using APIs at a deeper level as Juniper daemons communicate using a standard API. If you want to develop software that communicates using the same API you can by using JET. It’s language agnostic, works with gRPC or Netconf, can be deployed on the box itself. So you have full control, like injecting state directly in the routing control plane.

Whenever someone configures a route it first goes to the management plane where all the audit and verification are done, then it goes back to the control plane as that is where the RIB entries have to be updated, running on a dedicated routing engine. Then there is RIB to FIB translation and finally, it goes to silicon.

Using JET you can do a bypass of the management plane and you have more tools if you are willing to play with the control plane and accept the potential downsides of course.

In my opinion, this is one of the best methods that a vendor can offer since you are now treating the device as just some piece of software. In the above presentation, a demo is shown at around 16 min mark whereby MX480 and ACX’s are servicing vessels which themselves are connected using satellite modems to a satellite service provider. So you have sites that are on the move. Cool. So while the vessels are in the middle of the satellites spot beam and no weather change occurred this would be fine, but there needed to be a way to dynamically detect the service degradation once the vessels were at the edge of the beam or the weather worsened, and to reapply new CIR and PIR rates on the interface based on SNMP data received from the satellite modem.

The JET software can work from a server or from the MX itself, and how the software is tied with the underlying software is with a remote procedure call framework. This framework is similar to gRPC but it’s from the Apache Foundation.

From one of the contributors of Thrift:

There’s a lot of repeated work you have to do when you’re writing a server — primarily designing a protocol and writing code to serialize and deserialize messages on the protocol, but also dealing with sockets and managing concurrency, and writing clients in many languages. Thrift automatically does all of this, given a description of the functions you want to expose from your server to clients. It’s also useful for serializing data on disk or into shared memory (where many of the same problems come up).

Instead of fiddling around with bytes, encodings and fighting the subtleties of sockets, Apache Thrift allows you to create bindings to Thrift API in many languages. The reason why this matter is that for an automation framework to accomplish its task is that the device would need to speak a protocol such that it’s easy to pack/unpack data to/from, stream data from networked devices and these networked devices need to speak a common protocol. Be it JSON, XML, or gRPC. However based on the vendor, product or version it may be difficult to provide a consistent mechanism to achieve more complex tasks like the one described above with 100’s of devices from 2-3 vendors. End of Part 1