How To Draw Neural Network Diagram

Introduction Graph Neural Networks

Graph neural networks (GNNs) are a set of deep learning methods that work in the graph domain. These networks have recently been practical in multiple areas including; combinatorial optimization, recommender systems, computer vision – simply to mention a few. These networks can likewise be used to model big systems such as social networks, poly peptide-protein interaction networks, cognition graphs among other research areas. Unlike other data such equally images, graph information works in the non-euclidean space. Graph analysis is therefore aimed at node nomenclature, link prediction, and clustering.

In this article, permit's explore Graph neural networks (GNNs) further.

What is a Graph?

A Graph is a data structure containing nodes and vertices. The relationship between the various nodes is defined past the vertices. If the management is specified in the nodes the graph is said to exist directed, otherwise, it is undirected.

A corking example of graphs in use is modeling the connection betwixt diverse people in a social network.

GNN (Graph Neural Networks)

Graph Neural Networks are a special form of neural networks that are capable of working with data that is represented in graph form. These networks are heavily motivated by Convolutional Neural Networks (CNNs) and graph embedding . CNN's are not able to handle graph data because the nodes in the graphs aren't represented in any order and the fact that dependency information between two nodes is represented by edges.

Graphs with NetworkX

Let's accept a infinitesimal and expect at how i can create graphs using NetworkX. NetworkX is a Python package that tin be used for creating graphs. Here is how you tin use the package to create an empty graph with no nodes.

                                import networkx as nx  Yard = nx.Graph()

Y'all tin then add together some nodes to the graph using the `add_nodes` role.

Next, add some edges to the graph using the `add_edges_from` function.

                                edges = [(2,one),(2,ii),(3,ii),(4,iii),(half dozen,4),(7,5),(xiv,5)]  G.add_edges_from(edges)

The graph tin exist visualized using Matplotlib. That is done past calling the `draw` role and using Matpotlib to show the graph.

                                nx.describe(Thousand, with_labels=True, font_weight='bold')  import matplotlib.pyplot every bit plt  plt.bear witness()

How exercise Graph Neural Networks work?

The idea of graph neural network (GNN) was offset introduced past Franco Scarselli Bruna et al in 2009. In their paper dubbed " The graph neural network model ", they proposed the extension of existing neural networks for processing data represented in graphical form. The model could process graphs that are acyclic, cyclic, directed, and undirected. The objective of GNN is to acquire a state embedding that encapsulates the information of the neighborhood for each node. This embedding is used to produce the output. The output can exist, for case, a node label.

The original GNN proposal had a couple of limitations:

Updating of the hidden states of nodes was inefficient for a fixed point
The GNNs used the aforementioned parameters in each iteration while other neural networks use different parameters in each layer
Modeling of informative features obtained from the edges was difficult

Traditional Graph Assay methods

Graphs can as well be analyzed using traditional methods . These methods are usually algorithms. They include:

Shortest path algorithms such as Dijkstra'southward algorithm
Searching algorithms such as the Breadth-first search algorithm
Spanning tree algorithms such as Prim's algorithm

The challenge of these methods is the requirement of prior knowledge hence they can not be used for graph nomenclature.

Types of Graph Neural Networks

There are several types of Graph Neural Networks. Permit'due south take a await at a couple of them.

Graph Convolutional Networks (GCNs)

Graph Convolutional Networks (GCNs) use the same convolution operation as in normal Convolutional Neural Networks . GCNs learn features through the inspection of neighboring nodes. They are unremarkably made up of a Graph convolution, a linear layer, and not-linear activation. GNNs work by accumulation vectors in the neighborhood, passing the result to a dense neural net layer, and finally applying non-linearity.

GNNs differ from CNNs in that they are built to work with non-Euclidian structured data. There are 2 major types of GCNs namely:

Spatial Convolutional Networks. In these networks, the features of neighboring nodes are combined into a key node. The features are summed similar to the normal convolution operation.
Spectral Convolutional Network : In Spectral networks, the convolution operation is defined in the Fourier domain by calculating the eigendecomposition of the graph Laplacian .

Graph Auto-Encoder Networks

Graph Auto-Encoder Networks are made up of an encoder and a decoder. The 2 networks are joined by a bottleneck layer. An encode obtains features from an image by passing them through convolutional filters. The decoder attempts to reconstruct the input. Autoencoder models are known to deal with extreme class imbalance that is mutual in link prediction problems. Graph Auto-Encoder Networks, therefore, attempt to acquire graph representations and then re-build the graphs using the decoder.

Recurrent Graph Neural Networks

Graph recurrent neural networks (GRNNs) utilize multi-relational graphs and use graph-based regularizers to heave smoothness and mitigate over-parametrization. Since the exact size of the neighborhood is not e'er known a Recurrent GNN layer is used to make the network more flexible. GRNN tin learn the all-time diffusion design that fits the information. It is also able to handle situations where a node is involved in multiple relations. The network is also computationally inexpensive because the number of operations is scaled linearly with regard to the number of graph edges.

Gated Graph Neural Network (GGNN)

Gated Graph Neural Networks (GGNNs) perform better than Recurrent Graph Neural Networks on bug with long-term dependencies. The long-term dependencies are encoded by node and edge gates. Long-term temporal dependencies are encoded by fourth dimension gates. Therefore, Gated Graph Neural Networks improve Recurrent Graph Neural Networks past adding gating mechanisms. These gates are responsible for remembering and forgetting data in unlike states.

List of GNN Applications

Permit'southward now take a moment to look at what GNNs can do:

Node classification : The objective here is to predict the labels of nodes by considering the labels of their neighbors.
Link prediction : In this case, the goal is to predict the relationship between various entities in a graph. This can for example be applied in prediction connections for social networks.
Graph clustering : This involves dividing the nodes of a graph into clusters. The partitioning can be done based on edge weights or border distances or past considering the graphs as objects and group similar objects together.
Graph classification : This entails classifying a graph into a category. This tin can be applied in social network assay and categorizing documents in natural language processing. Other applications in NLP include text classification, extracting semantic relationships between texts, and sequence labeling.
Computer vision : In the estimator vision world, GNNs tin can be used to generate regions of interest for object detection. They can also exist used in image nomenclature whereby a scene graph is generated. The scene generation model and so identifies objects in the prototype and the semantic human relationship between them. Other applications in this field include interaction detection and region classification.

Issues Associated with GNNs

Graph Neural Networks are powerful networks. Yet, there are a couple of known problems associated with them:

Shallow in nature : Traditional neural networks can go very deep to obtain improve operation. Unfortunately, GNNs are usually shallow with the bulk having just three layers. The creation of deep GNNs is yet an active research area.
Dynamic Graphs : Dynamic graphs have a construction that keeps changing hence making them hard to model. Dynamic GNN is too an active research expanse.
Lack of standard graph generation methods : There is no standard fashion of generating graphs. In some applications, fully connected graphs are used while in others algorithms detect graph nodes.
Scalability : Applying GNNs in applications such as recommender systems and social networks at scale is a challenge. The main hurdle here is that these methods are computationally expensive.

Example: Graph Neural Networks with PyTorch

PyTorch can be coupled with DGL to build Graph Neural Networks for node prediction. Deep Graph Library (DGL) is a Python package that can be used to implement GNNs with PyTorch and TensorFlow. The official docs provide this example on how to get started .

Let's have a look at a PyTorch case. The showtime step is to import the packages and load the data.

                                import dgl  import torch  import torch.nn as nn  import torch.nn.functional equally F  import dgl.data  dataset = dgl.data.CoraGraphDataset()  g = dataset[0]

The case shows how to build a GNN for a semi-supervised node nomenclature model on the Cora dataset. The next step is to ascertain the Graph Convolutional Network that will compute node representations using neighborhood information. This is washed using `dgl.nn.GraphConv`.

                                from dgl.nn import GraphConv  class GCN(nn.Module):      def __init__(self, in_feats, h_feats, num_classes):          super(GCN, cocky).__init__()          self.conv1 = GraphConv(in_feats, h_feats)          self.conv2 = GraphConv(h_feats, num_classes)        def forrad(cocky, thousand, in_feat):          h = self.conv1(g, in_feat)          h = F.relu(h)          h = self.conv2(thousand, h)          return h  # Create the model with given dimensions  model = GCN(g.ndata['feat'].shape[1], sixteen, dataset.num_classes)

The next move is to train the neural network. The training is done similar to how you would have done training in PyTorch or TensorFlow.

                                def train(yard, model):      optimizer = torch.optim.Adam(model.parameters(), lr=0.01)      best_val_acc = 0      best_test_acc = 0        features = m.ndata['feat']      labels = g.ndata['label']      train_mask = one thousand.ndata['train_mask']      val_mask = g.ndata['val_mask']      test_mask = 1000.ndata['test_mask']      for e in range(100):          # Forrard          logits = model(1000, features)            # Compute prediction          pred = logits.argmax(1)            # Compute loss          # Note that y'all should only compute the losses of the nodes in the grooming set.          loss = F.cross_entropy(logits[train_mask], labels[train_mask])            # Compute accuracy on training/validation/exam          train_acc = (pred[train_mask] == labels[train_mask]).float().hateful()          val_acc = (pred[val_mask] == labels[val_mask]).bladder().mean()          test_acc = (pred[test_mask] == labels[test_mask]).float().mean()            # Salvage the best validation accuracy and the respective test accuracy.          if best_val_acc < val_acc:              best_val_acc = val_acc              best_test_acc = test_acc            # Astern          optimizer.zero_grad()          loss.backward()          optimizer.pace()            if due east % five == 0:              print('In epoch {}, loss: {:.3f}, val acc: {:.3f} (best {:.3f}), test acc: {:.3f} (best {:.3f})'.format(                  e, loss, val_acc, best_val_acc, test_acc, best_test_acc))  model = GCN(yard.ndata['feat'].shape[i], 16, dataset.num_classes)  railroad train(g, model)

You can also employ the Deep Graph Library with TensorFlow. That will require yous to export that backend in your environment. Here is how that can be done on Google Colab .

                                !export DGLBACKEND tensorflow

Implementing Graph Neural Networks in TensorFlow and Keras

This paper proposes the Keras Graph Convolutional Neural Network Python packet (kgcnn) based on TensorFlow and Keras. It provides Keras layers for Graph Neural Networks. The official page provides numerous examples of how to utilise the bundle. 1 of the examples is how to use kgcnn for node nomenclature using the Cora dataset . Allow'due south take a look at a snippet of this analogy.

Grooming a GNN for graph nomenclature

The commencement step is usually to load the required packages.

                                from kgcnn.data.cora.cora import cora_graph  from kgcnn.literature.GCN import make_gcn  from kgcnn.utils.adj import precompute_adjacency_scaled, convert_scaled_adjacency_to_list, make_adjacency_undirected_logical_or  from kgcnn.utils.information import ragged_tensor_from_nested_numpy  from kgcnn.utils.learning import lr_lin_reduction

The adjacent step is to load the information and convert it into a dense matrix.

                                # Download and load Dataset  A_data, X_data, y_data = cora_graph()  # Brand node features dense  nodes = X_data.todense()

The next stride is to precompute the scaled and undirected adjacency matrix and map the adjacency matrix to the index listing plus border weights. Afterward that, the shape of the assortment is converted using NumPy.

                                # Precompute scaled and undirected (symmetric) adjacency matrix  A_scaled = precompute_adjacency_scaled(make_adjacency_undirected_logical_or(A_data))  # Use edge_indices and weights instead of adj_matrix  edge_index, edge_weight = convert_scaled_adjacency_to_list(A_scaled)  edge_weight = np.expand_dims(edge_weight, axis=-1)

Side by side, one-hot encodes the labels.

                                # Modify labels to one-hot-encoding  labels = np.expand_dims(y_data, axis=-one)  labels = np.array(labels == np.arange(70), dtype=np.float)

The model can be defined using the `make_gcn` function. The function expects the shape of the input node, the shape of the input edges, depth among others.

                                model = make_gcn(      input_node_shape=[None, 8710],      input_edge_shape=[None, 1],      # Output      output_embedd={"output_mode": 'node'},      output_mlp={"use_bias": [Truthful, Truthful, Fake], "units": [140, 70, lxx], "activation": ['relu', 'relu', 'softmax']},      # model specs      depth=3,      gcn_args={"units": 140, "use_bias": True, "activation": "relu", "has_unconnected": Truthful}  )

Hither is a summary of the model.

The next step is to railroad train this model. The training concluded after 300 epochs on this Google Colab .

                                # Training loop  trainlossall = []  testlossall = []  start = fourth dimension.process_time()  for iepoch in range(0, epo, epostep):      hist = model.fit(xtrain, ytrain,                       epochs=iepoch + epostep,                       initial_epoch=iepoch,                       batch_size=ane,                       callbacks=[cbks],                       verbose=one,                       sample_weight=train_mask  # Of import!!!                       )        trainlossall.append(hist.history)      testlossall.append(model.evaluate(xtrain, ytrain, sample_weight=val_mask))  cease = time.process_time()  print("Print Fourth dimension for taining: ", stop - outset)

You can then check the grooming and test loss by plotting them using Matplotlib.

                              plt.effigy(figsize=(12,8))  plt.plot(np.arange(one, len(trainlossall) + 1), trainlossall, label='Preparation Loss', c='blue')  plt.plot(np.arange(epostep, epo + epostep, epostep), testlossall[:, 1], label='Test Loss', c='red')  plt.xlabel('Epochs')  plt.ylabel('Accurarcy')  plt.title('GCN')  plt.legend(loc='lower right', fontsize='10-large')  plt.savefig('gcn_loss.png')  plt.evidence()