Apache MXNet - Python API Symbol

Apache MXNet - Python API: Symbol

This chapter introduces the mxnet.symbol API (often imported as mx.sym), which enables symbolic programming in Apache MXNet. Symbolic programming involves defining a computation graph first, which can then be compiled, optimized, and executed.

Understanding mxnet.symbol

MXNet’s Symbol API allows you to construct a computation graph by composing symbolic operations. Unlike imperative programming (like with NDArrays, where operations execute immediately), symbolic programming separates graph definition from execution.

Key features and benefits:

• Computational Graphs: Symbols represent nodes in a graph (operations or variables). This graph defines the flow of data and computations.
• Optimization: Once defined, the graph can be analyzed and optimized by MXNet’s engine (e.g., memory optimization, operation fusion) before execution.
• Deployment: Symbolic graphs can be easily serialized and deployed to various environments (e.g., mobile devices, servers) without needing the Python code that defined them.
• Declarative Nature: You declare what to compute, not how to compute it step-by-step.

In modern MXNet, especially with the Gluon API, users often don’t construct purely symbolic graphs from scratch extensively. Instead, they use gluon.HybridBlock which allows defining models imperatively. Then, by calling .hybridize(), the forward pass of the HybridBlock is converted into a symbolic graph internally for performance benefits. However, understanding mx.symbol is crucial for grasping how this hybridization works and for advanced use cases.

Example of a simple symbolic expression z = x + y:

import mxnet as mx

# Create symbolic variables (placeholders for data)
x = mx.sym.Variable('x')
y = mx.sym.Variable('y')

# Construct a new symbol 'z' by applying the '+' operator to 'x' and 'y'
z = x + y

print(f"Symbol x: {x}")
print(f"Symbol y: {y}")
print(f"Symbol z (x+y): {z}")

Output shows the names of the symbols and the operation for z:

<html>
 <body>
  <p>
Symbol x: <Symbol x>
Symbol y: <Symbol y>
Symbol z (x+y): <Symbol _plus0>
  </p>
 </body>
</html>

To execute this symbolic graph, you would typically bind it with actual data (NDArrays) and an executor, or more commonly, use it within a Gluon HybridBlock.

Core Class: Symbol

Class	Definition
Symbol(handle)	The main class representing a node or an entire computation graph in MXNet’s symbolic API. It can represent variables, constants, or operations.

Common Functions and Parameters in mx.symbol

Many functions in mxnet.symbol mirror those in mxnet.ndarray but operate on Symbol objects to build the graph. They do not perform computations immediately but return new Symbol objects representing the result of the operation.

Function & its Parameters	Definition
Activation(data, act_type=‘relu’, name=None, attr=None, out=None, **kwargs)	Creates a symbol that applies an activation function. data is an input Symbol.
BatchNorm(data, gamma, beta, moving_mean, moving_var, eps=0.001, momentum=0.9, fix_gamma=False, use_global_stats=False, output_mean_var=False, axis=1, cudnn_off=False, min_calib_range=None, max_calib_range=None, name=None, attr=None, out=None, **kwargs)	Creates a symbol for batch normalization.
BilinearSampler(data, grid, cudnn_off=False, name=None, attr=None, out=None, **kwargs)	Creates a symbol for bilinear sampling.
BlockGrad(data, name=None, attr=None, out=None, **kwargs)	Creates a symbol that stops gradient computation for its input data symbol.
cast(data, dtype, name=None, attr=None, out=None, **kwargs)	Creates a symbol that casts its input data symbol to a new dtype.
zeros(shape, dtype=‘float32’, ctx_group=None, **kwargs)	Creates a symbol representing an array of zeros with the given shape and dtype.
ones(shape, dtype=‘float32’, ctx_group=None, **kwargs)	Creates a symbol representing an array of ones.
full(shape, val, dtype=‘float32’, ctx_group=None, **kwargs)	Creates a symbol representing an array filled with a scalar val.
arange(start, stop=None, step=1.0, repeat=1, infer_range=True, dtype=‘float32’, ctx_group=None, **kwargs)	Creates a symbol representing a sequence of evenly spaced values.
linspace(start, stop, num, endpoint=True, name=None, attr=None, out=None, **kwargs)	Creates a symbol representing num evenly spaced numbers over a specified interval [start, stop].
histogram(data, bins, range=None, name=None, attr=None, out=None, **kwargs)	Creates a symbol that computes the histogram of the input data symbol.
power(base, exp, name=None, attr=None, out=None, **kwargs)	Element-wise exponentiation. base and exp can be Symbols or scalars.
softmax(data, axis=-1, temperature=None, dtype=None, name=None, attr=None, out=None, **kwargs)	Applies the softmax function, typically used as the output activation for multi-class classification. (Note: SoftmaxActivation is older; softmax is preferred).

Implementation Examples (Symbolic Operations)

Using mx.sym.power symbolically:

import mxnet as mx
import numpy as np

# Scalar power (evaluated at definition time by Python, not a symbol operation)
# print(mx.sym.power(3, 5)) # This would cause an error as mx.sym.power expects symbols or for a symbol to be the base/exp

# Power with symbolic variables
x_sym = mx.sym.Variable('x_data')
y_sym = mx.sym.Variable('y_data')

# Symbol for x^3
z_pow_x_cubed = mx.sym.power(x_sym, 3)
# Symbol for 4^y
z_pow_4_y = mx.sym.power(4, y_sym) # Scalar base, symbolic exponent
# Symbol for x^y
z_pow_x_y = mx.sym.power(x_sym, y_sym)

print(f"Symbol for x^3: {z_pow_x_cubed}")
print(f"Symbol for 4^y: {z_pow_4_y}")
print(f"Symbol for x^y: {z_pow_x_y}")

# To execute, bind data and use an executor (or use within Gluon HybridBlock)
# Example of evaluation (older style, for illustration of binding)
x_nd = mx.nd.array([1, 2, 3], ctx=mx.cpu())
executor_x_cubed = z_pow_x_cubed.simple_bind(ctx=mx.cpu(), x_data=x_nd.shape)
output_x_cubed = executor_x_cubed.forward(x_data=x_nd)[0]
print(f"Result of x^3 for x={x_nd.asnumpy()}: {output_x_cubed.asnumpy()}")

Output:

<html>
 <body>
  <p>
Symbol for x^3: <Symbol _power_scalar0>
Symbol for 4^y: <Symbol _rpower_scalar0>
Symbol for x^y: <Symbol _power0>
Result of x^3 for x=[1. 2. 3.]: [ 1.  8. 27.]
  </p>
 </body>
</html>

Using mx.sym.softmax (preferred over SoftmaxActivation):

data_sym = mx.sym.Variable('input_data')
softmax_output_sym = mx.sym.softmax(data_sym, axis=1) # Softmax across columns for each row
print(f"Softmax symbol: {softmax_output_sym}")

# Example evaluation
input_nd = mx.nd.array([[2., 0.9, -0.5, 4., 8.],
                       [4., -.7, 9., 2., 0.9]], ctx=mx.cpu())

# Using simple_bind for evaluation (illustrative)
# In practice, Gluon handles execution within its framework.
executor_softmax = softmax_output_sym.simple_bind(ctx=mx.cpu(), input_data=input_nd.shape)
output_softmax = executor_softmax.forward(input_data=input_nd)[0]
print(f"Softmax output:\n{output_softmax.asnumpy()}")

Output:

<html>
 <body>
  <p>
Softmax symbol: <Symbol softmax0>
Softmax output:
[[2.4258138e-03 8.0748333e-04 1.9912292e-04 1.7924475e-02 9.7864312e-01]
 [6.6843745e-03 6.0796250e-05 9.9204916e-01 9.0463174e-04 3.0112563e-04]]
  </p>
 </body>
</html>

symbol.contrib: Experimental Symbolic Features

Similar to ndarray.contrib, mxnet.symbol.contrib provides experimental symbolic operations. These may change or be removed in future MXNet versions.

Selected Functions from symbol.contrib

Function and its Parameters	Definition
rand_zipfian(true_classes, num_sampled, range_max, name=None, attr=None, out=None, **kwargs)	Symbolic version of rand_zipfian for drawing samples from an approximate Zipfian distribution within a computation graph.
foreach(body, data, init_states, name=None)	Symbolic version of foreach for defining loop-like computations within a graph.
while_loop(cond, func, loop_vars, max_iterations=None, name=None)	Symbolic version of while_loop for defining conditional iterative computations in a graph.
cond(pred, then_branch, else_branch, name=None)	Symbolic version of cond for defining conditional branching within a graph. then_branch and else_branch are functions that return Symbols.
getnnz(data, axis=None, name=None, attr=None, out=None, **kwargs)	Symbolic operator to count non-zero elements in a sparse tensor Symbol.
requantize(data, min_range, max_range, min_calib_range=None, max_calib_range=None, out_type=‘int8’, name=None, attr=None, out=None, **kwargs)	Symbolic operator for requantizing data, typically used in quantized graph inference.
index_copy(old_tensor, index_vector, new_tensor, name=None, attr=None, out=None, **kwargs)	Symbolically copies elements from new_tensor into old_tensor at specified index_vector locations.
interleaved_matmul_encdec_qk(queries, keys, num_heads, name=None, attr=None, out=None, **kwargs)	Computes matrix multiplication for query-key projections in multi-head attention (encoder-decoder). A specialized op for Transformer models.

Implementation Examples (symbol.contrib)

Using mx.sym.contrib.rand_zipfian symbolically:

from mxnet.symbol import contrib as contrib_sym

true_cls_sym = mx.sym.Variable('true_cls_data')
num_sampled = 5
range_max = 10

zipfian_group = contrib_sym.rand_zipfian(true_cls_sym, num_sampled, range_max)
# rand_zipfian returns a group of symbols: samples, expected_count_true, expected_count_sample
samples_sym = zipfian_group[0]

print(f"Symbol for Zipfian samples: {samples_sym}")

# Example evaluation
true_cls_nd = mx.nd.array([3, 1], ctx=mx.cpu())
# For group symbols, bind each element or use a wrapper that handles groups.
# Here, we'll just show binding for the 'samples' part for simplicity.
# Note: Proper binding of grouped symbols can be more complex.
executor_zipfian = samples_sym.simple_bind(ctx=mx.cpu(), true_cls_data=true_cls_nd.shape)
output_zipfian = executor_zipfian.forward(true_cls_data=true_cls_nd)[0]
print(f"Evaluated Zipfian samples (will vary):\n{output_zipfian.asnumpy()}")

Output (actual samples will vary):

<html>
 <body>
  <p>
Symbol for Zipfian samples: <Symbol rand_zipfian0_output0>
Evaluated Zipfian samples (will vary):
[[0 1 2 4 5]
 [0 2 3 4 5]]
  </p>
 </body>
</html>

symbol.image: Symbolic Image Operations

The mxnet.symbol.image module provides symbolic versions of image processing functions, analogous to mxnet.ndarray.image. These are used to define image augmentation and preprocessing steps within a computation graph, which can be beneficial for performance as these operations can be fused or optimized by the graph executor.

Selected Functions from symbol.image

Functions like adjust_lighting, crop, normalize, random_crop, random_lighting, random_resized_crop, resize, to_tensor exist in mx.sym.image and behave like their mx.nd.image counterparts but operate on Symbols to build a graph.

Implementation Example (symbol.image)

img_h, img_w, img_c = 224, 224, 3

# Symbolic placeholder for an image batch (N H W C format for many image ops)
image_data_sym = mx.sym.Variable('image_batch_data', shape=(None, img_h, img_w, img_c))

# Symbolic to_tensor (HWC -> CHW, scale 0-255 to 0-1)
# Note: to_tensor in symbol API might expect NCHW or NHWC depending on specific version/op.
# For this example, let's assume it handles NHWC -> NCHW and scales.
tensor_sym = mx.sym.image.to_tensor(image_data_sym)

# Symbolic normalize
mean_rgb = (0.485*255, 0.456*255, 0.406*255) # Means for 0-255 range data
std_rgb = (0.229*255, 0.224*255, 0.225*255)   # Stds for 0-255 range data
# After to_tensor, data is 0-1, so means/stds should be for 0-1 range.
normalized_sym = mx.sym.image.normalize(tensor_sym,
                                        mean=(0.485, 0.456, 0.406),
                                        std=(0.229, 0.224, 0.225))

print(f"Symbol for normalized image data: {normalized_sym}")

# This `normalized_sym` could then be fed into a symbolic CNN definition.
# For evaluation, one would bind it with actual image data.
# dummy_image_batch_nd = mx.nd.random.uniform(0, 255, (1, img_h, img_w, img_c), ctx=mx.cpu())
# exec_norm = normalized_sym.simple_bind(ctx=mx.cpu(), image_batch_data=dummy_image_batch_nd.shape)
# output_norm = exec_norm.forward(image_batch_data=dummy_image_batch_nd)[0]
# print(f"Shape of normalized output: {output_norm.shape}")

Output:

<html>
 <body>
  <p>
Symbol for normalized image data: <Symbol normalize0>
  </p>
 </body>
</html>

symbol.random: Symbolic Random Number Generation

The mxnet.symbol.random module provides functions to create Symbols that, when executed, will generate random numbers. This is useful for adding stochasticity to models (e.g., dropout) or for initializing parameters symbolically if needed in very specific scenarios (though typically parameters are initialized as NDArrays).

Selected Functions from symbol.random

Functions like uniform, normal, randn, poisson, exponential, gamma, multinomial, negative_binomial, generalized_negative_binomial, shuffle, randint, and various *_like functions are available. They take shape parameters and distribution parameters, returning a Symbol representing the random sampling operation.

Implementation Example (symbol.random)

# Symbolic random uniform generation
rand_unif_sym = mx.sym.random.uniform(low=-1, high=1, shape=(2, 3))
print(f"Symbol for uniform randoms: {rand_unif_sym}")

# Symbolic shuffle of a variable
data_to_shuffle = mx.sym.Variable('shuffle_input')
shuffled_data_sym = mx.sym.random.shuffle(data_to_shuffle)
print(f"Symbol for shuffled data: {shuffled_data_sym}")

# Evaluation example for shuffle
input_nd_for_shuffle = mx.nd.array([[1,2,3],[4,5,6],[7,8,9]], ctx=mx.cpu())
exec_shuffle = shuffled_data_sym.simple_bind(ctx=mx.cpu(), shuffle_input=input_nd_for_shuffle.shape)
output_shuffled = exec_shuffle.forward(shuffle_input=input_nd_for_shuffle)[0]
print(f"Original data:\n{input_nd_for_shuffle.asnumpy()}")
print(f"Shuffled data (will vary):\n{output_shuffled.asnumpy()}")

Output (shuffled data will vary):

<html>
 <body>
  <p>
Symbol for uniform randoms: <Symbol random_uniform0>
Symbol for shuffled data: <Symbol random_shuffle0>
Original data:
[[1. 2. 3.]
 [4. 5. 6.]
 [7. 8. 9.]]
Shuffled data (will vary):
[[4. 5. 6.]
 [7. 8. 9.]
 [1. 2. 3.]]
  </p>
 </body>
</html>

symbol.sparse: Symbolic Sparse Operations

MXNet supports operations on sparse matrices through mxnet.symbol.sparse (and corresponding mxnet.ndarray.sparse and mxnet.numpy.sparse modules). This is crucial for applications dealing with high-dimensional, sparse data, such as recommendation systems or natural language processing with large vocabularies.

Selected Functions from symbol.sparse

The sparse API provides symbolic operators for common tasks:

Function and its Parameters	Definition
ElementWiseSum(*args, **kwargs)	Symbolically adds multiple input sparse or dense Symbols element-wise. More efficient than repeated additions.
Embedding(data, weight, input_dim, output_dim, dtype=‘float32’, sparse_grad=False, name=None, attr=None, out=None, **kwargs)	Creates an Embedding layer Symbol. Maps integer indices (often from sparse input data) to dense vector representations using a weight matrix (the embedding matrix). sparse_grad=True enables sparse gradient updates for the weight, which is memory-efficient for large embeddings.
LinearRegressionOutput(data, label, grad_scale=1.0, name=None, attr=None, out=None, **kwargs)	Computes squared loss for linear regression. data is prediction, label is target. Typically used as the final layer in a symbolic graph for regression tasks.
LogisticRegressionOutput(data, label, grad_scale=1.0, name=None, attr=None, out=None, **kwargs)	Applies a sigmoid function and computes logistic loss. For binary classification.
MAERegressionOutput(data, label, grad_scale=1.0, name=None, attr=None, out=None, **kwargs)	Computes Mean Absolute Error loss.
abs(data, name=None, attr=None, out=None, **kwargs)	Element-wise absolute value for sparse/dense Symbols.
dot(lhs, rhs, transpose_a=False, transpose_b=False, name=None, attr=None, out=None, **kwargs)	Computes dot product of two Symbols, which can be sparse or dense. Supports matrix multiplication with optimizations for sparse inputs.
elemwise_add/sub/mul/div(lhs, rhs, name=None, attr=None, out=None, **kwargs)	Element-wise arithmetic operations supporting sparse Symbols where appropriate.
exp, sin, sign, sigmoid, etc.	Various mathematical functions applicable to sparse/dense Symbols.
Optimizer-specific update functions (e.g., adagrad_update, adam_update, sgd_update)	Symbolic representations of optimizer update rules, often used internally by the training mechanisms when dealing with sparse gradients.

Implementation Example (symbol.sparse Embedding)

Defining a symbolic Embedding layer:

input_dim = 10000  # Vocabulary size
output_dim = 100   # Embedding dimension
batch_size = 4
sequence_length = 10

# Symbolic input data (e.g., integer word indices)
# Shape: (batch_size, sequence_length) or other appropriate shape
input_indices_sym = mx.sym.Variable('input_indices', shape=(batch_size, sequence_length))

# Symbolic embedding weight matrix
embedding_weight_sym = mx.sym.Variable('embedding_weight', shape=(input_dim, output_dim))

# Create the Embedding symbol
embedded_features_sym = mx.sym.Embedding(data=input_indices_sym,
                                         weight=embedding_weight_sym,
                                         input_dim=input_dim,
                                         output_dim=output_dim,
                                         sparse_grad=False) # Set True for sparse updates if input_dim is very large

print(f"Symbol for embedded features: {embedded_features_sym}")

# To use this, you would bind it with actual NDArray data for indices and weights.
# e.g., input_indices_nd = mx.nd.random.randint(0, input_dim, (batch_size, sequence_length), ctx=mx.cpu())
# embedding_weight_nd = mx.nd.random.uniform(-0.01, 0.01, (input_dim, output_dim), ctx=mx.cpu())

# executor = embedded_features_sym.simple_bind(ctx=mx.cpu(), \
#                                            input_indices=input_indices_nd.shape, \
#                                            embedding_weight=embedding_weight_nd.shape)
# output_embedded = executor.forward(is_train=False, \
#                                  input_indices=input_indices_nd, \
#                                  embedding_weight=embedding_weight_nd)[0]
# print(f"Shape of embedded output: {output_embedded.shape}") # Expected: (batch_size, sequence_length, output_dim)

This symbolic embedded_features_sym would then be connected to subsequent layers in a neural network definition (e.g., RNNs, Transformers).