Support

Package containing modules that are used internally by Numenta Python tools and plugins to extend standard library functionality. These modules should NOT be used by client applications.

nupic.support.aggregationDivide(dividend, divisor)

Return the result from dividing two dicts that represent date and time.

Both dividend and divisor are dicts that contain one or more of the following keys: ‘years’, ‘months’, ‘weeks’, ‘days’, ‘hours’, ‘minutes’, seconds’, ‘milliseconds’, ‘microseconds’.

For example:

aggregationDivide({'hours': 4}, {'minutes': 15}) == 16
Parameters:
  • dividend – (dict) The numerator, as a dict representing a date and time
  • divisor – (dict) the denominator, as a dict representing a date and time
Returns:

(float) number of times divisor goes into dividend
nupic.support.aggregationToMonthsSeconds(interval)

Return the number of months and seconds from an aggregation dict that represents a date and time.

Interval is a dict that contain one or more of the following keys: ‘years’, ‘months’, ‘weeks’, ‘days’, ‘hours’, ‘minutes’, seconds’, ‘milliseconds’, ‘microseconds’.

For example:

aggregationMicroseconds({'years': 1, 'hours': 4, 'microseconds':42}) ==
    {'months':12, 'seconds':14400.000042}
Parameters:interval – (dict) The aggregation interval representing a date and time
Returns:(dict) number of months and seconds in the interval: {months': XX, 'seconds': XX}. The seconds is a floating point that can represent resolutions down to a microsecond.
nupic.support.getArgumentDescriptions(f)

Get the arguments, default values, and argument descriptions for a function.

Parses the argument descriptions out of the function docstring, using a format something lke this:

[junk]
argument_name:     description...
  description...
  description...
[junk]
[more arguments]

It will find an argument as long as the exact argument name starts the line. It will then strip a trailing colon, if present, then strip the rest of the line and use it to start the description. It will then strip and append any subsequent lines with a greater indent level than the original argument name.

Parameters:f – (function) to inspect
Returns:(list of tuples) (argName, argDescription, defaultValue) If an argument has no default value, the tuple is only two elements long (as None cannot be used, since it could be a default value itself).
nupic.support.getCallerInfo(depth=2)

Utility function to get information about function callers

The information is the tuple (function/method name, filename, class) The class will be None if the caller is just a function and not an object method.

Parameters:depth – (int) how far back in the callstack to go to extract the caller info
nupic.support.initLogging(verbose=False, console='stdout', consoleLevel='DEBUG')

Initilize NuPic logging by reading in from the logging configuration file. The logging configuration file is named nupic-logging.conf and is expected to be in the format defined by the python logging module.

If the environment variable NTA_CONF_PATH is defined, then the logging configuration file is expected to be in the NTA_CONF_PATH directory. If NTA_CONF_PATH is not defined, then it is found in the ‘conf/default’ subdirectory of the NuPic installation directory (typically ~/nupic/current/conf/default)

The logging configuration file can use the environment variable NTA_LOG_DIR to set the locations of log files. If this variable is not defined, logging to files will be disabled.

Parameters:
  • console – Defines console output for the default “root” logging configuration; this may be one of ‘stdout’, ‘stderr’, or None; Use None to suppress console logging output
  • consoleLevel – Logging-level filter string for console output corresponding to logging levels in the logging module; may be one of: ‘DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’, or ‘CRITICAL’. E.g., a value of’WARNING’ suppresses DEBUG and INFO level output to console, but allows WARNING, ERROR, and CRITICAL
nupic.support.title(s=None, additional='', stream=<open file '<stdout>', mode 'w'>)

Utility function to display nice titles

It automatically extracts the name of the function/method it is called from and you can add additional text. title() will then print the name of the function/method and the additional text surrounded by tow lines of dashes. If you don’t want the name of the function, you can provide alternative text (regardless of the additional text)

Parameters:
  • s – (string) text to display, uses the function name and arguments by default
  • additional – (string) extra text to display (not needed if s is not None)
  • stream – (stream) the stream to print to. Ny default goes to standard output

Examples:

def foo():
  title()

will display:

---
foo
---

def foo():
  title(additional='(), this is cool!!!')

will display:

----------------------
foo(), this is cool!!!
----------------------

def foo():
  title('No function name here!')

will display:

----------------------
No function name here!
----------------------

Configuration Base

This is the base Configuration implementation. It provides for reading configuration parameters from nupic-site.xml and nupic-default.xml.

class nupic.support.configuration_base.Configuration

This class can be used to fetch NuPic configuration settings which are stored in one or more XML files.

If the environment variable NTA_CONF_PATH is defined, then the configuration files are expected to be in the NTA_CONF_PATH search path, which is a ‘:’ separated list of directories (on Windows the separator is a ‘;’). If NTA_CONF_PATH is not defined, then it is loaded via pkg_resources.

classmethod clear()

Clear out the entire configuration.

classmethod dict()

Return a dict containing all of the configuration properties

Returns:(dict) containing all configuration properties.
classmethod findConfigFile(filename)

Search the configuration path (specified via the NTA_CONF_PATH environment variable) for the given filename. If found, return the complete path to the file.

Parameters:filename – (string) name of file to locate
classmethod get(prop, default=None)

Get the value of the given configuration property as string. This returns a string which is the property value, or the value of “default” arg. If the property is not found, use getString() instead.

Note

it’s atypical for our configuration properties to be missing - a missing configuration property is usually a very serious error. Because of this, it’s preferable to use one of the getString(), getInt(), getFloat(), etc. variants instead of get(). Those variants will raise KeyError when an expected property is missing.

Parameters:
  • prop – (string) name of the property
  • default – default value to return if property does not exist
Returns:

(string) property value, or default if the property does not exist
classmethod getBool(prop)

Retrieve the requested property and return it as a bool. If property does not exist, then KeyError will be raised. If the property value is neither 0 nor 1, then ValueError will be raised

Parameters:prop – (string) name of the property
Raises:KeyError, ValueError
Returns:(bool) property value
classmethod getConfigPaths()

Return the list of paths to search for configuration files.

Returns:(list) of paths
classmethod getFloat(prop)

Retrieve the requested property and return it as a float. If property does not exist, then KeyError will be raised.

Parameters:prop – (string) name of the property
Returns:(float) property value
classmethod getInt(prop)

Retrieve the requested property and return it as an int. If property does not exist, then KeyError will be raised.

Parameters:prop – (string) name of the property
Returns:(int) property value
classmethod getString(prop)

Retrieve the requested property as a string. If property does not exist, then KeyError will be raised.

Parameters:prop – (string) name of the property
Raises:KeyError
Returns:(string) property value
classmethod readConfigFile(filename, path=None)

Parse the given XML file and store all properties it describes.

Parameters:
  • filename – (string) name of XML file to parse (no path)
  • path – (string) path of the XML file. If None, then use the standard configuration search path.
classmethod set(prop, value)

Set the value of the given configuration property.

Parameters:
  • prop – (string) name of the property
  • value – (object) value to set
classmethod setConfigPaths(paths)

Modify the paths we use to search for configuration files.

Parameters:paths – (list) of paths to search for config files.

Configuration Custom

This Configuration implementation allows for persistent configuration updates stored in nupic-custom.xml in the site conf folder.

class nupic.support.configuration_custom.Configuration

This class extends the ConfigurationBase implementation with the ability to read and write custom, persistent parameters. The custom settings will be stored in the nupic-custom.xml file.

If the environment variable ‘NTA_CONF_PATH’ is defined, then the configuration files are expected to be in the NTA_CONF_PATH search path, which is a ‘:’ separated list of directories (on Windows the seperator is a ‘;’). If NTA_CONF_PATH is not defined, then it is assumed to be NTA/conf/default (typically ~/nupic/current/conf/default).

classmethod clear()

Clear all configuration properties from in-memory cache, but do NOT alter the custom configuration file. Used in unit-testing.

classmethod getCustomDict()

Return a dict containing all custom configuration properties

retval: dict containing all custom configuration properties.

classmethod loadCustomConfig()

Loads custom configuration settings from their persistent storage. DO NOT CALL THIS: It’s typically not necessary to call this method directly - see NOTE below.

NOTE: this method exists solely for the benefit of prepare_conf.py, which needs to load configuration files selectively.

classmethod resetCustomConfig()

Clear all custom configuration settings and delete the persistent custom configuration store.

classmethod setCustomProperties(properties)

Set multiple custom properties and persist them to the custom configuration store.

properties: a dict of property name/value pairs to set

classmethod setCustomProperty(propertyName, value)

Set a single custom setting and persist it to the custom configuration store.

propertyName: string containing the name of the property to get value: value to set the property to

Console Printer

This module defines ConsolePrinterMixin and Tee.

The ConsolePrinterMixin is used by objects that need to print to the screen under the control of a verbosity level.

The Tee class is used to redirect standard output to a file in addition to sending it to the console.

class nupic.support.console_printer.ConsolePrinterMixin(verbosity=1)

Mixin class for printing to the console with different verbosity levels.

Parameters:verbosity – (int) 0: don’t print anything to stdout 1: normal (production-level) printing 2: extra debug information 3: lots of debug information
cPrint(level, message, *args, **kw)

Print a message to the console.

Prints only if level <= self.consolePrinterVerbosity Printing with level 0 is equivalent to using a print statement, and should normally be avoided.

Parameters:
  • level – (int) indicating the urgency of the message with lower values meaning more urgent (messages at level 0 are the most urgent and are always printed)
  • message – (string) possibly with format specifiers
  • args – specifies the values for any format specifiers in message
  • kw – newline is the only keyword argument. True (default) if a newline should be printed
class nupic.support.console_printer.Tee(outputFile)

This class captures standard output and writes it to a file in addition to sending it to the console

File System Helpers

This module contains file-system helper functions.

nupic.support.fs_helpers.makeDirectoryFromAbsolutePath(absDirPath)

Makes directory for the given directory path with default permissions. If the directory already exists, it is treated as success.

Parameters:absDirPath – (string) absolute path of the directory to create.
Raises:OSError if directory creation fails
Returns:(string) absolute path provided

Group By

nupic.support.group_by.groupby2(*args)

Like itertools.groupby, with the following additions:

  • Supports multiple sequences. Instead of returning (k, g), each iteration returns (k, g0, g1, ...), with one g for each input sequence. The value of each g is either a non-empty iterator or None.
  • It treats the value None as an empty sequence. So you can make subsequent calls to groupby2 on any g value.

Note

Read up on groupby here: https://docs.python.org/dev/library/itertools.html#itertools.groupby

Parameters:args – (list) Parameters alternating between sorted lists and their respective key functions. The lists should be sorted with respect to their key function.
Returns:(tuple) A n + 1 dimensional tuple, where the first element is the key of the iteration, and the other n entries are groups of objects that share this key. Each group corresponds to the an input sequence. groupby2 is a generator that returns a tuple for every iteration. If an input sequence has no members with the current key, None is returned in place of a generator.

Lock Attributes

The lock attributes machinery is engaged by default. To deactivate it define the NTA_DONT_USE_LOCK_ATTRIBUTES environment variable. The benefit is that there will be no runtime overhead (Except for a one-time check when classes that derive from LockAttributesMixin are defined or methods decorated with _canAddAttributes are defined).

class nupic.support.lock_attributes.LockAttributesMetaclass(name, bases, dict)

This metaclass makes objects attribute-locked by decorating their __init__ and __setstate__ methods with the _allow_new_attributes decorator.

It doesn’t do anything unless the environment variable NTA_USE_LOCK_ATTRIBUTES is defined. That allows for verifying proper usage during testing and skipping it in production code (that was verified during testing) to avoid the cost of verifying every attribute setting.

It also replaces the __setattr__ magic method with a custom one that verifies new attributes are set only in code that originates from a decorated method (normally __init__ or __setstate__).

If the target class has no __init__ method it adds a trivial __init__ method to provide a hook for the decorator (the _simple_init function defined above)

class nupic.support.lock_attributes.LockAttributesMixin(*args, **kw)

This class serves as a base (or mixin) for classes that want to enforce the locked attributes pattern (all attributes should be defined in __init__ or __setstate__.

All the target class has to do add LockAttributesMixin as one of its bases (inherit from it).

The metaclass will be activated when the application class is created and the lock attributes machinery will be injected (unless the deactivation_key is defined in the environment)

PyMySQL Helpers

Helper utilities for python scripts that use pymysql

pymysql_helpers.retrySQL(timeoutSec=300, logger=None)

Return a closure suitable for use as a decorator for retrying a pymysql DAO function on certain failures that warrant retries ( e.g., RDS/MySQL server down temporarily, transaction deadlock, etc.). We share this function across multiple scripts (e.g., ClientJobsDAO, StreamMgr) for consitent behavior.

Note

Please ensure that the operation being retried is idempotent.

Note

logging must be initialized before any loggers are created, else there will be no output; see nupic.support.initLogging()

Usage Example:

@retrySQL()
def jobInfo(self, jobID):
    ...
Parameters:
  • timeoutSec – How many seconds from time of initial call to stop retrying (floating point)
  • logger – User-supplied logger instance.