ufotest package

Submodules

ufotest.camera module

A module containing the functionality related to interacting with the camera

class ufotest.camera.AbstractCamera(config: ufotest.config.Config)

Bases: object

This is the abstract base class for wrapping access to a specific camera.

DESIGN CHOICE

Here is the reasoning for why this is necessary: Previously the access to the camera was managed by a few isolated functions in this module. That was a simpler and less bloated version, but it also was not sub optimal for the following reasons:

(1) There is the desire to be able to actually use different cameras with the ufotest framework. The isolated functions only implemented access to one specific model. So the reasonable alternative is to defines a set of methods which have to be implemented by every camera and other than that leave room for individual requirements. This needs an interface (this very class in other words)

(2) Representing the camera by a class has a secondary advantage: A class has a state. This leaves the option to potentially cache certain values from the camera interaction and thus be more efficient. Or some functionality inherently requires an external state management.

get_frame() → numpy.array

This method should wrap all interaction which is required to obtain a single frame from the camera. The frame should then be converted into a simple two dimensional numpy array and returned.

get_prop(key: str) → Any

This method is supposed to return the currently configured value of the internal property with the name key

poll() → bool

This method should return a boolean value of whether or not the camera is currently usable. This will most likely be false before a init sequence has established a connection to the camera and true afterwards. But it can also be used to indicate a possible temporary blocking during a readout operation or the like.

reset() → None

This method is supposed to reset the camera into it’s default internal state. It would be important that this also works in case the camera has issues and might not communicate correctly. Thus, this should be a hard reset to tear down the entire connection and then set it up from scratch would be best.

set_prop(key: str, value: Any) → None

This method is supposed to set the internal property of the camera with the name key to the new value

set_up() → None

This method is supposed to execute all necessary interactions with the camera to ensure that afterwards the camera can be used -> return frames, manipulate properties

supports_prop(key: str) → bool

This method should return the boolean value of whether or not, the specific camera implementation supports a a property of the given name.

tear_down() → None

This method is supposed to execute all code which is needed to properly terminate the connection with the camera. All cleanup operations so to say. It should NOT permanently disable the camera though. Subsequently calling set_up and tear_down should be possible

class ufotest.camera.InternalDictMixin

Bases: object

This is a mixin which can be used for subclasses of AbstractCamera. This abstract camera interface expects the ability to manipulate internal camera properties using three distinct functions. This mixin offers a default implementation for these methods, which will manage these internal camera properties in an internal dict “self.values”. The default implementation will simple read and write these values to this internal dict. But this mixin also allows to implement overwrite getter and setter methods for each prop to customize the behavior.

EXAMPLE

In the order of the multiple inheritance, the mixins are best placed first. This mixin furthermore expects the subclass to define a static attribute “default_values”, which is a dict and contains the default values for all the props. This dict should contain all (and only) entries for each prop which is supported by the camera, because the outcome of “supports_prop” is defined by whether or not the prop is defined in this default dict.

The behavior of the get / set operation for a specific prop can be customized by simply implementing a method which is called get_{prop name} / set_{prop name}

WHAT IT EXPECTS

• static attribute dict default_dict
• no instance attribute named values

get_prop(key: str) → Any

Returns the current value of the prop with the name prop

Parameters:key (str) – The string name of the prop Returns:The value of the prop

set_prop(key: str, value: Any) → None

Sets the new value of the prop with the name key to value

Parameters:

• key (str) – The string name of the prop
• value – The new value

Returns:

void

supports_prop(key: str) → bool

Returns whether or not the prop with the name key is supported. This is determined by it’s existence as a key within default_values.

Parameters:key (str) – The string name of the prop Returns bool:

class ufotest.camera.MockCamera(config: ufotest.config.Config)

Bases: ufotest.camera.InternalDictMixin, ufotest.camera.AbstractCamera

This is a mock implementation of the AbstractCamera interface. It does not actually interface with any real hardware, it only simulates camera behavior testing purposes.

FRAMES

As expected by the AbstractCamera interface, this class implements a functional “get_frame” method. This method returns a static picture which is based on the “sample.png” image from the static folder of the ufotest installation. The image itself displays some kind of landscape.

SET UP

The camera does indeed require the “set_up” method to be called before any frames can be captured. This is however not due to any specific reason. The setup status is internally simple represented as a boolean flag. This is to simulate camera behavior as well as possible.

EXPOSURE TIME

This class supports the “exposure_time” prop. It can be set as int values between 1 and 100. This class actually attempts to simulate the effects of exposure time somewhat. With higher values the static image becomes (1) brighter, as in higher pixel values overall and (2) there is more noise: The magnitude of the additive gaussian noise gets more with higher exposure time.

classmethod add_gaussian_noise(frame_array: numpy.ndarray, intensity: float) → numpy.ndarray

Given a base frame frame_array this method generates a random gaussian noise centered around 0 and with a standard deviation of intensity with the original frames shape and then adds the noise array to this base frame and returns the result.

Returns:The modified frame array

default_values = {'exposure_time': 1, 'hardware_version': '1.2.2', 'max_exposure_time': 100, 'min_exposure_time': 1, 'sensor_version': '1.2.2'}

get_frame() → numpy.array

This method should wrap all interaction which is required to obtain a single frame from the camera. The frame should then be converted into a simple two dimensional numpy array and returned.

load_image

Uses pillow to load the image with the given image_path as a grayscale image object.

DESIGN CHOICE

One could say that this method is redundant, because the little code within I could have just called within the constructor as it is. That is true, but the important thing is that this method is cached. The loading of the image is relatively time intensive. This is not a problem were this class be used in the normal ufotest routine the camera class is only instantiated once. But the mock implementation is mainly for testing and for testing the camera class is instantiated many more times, such that this runtime becomes an issue…

Parameters:image_path – The str absolute path to the image file Returns Image:the image object

poll()

This method should return a boolean value of whether or not the camera is currently usable. This will most likely be false before a init sequence has established a connection to the camera and true afterwards. But it can also be used to indicate a possible temporary blocking during a readout operation or the like.

reset()

This method is supposed to reset the camera into it’s default internal state. It would be important that this also works in case the camera has issues and might not communicate correctly. Thus, this should be a hard reset to tear down the entire connection and then set it up from scratch would be best.

resize_image

set_up()

This method is supposed to execute all necessary interactions with the camera to ensure that afterwards the camera can be used -> return frames, manipulate properties

tear_down()

This method is supposed to execute all code which is needed to properly terminate the connection with the camera. All cleanup operations so to say. It should NOT permanently disable the camera though. Subsequently calling set_up and tear_down should be possible

class ufotest.camera.UfoCamera(config: ufotest.config.Config)

Bases: ufotest.camera.InternalDictMixin, ufotest.camera.AbstractCamera

Implements the interface to interact with the UFO camera.

decode_frame()

Uses the raw frame data stored in the file at “data_path” to decode it into an actual image of the RAW format, which is stored at the path “frame_path”

Raises:FrameDecodingError – If there is any problem during the decoding process Returns:void

default_values = {'exposure_time': 1, 'firmware_version': '-', 'hardware_version': '-', 'sensor_version': '-'}

execute_command(command: str, cwd: Optional[str] = None) → dict

Executes a terminal command in a separate shell process and returns a dict which contains the information about the outcome of the command.

The returned dict contains the following fields: - stdout: A string which contains the entire console output of the command - stderr: A string which contains the entire error output of the command - exit_code: The int exit code of the command

Parameters:

• command – The bash string to be executed in the terminal
• cwd – An optional string to define the absolute path of the folder in which the command should be executed in.

Returns:

dict

get_frame() → numpy.ndarray

Returns a frame from the camera as a numpy array. The dimensions of this array are dependent on the camera sensor / the configuration of the camera sensor selected in the config file of the project.

The process of retrieving a frame can be roughly outlined like this. Using the command line interface of “pcitool” specific registers are set, which instruct the camera to acquire a frame. The data of this frame is then saved as a raw bytestream. By using the command line interface of ipedecode this frame is decoded into the RAW image format, which is then loaded into a numpy array and returned.

Returns:np.ndarray

parse_status_to_dict(status_output: str) → Dict[str, List[str]]

pci_read(addr: str, size: int) → str

Uses the “pci” command to perform a register readout operation of the FPGA at the given address and with the given size. Returns the string output of the console command.

Returns:string

pci_write(addr: str, value: str) → bool

Uses the “pci” command to write a new value to the given register address of the FPGA. Returns the boolean value of whether or not the write operation was successful.

Parameters:

• addr – The string representation of the register address to write to
• value – The actual value to write to the register

Returns:

boolean

poll() → bool

Returns whether or not the camera can be used.

Returns:bool

receive_frame(force=True)

Reads out the raw byte data of the previously requested frames and stores it into a temporary file whose path is defined in the “data_path” property of this class.

Parameters:force – Boolean flag of whether or not to forcefully delete the previous data file. Default is True Raises:PciError – If there is any problem with receiving the frame Returns:void

request_frame()

Writes the necessary registers of the camera to indicate that a new frame is requested

Returns:void

reset()

This method is supposed to reset the camera into it’s default internal state. It would be important that this also works in case the camera has issues and might not communicate correctly. Thus, this should be a hard reset to tear down the entire connection and then set it up from scratch would be best.

set_exposure_time(value: int, r=array([41216., 41219.34, 41222.68, 41226.02, 41229.36, 41232.7, 41236.04, 41239.38, 41242.72, 41246.06, 41249.4, 41252.74, 41256.08, 41259.42, 41262.76, 41266.1, 41269.44, 41272.78, 41276.12, 41279.46, 41282.8, 41286.14, 41289.48, 41292.82, 41296.16, 41299.5, 41302.84, 41306.18, 41309.52, 41312.86, 41316.2, 41319.54, 41322.88, 41326.22, 41329.56, 41332.9, 41336.24, 41339.58, 41342.92, 41346.26, 41349.6, 41352.94, 41356.28, 41359.62, 41362.96, 41366.3, 41369.64, 41372.98, 41376.32, 41379.66, 41383., 41386.34, 41389.68, 41393.02, 41396.36, 41399.7, 41403.04, 41406.38, 41409.72, 41413.06, 41416.4, 41419.74, 41423.08, 41426.42, 41429.76, 41433.1, 41436.44, 41439.78, 41443.12, 41446.46, 41449.8, 41453.14, 41456.48, 41459.82, 41463.16, 41466.5, 41469.84, 41473.18, 41476.52, 41479.86, 41483.2, 41486.54, 41489.88, 41493.22, 41496.56, 41499.9, 41503.24, 41506.58, 41509.92, 41513.26, 41516.6, 41519.94, 41523.28, 41526.62, 41529.96, 41533.3, 41536.64, 41539.98, 41543.32, 41546.66, 41550. ]))

KIND OF sets the exposure time of the camera. At the current point in time, this method does modify the exposure time of the camera, but which value in ms it actually is, is unclear. Supported are int values up to 100. Generally the higher the value the higher the exposure time.

Return void:

set_up()

Executes the routine to set up the camera for subsequent operations.

Specifically this involves the following series of scripts: - pcie_init - reset_fpga - power_up - reset

Returns:void

tear_down()

This method is supposed to execute all code which is needed to properly terminate the connection with the camera. All cleanup operations so to say. It should NOT permanently disable the camera though. Subsequently calling set_up and tear_down should be possible

ufotest.camera.decode_frame(data_path: str) → str

Decodes the frame data given at data_path and returns the path to the .raw image file

This function can only be called, after the actual frame data has been received from the camera. The file with the raw data has to already exist. This function is based on one of Michele’s scripts called “frame.sh”

Parameters:data_path – The string path of the file which contains the raw frame data. Raises:FrameDecodingError – When the decode command fails. The error message contains some part of the commands output. Returns:The string path to the decoded .raw image file

ufotest.camera.get_frame(tmp_path: str = '/tmp') → str

Requests a frame from the camera, receives the data, decodes it into a ‘.raw’ image and saves it.

Parameters:

tmp_path – A string path to a temporary folder, which is used to store the intermediate files which are produced during the reception and decoding process. The final image file will also be saved to this folder. Defaults to ‘/tmp’

Raises:

• PciError – Whenever anything goes wrong with the PCI communication with the camera.
• FrameDecodingError – Whenever anything goes wrong during the decoding process of the camera.

Returns:

The string path of the final ‘.raw’ file.

ufotest.camera.import_raw(path: str, n: int, sensor_width: int, sensor_height: int)

ufotest.camera.pci_read(addr: str, size)

ufotest.camera.pci_write(addr: str, value: str)

ufotest.camera.receive_frame(data_path: str) → None

Receives the raw data for a frame over the PCI interface and saves it into the file data_path

This function can only be called when a request for frame data has previously been sent to the camera! This function is based on one of Michele’s scripts called “frame.sh”

Raises:PciError – When the data reception command exits with exit code 1. The error message contains some of the output of the command. Parameters:data_path – The string path of the file into which to save the frame data. Does not have to exist yet.

ufotest.camera.request_frame() → None

Sends the necessary PCI instructions to request a frame from the camera

This function is based on one of Michele’s scripts called “frame.sh”

ufotest.camera.save_frame(path: str, tmp_path: str = '/tmp') → int

Deprecated:

ufotest.camera.set_up_camera(verbose: bool = False)

ufotest.camera.tear_down_camera(verbose: bool = False)

ufotest.cli module

Module containing all the actual console scripts of the project.

ufotest.cli.DEPENDENCY_INSTALL_FUNCTIONS = {'fastwriter': <function install_fastwriter>, 'ipecamera': <function install_ipecamera>, 'libuca': <function install_libuca>, 'libufodecode': <function install_libufodecode>, 'pcitool': <function install_pcitools>}

A dictionary whose keys are possible string argument names for installable dependencies and the values are function objects, which actually perform the installation of that dependency. Each of these functions should accept a single positional argument which is the folder path into which they are supposed to be installed.

class ufotest.cli.PluginCommands(name=None, commands=None, **attrs)

Bases: click.core.Group

get_command(ctx, cmd_name)

Given a context and a command name, this returns a Command object if it exists or returns None.

list_commands(ctx)

Returns a list of subcommand names in the order they should appear.

ufotest.config module

Module containing the functions to access the configuration of ufotest.

class ufotest.config.Config

Bases: object

This is a singleton class, which implements the access to the config file.

DESIGN CHOICE

So I feel like I want to explain my reasoning for this class here. This is indeed not my first implementation for config access within this project. Previously I was simply loading the dictionary into global variable within this file and then I could import this global variable and just use the dictionary.

But while working with that implementation I have found two major problems with it:

(1) Reloading: The problem is that the dictionary is loaded into the global variable at exactly the point when it is imported the first time at some other module. This means that there is no way of globally reloading the config to react to file system changes for example. Even if you were to assign the variable with a new dict for example, this change would not translate to other files! The reloaded version would only be present in a single file. With a singleton of course you can simply solve this problem with a “reload” method.

(2) Reacting to change: There is a significant downside to using the config dict directly without some sort of intermediary layer. If you need some functionality say “CONFIG[‘camera’][‘sensor_height’]” then this is the path which is directly implemented like this in the config file structure. This means that if you were to change the structure of the config file you would have to change every occasion in the code, which uses this attribute… This is obviously a bad SOC. With a class you could write methods, which wrap certain behaviour. After a change only this method would have to be changed.

USAGE

In a sense the config singleton instance is the central data structure upon which all the other ufotest functionality relies. This instance needs to be accessible by all other code. This is for several reasons. The main reason is that this class wraps the access to all the values defined in the ufotest config file. Naturally much of the functionality is influenced by the concrete values of these config fields.

Another reason is because the config singleton also contains the references to the essential plugin manager and script manager. The plugin manager is especially important, because it contains the registration of callbacks to specific hooks and the instance itself has to be used to invoke a certain hook.

The config singleton is automatically created whenever this config module is first imported. This is also the point at which the config file is being read. So after this point, all the config values themselves can be accessed. But the plugin and script manager are NOT yet initialized for this to happen the “prepare” method has to be called at the very beginning of the new runtime.

import Config from ufotest.config
# At this point an instance of Config is already loaded and it also contains the values from the actual config
# file

# This does not actually create a new instance but returns the only existing one "Config" is a singleton class
config = Config()
# Only after this line the plugin manager is actually initialized
config.prepare()

# Internally the config instance saves the toml structure as a dict. This is the bad way to access the config
# values since that structure is subject to change.
config.data['ci']['hostname']
# Better use the dedicated wrapper method
config.get_hostname()

# The plugin manager is saved as "pm"
config.pm.do_action('my_custom_action')

# The script manager is saved as "sm"
config.sm.invoke('my_script')

apply_overwrite(overwrite_string: str)

get_archive_path()

get_builds_path() → str

Returns the string absolute path to the “builds” folder within the ufotest installation folder. Simply appends “builds” to the installation path.

get_ci_bitfile_path()

get_ci_branch()

get_ci_repository_name() → str

Returns the string name of the remote repository used for the continuous integration. This name is derived from the repo url, which is the only thing saved in the config file

get_ci_repository_url()

get_ci_script_definitions() → List[dict]

Returns the scripts definitions for the CI repository. This is a list of dicts, where each dict contains information about where to find a certain script within the source repository.

More specifically each of these dicts needs to contain AT LEAST the following fields:

• name: The unique string name by which the script will be identified within ufotest
• author: string defining the author name
• description: A description of the purpose of the script
• relative_path: A relative path providing the location of the script in question relative to the repo root
• class: The string class name of a valid implementation of scripts.AbstractScript which defines the type of script

Returns:list of dicts defining scripts within the version controlled source repo

get_ci_suite()

get_data_or_default(key_query: List[str], default: Any) → Any

get_date_format()

get_documentation_url()

get_email_address()

get_email_password()

get_hostname()

get_path() → str

Returns the path of the ufotest installation folder

get_plugin_folder()

get_port()

get_query(key_query: List[str])

get_repository_url()

get_script_definitions()

get_sensor_height() → int

Returns the sensor height of the currently selected camera in pixels. Derives this value from the camera selection. The config file itself defines multiple different camera profiles. What can be set in the config is which profile is active. So this method has to first get the profile to then return the value from the according profile section.

Returns:the int camera height in pixels

get_sensor_width() → int

Returns the sensor width of the currently selected camera in pixels. Derives this value from the camera selection. The config file itself defines multiple different camera profiles. What can be set in the config is which profile is active. So this method has to first get the profile to then return the value from the according profile section.

Returns:the int camera width in pixels

get_test_folder()

get_test_suites()

get_time_format()

is_prepared() → bool

Returns whether or not the plugin and script manager have been initialized

Returns:Whether or not the plugin and script manager are initialized

items()

keys()

prepare() → None

This method needs to be called to properly initialize both the plugin manager and the script manager! Only after this method was called the config.sm and config.pm values actually hold references to the manager instances.

Returns:void

reload()

Reloads the config values from the config file, replacing the current object state with the values read from the file.

Returns:void

static(name) → str

url(*paths) → str

values()

verbose()

class ufotest.config.Singleton

Bases: type

This is metaclass definition, which implements the singleton pattern. The objective is that whatever class uses this as a metaclass does not work like a traditional class anymore, where upon calling the constructor a NEW instance is returned. This class overwrites the constructor behavior to return the same instance upon calling the the constructor. This makes sure that always just a single instance exists in the runtime!

USAGE

To implement a class as a singleton it simply has to use this class as the metaclass.

class MySingleton(metaclass=Singleton):

    def __init__(self):
        # The constructor still works the same, after all it needs to be called ONCE to create the
        # the first and only instance.
        pass

# All of those actually return the same instance!
a = MySingleton()
b = MySingleton()
c = MySingleton()

print(a is b) # true

ufotest.config.get_builds_path() → str

Returns the string path of the ‘builds’ folder within the ufotest installation folder.

Returns:the string absolute path of the builds folder

ufotest.config.get_config_path()

Returns the path of the config file for ufotest.

ufotest.config.get_path(*sub_paths) → str

Returns the path of the installation folder of the ufotest app.

This installation folder contains for example the following things: - The config file for the project. This is a toml file which manages the global config settings for ufotest - The ‘archive’ folder which saves archives of the test reports of all the executed test runs. - The ‘builds’ folder which saves archives of all the reports for the issued builds of the remote source repo.

Returns:the string absolute path to the installation folder

ufotest.config.load_config()

Loads the configuration from the config file “config.toml” and returns it as a dictionary

ufotest.config.reload_config()

ufotest.install module

A module, which contains the code related to the installation process of the UFO dependencies

ufotest.install.git_clone(path: str, git_url: str, verbose: bool, branch: str = 'master') → Tuple[str, str]

Clones the repository from git_url into the given folder path

The additional arg verbose is a boolean which controls whether the output of this operation is being written to stdout stream.

Parameters:

• path – The path of the folder into which the repository is supposed to be clonedS
• git_url – The string url of the git repository from which to clone
• verbose – Whether or not to print additional output to the stdout stream
• branch – The string name of the branch to be cloned. Default is master branch.

Returns:

A tuple of two values, where the first one is the string name of the repository and the second is the

string of the absolute path of the clones location within the filesystem.

ufotest.install.install_dependencies(verbose=True)

Installs all the system packages, which are listed in the config file.

ufotest.install.install_fastwriter(path: str, verbose: bool = True) → dict

Installs the “fastwriter” repository into the given path. Returns a dict which contains some information about the installation process.

The returned dict contains the following items: - success: boolean value of whether or not the installation succeeded - path: the string path of the file - git: the string URL of the git repository from which it was installed

Parameters:

• path – The string path of the folder into which to install this dependency
• verbose – Whether or not to produce verbose output. DEPRECATED: Uses config field to determine verbosity

Returns:

A dict, which contains information about the installation process.

ufotest.install.install_generic_cmake(path: str, git_url: str, verbose: bool, cmake_args: dict)

Installs the cmake project from the “git_url” into the given folder “path”.

This function first clones the repository which is given by the git url, then it enters this folder locally, creates a build folder and attempts to run a cmake installation process within this folder. The cmake_args can be used to pass additional options to the cmake build process.

ufotest.install.install_ipecamera(path: str, verbose: bool = True) → dict

Installs the “ipecamera” repository into the given path. Returns a dict which contains information about the installation process.

The returned dict contains the following items: - success: boolean value of whether or not the installation succeeded - path: the string path of the file - git: the string URL of the git repository from which it was installed

Parameters:

• path – The string path of the folder into which to install this dependency
• verbose – Whether or not to produce verbose output. DEPRECATED: Uses config field to determine verbosity

Returns:

A dict, which contains information about the installation process.

ufotest.install.install_libuca(path: str, verbose: bool = True) → dict

Installs the libuca repository into the given path. Returns a dict which contains some information about the installation process.

The returned dict contains the following items: - success: boolean value of whether or not the installation succeeded - path: the string path of the file - git: the string URL of the git repository from which it was installed

Parameters:

• path – The string path of the folder into which to install this dependency
• verbose – Whether or not to produce verbose output. DEPRECATED: Uses config field to determine verbosity

Returns:

A dict, which contains information about the installation process.

ufotest.install.install_libufodecode(path: str, verbose: bool = True) → dict

Installs the “libufodecode” repository into the given path. Returns a dict which contains some information about the installation process.

The returned dict contains the following items: - success: boolean value of whether or not the installation succeeded - path: the string path of the file - git: the string URL of the git repository from which it was installed

Parameters:

• path – The string path of the folder into which to install this dependency
• verbose – Whether or not to produce verbose output. DEPRECATED: Uses config field to determine verbosity

Returns:

A dict, which contains information about the installation process.

ufotest.install.install_package(package_name: str, verbose=True)

Installs a system package with the given “package_name”

ufotest.install.install_pcitools(path: str, verbose: bool = True) → dict

Installs the “pcitool” repository into the given path. Returns a dict which contains information about the installation process.

The returned dict contains the following items: - success: boolean value of whether or not the installation succeeded - path: the string path of the file - git: the string URL of the git repository from which it was installed

Parameters:

• path – The string path of the folder into which to install this dependency
• verbose – Whether or not to produce verbose output. DEPRECATED: Uses config field to determine verbosity

Returns:

A dict, which contains information about the installation process.

ufotest.install.install_uca_ufo(path: str, verbose: bool = True) → dict

Installs the uca-ufo repository into the given path. Returns a dict which contains some information about the installation process.

The returned dict contains the following items: - success: boolean value of whether or not the installation succeeded - path: the string path of the file - git: the string URL of the git repository from which it was installed

Parameters:

• path – The string path of the folder into which to install this dependency
• verbose – Whether or not to produce verbose output. DEPRECATED: Uses config field to determine verbosity

Returns:

A dict, which contains information about the installation process.

ufotest.install.install_vivado(path: str, verbose=True)

ufotest.install.mock_install_repository(path: str)

Does not actually perform anything, but still returns a result dict like the other install functions. This result dict contains the following fields: - success: True - path: The path passed as parameter - git: Github URL

Returns:dict

ufotest.logging module

ufotest.scripts module

class ufotest.scripts.AbstractScript(script_definition: Dict[str, Any])

Bases: object

The abstract base class for representing scripts.

The script manager loads knowledge about external scripts based on a dict representation providing important information about that script. But this is only the most convenient human readable representation of the script knowledge. During the loading process, this dict information is converted into a script wrapper object. These have to be specific sub classes of this base class. Each specific implementation of this base class represents a different kind of script. And each type differs in how it is supposed to be handled / invoked. Some scripts may be python modules, bash scripts, php scripts or whatever. Each of those can be supported by creating a subclass which implements the appropriate way to handle an invocation.

Any kind of script wrapper is constructed by passing the script definition dict (the way it was initially described by users) as the only argument to the constructor. This dict has to contain at least the following fields to describe a valid script:

• name: The string identifier by which the script can be invoked from within the ufotest system. Passing this name will be required for the script manager to select the appropriate script.
• author: The string describing the name (and mail address) of the author which has created the script
• path: The ABSOLUTE string path to the actual script file.
• description: A string description of the purpose of the script and potentially other information one should have about it’s behavior

EXPECTED IMPLEMENTATIONS

A subclass is expected to implement the “invoke” method. This method is supposed to handle actual execution of the script. Aside from that there are no hard requirements. The method accepts one argument, which could be anything depending on what is needed for that specific class. And the method should return the outcome of the script execution in some way. If it is a multitude of information preferably as a dict.

A subclass may overwrite the “check_syntax” method. The purpose of this method is to provide additional (optional) functionality for scripts. This method should allow to perform a simple syntax check of the script without having to actually invoke it. This is an optional feature and can be used to verify the functionality of the scripts with an additional test case for example. The method is supposed to return a tuple (bool, str) where the first element is the boolean value of whether the syntax is ok (=True) or has errors (=False). The second element is a string which gives a description of the syntax error if one is present. An empty string should be returned if no error is found. This method has a default implementation which always returns true. Implementation for a custom subclass is optional but encouraged.

If a subclass does indeed implement such a syntax check. The subclass should also override the string value of the “syntax_check_method” class variable. This class variable should contain a short descriptive string which explains how the syntax check is performed for this script type.

check_syntax() → Tuple[bool, str]

This method can be implemented by a subclass to implement syntax checking for custom script types. The method has to return a tuple, where the first element is a boolean which indicates if the syntax is OK (=True) or has errors (=False). The second element is a string which contains the exact syntax error message if an error was found.

invoke(args: Optional[Any] = None) → Any

syntax_check_method = 'Syntax is not being checked for this script type'

class ufotest.scripts.BashScript(script_definition: Dict[str, Any])

Bases: ufotest.scripts.AbstractScript

This class represents a wrapper for storing the information about a bash script. It does not expect the passed script_definition to have any additional fields aside from those basic ones required for all AbstractScripts.

check_syntax() → Tuple[bool, str]

Checks the syntax of the bash script.

IMPLEMENTATION

Luckily syntax checking of bash scripts is really easy. One can simply invoke the default “bash” command with the “-n” option, pass the absolute path of the command and that will perform a syntax check. If the command exits with return code 0 then there is no issue otherwise it will print an error message.

Returns Tuple[bool, str]:  The first element indicates whether or not the syntax is ok and the second element is a string which contains the error message in case there is an error, otherwise an empty string.

invoke(args: Optional[dict] = None) → dict

syntax_check_method = 'bash -n {script_path}'

class ufotest.scripts.MockScript(script_definition: Dict[str, Any])

Bases: ufotest.scripts.AbstractScript

This is a mock implementation of AbstractScript mainly intended for testing purposes. It expects one additional field within the script_definition, aside from those required for all AbstractScript implementations:

• code: This is a string field. The string should contain the python code which preferably evaluates to a single expression. When calling the invoke method of a mock script. This code will be dynamically interpreted with the “eval” builtin and the result of the expression will be returned as the result of the invoke method.

invoke(args: Optional[str] = None) → Any

class ufotest.scripts.ScriptManager(config)

Bases: object

The script manager is responsible for wrapping all interactions with the external camera scripts.

BACKGROUND

The main purpose of the ufotest program is to interface with a camera hardware and perform various tests with it to confirm that it is working properly. HOW to actually interface with this camera is not part of ufotest however. These implementation details are left to the developers of the camera hardware. Ufotest assumes that the interfacing options are wrapped in scripts (any external, callable program) which are provided alongside the actual hardware. These scripts are loaded by ufotest and it is expected that calling them will achieve some sort of higher level function like resetting the camera or etc. Access and interaction with these scripts is managed by the ScriptManager

During the ufotest runtime, usually only one script manager instance is created as part of the “Config” singleton during the start up phase of the program, after the config file was being read.

CONTINUOUS INTEGRATION

Ufotest also provides CI functionality: The source code for the camera firmware is periodically fetched from a remote repository, flashed to the local camera hardware and a test suite is repeated to check for errors with the new firmware version.

Additionally, the external scripts which are used to interact with the camera can also be part of the CI system. The scripts can also be placed into the very same source repository and then they will be fetched for each new build as well. If they are properly registered in the config file, the script manager will load these versions of the most recent build and run the test suite using those!

This is only an option however, if some scripts are not registered for the source repo or for some reason missing in some commit or another, the script manager will use a fallback version instead. These fallback versions are part of the ufotest installation and can be changed through a plugin.

Variables:

• AbstractScript] fallback_scripts (Dict[str,) – A dict, whose keys are the unique string names of the scripts and the values are subclass instances of AbstractScript, which describe the script in question. This value specifically contains all the fallback scripts, which come with the ufotest installation itself. By usual circumstances, this dict should contain ALL the necessary scripts.
• AbstractScript] scripts (Dict[str,) – A dict, whose keys are the unique string names of the scripts and the values are subclass instances of AbstractScript, which describe the script in question. It should contain ALL the necessary scripts as well, because it starts out as a copy of the fallback_scripts dict, but if a more recent build version of a script is available, the corresponding entry of this dict is replaced with that build version.

get(script_name: str, use_fallback: bool = False) → ufotest.scripts.AbstractScript

invoke(script_name: str, args: Optional[Any] = None, use_fallback: bool = False) → Any

This method invokes the script identified by script_name passing the optional args. If the use_fallback flag is set it will attempt to use the fallback version.

load_build_scripts(build_folder_path: str)

load_fallback_scripts()

load_scripts() → None

This method loads all dem scripts.

After this method was called it can be assumed that a reference to all registered scripts has been loaded to the internal values “self.fallback_scripts” and “self.scripts” respectively.

This method first loads the fallback scripts. These are part of the main ufotest code. The scripts which are loaded are defined by the internal “fallback_script_definitions” list. The actual internal dict for the scripts (self.scripts) is then initialized as a copy of these fallback scripts. Then it is attempted to load the script from the latest cloned version of the remote repository (ci repo) and overwrite the self.script entries with those.

Raises:LookupError – If no build folder exists yet with a valid script substitutions Returns:None

most_recent_build_folder() → str

Returns the absolute path to the build folder of the most recent build.

Raises:LookupError – In case there are no builds yet, on other words: If the remote repo has never been cloned before, no scripts can be loaded from it either. Returns:string of absolute path

register_fallback_script(script_definition: Dict[str, Any]) → None

Based on the given *script_definition”, registers the script in question in the internal “fallback_scripts” dict. To do that the script will be converted into the appropriate script wrapper instance.

Parameters:script_definition – The dict which describes the script to be registered. It’s required fields depend on what type (class field) the script is of. Returns:void

register_script(script_definition: Dict[str, Any]) → None

Based on the given *script_definition”, registers the script in question in the internal “scripts” dict. To do that the script will be converted into the appropriate script wrapper instance.

Note that this method will create a warning when attempting to register a script with an identifier for which no fallback script exists.

Parameters:script_definition – The dict which describes the script to be registered. It’s required fields depend on what type (class field) the script is of. Returns:void

class ufotest.scripts.TclScript(script_definition: Dict[str, Any])

Bases: ufotest.scripts.AbstractScript

invoke(args: Optional[Dict[KT, VT]] = None) → dict

ufotest.util module

A module containing various utility functions for usage in other modules.

class ufotest.util.AbstractRichOutput

Bases: abc.ABC

to_dict() → dict

to_html() → str

to_latex() → str

to_markdown() → str

to_string() → str

class ufotest.util.HTMLTemplateMixin

Bases: object

This mixin can be used to provide a default implementation of the “to_html” method for the AbstractRichOutput interface. This implementation uses a jinja template string to render the html version of the object instance.

USAGE

Any class which inherits from this mixin has to define a static class member called “HTML_TEMPLATE” which is supposed to be the string representation of the jinja template to be rendered for the HTML representation. Upon calling the “to_html” method, this template will be passed a context where “self” references to the actual object instance, which allows for all the instance attributes to be used in the template. Within the template string all jinja syntax can be used. In fact, within the template the ufotest jinja environment is loaded, which means that there is even access to the global context variable “config”.

class Custom(HTMLTemplateMixin):

    HTML_TEMPLATE = "<p>{{ self.text }}</p>"

    def __init__(self):
        HTMLTemplateMixin.__init__(self)
        self.text = "Hello World"


c = Custom()
c.to_html() # <p>Hello World</p>

RECOMPILING

Aside from this main functionality, the more important feature is that this mixin also enables the inheriting class to be recompilable from the JSON / dict representation. For that, we first define this use case: Assuming we have saved an objects attributes as a json file and now load it again such that all the attribute values are part of a dict. Now we want to render the HTML template using this dict representation. For this case, the mixin adds it’s own implementation of “to_dict” which is used to convert it into json in the first place. This method saves the template string as a json field. The static method “html_from_dict” can now be used on such a dict to render the html template from it, without needing an actual object instance.

class Custom(HTMLTemplateMixin):

    HTML_TEMPLATE = "<p>{{ self.text }}</p>"

    def __init__(self):
        HTMLTemplateMixin.__init__(self)
        self.text = "Hello World"

    def to_dict(self):
        # It is important that the mixins implementation of to_dict is considered here!
        return {
            **HTMLTemplateMixin.to_dict(self),
            # It is also important that all the dict fields have the same name as the instance attributes
            'text': self.text
        }

c = Custom()
c_dict = c.to_dict()
c_dict['text'] = 'Bye World'
HTMLTemplateMixin.html_from_dict(c_dict) # <p>Bye World</p>

TEMPLATE_FIELD_NAME = '_html_template'

Variables:TEMPLATE_FIELD_NAME – This is a string which contains the key name under which the html template is being saved within the dict representation of the object

get_html_template_string() → str

Returns the jinja html template string of the object instance.

DESIGN CHOICE

Why does this method exists? On first glance it is useless, because it only returns a static value. At any position where this method is used, one could just use the actual value instead. This is correct, but this method exists for possible future changes. It wraps access to the template string, which means that the actual access to this template string can easily be changed. In the future it would be possible to replace the class constant with a object attribute for the template string rather easily if that is desired. More conveniently, having this method allows a subclass to potentially overwrite it with very custom behavior!

Returns str:The template string

classmethod html_from_dict(data: dict) → str

Given a “data” dict, which was created by the “to_dict” method of a class which implements this mixin, this method will use the template string which is saved in this dict and the other fields which represent the original instance attributes to render and return the appropriate html representation.

Raises:KeyError – If the given “data” dict does not actually originate from a class which implements this mixin which is indicated by the fact that it wont contain the necessary field for the template string. Returns str:The html representation according to the template string and the values contained in the dict.

to_dict() → dict

Returns the dict representation which contains the important fields required for the HTMLTemplateMixin functionality.

Returns dict:

to_html() → str

Returns the HTML string representation of the object which is based on the jinja html template.

Returns str:The rendered html string

ufotest.util.cerror(message: str) → None

Outputs the message as an error to the console.

Parameters:message – The message to be printed

ufotest.util.check_install()

ufotest.util.check_path(path: str, is_dir: bool = False)

ufotest.util.check_vivado()

ufotest.util.clean_pci_read_output(output: str)

Cleans the output from the “pci_read” function.

This essentially only means, that all the newlines and the leading whitespaces are being removed.

Parameters:output (str) – The output message of the “pci_read” function Returns:

ufotest.util.cparams(parameters: Dict[str, str]) → None

Outputs a dictionary of parameters as key value pairs to the console

Parameters:parameters – A dict of string keys and string values.

ufotest.util.cprint(message: str) → None

Outputs the message to the console

Parameters:message – The message to be printed

ufotest.util.create_folder(folder_path: str)

ufotest.util.cresult(message: str) → None

Outputs the message as progress to the console

Parameters:message – The message to be printed

ufotest.util.csubtitle(message: str) → None

Outputs the message as a subtitle to the console

Parameters:message – The message to be printed

ufotest.util.ctitle(message: str) → None

Outputs the message as a title to the console

Parameters:message – The message to be printed

ufotest.util.dynamic_import(module_name: str, file_path: str)

ufotest.util.execute_command(command: str, verbose: bool, cwd: Optional[str] = None, foreground=True)

Executes the given system “command”

The “verbose” flag controls whether or not the output of the command is written to stdout or not. With the “cwd” string a path can be passed, which is supposed to be used as the current working directory from which the command is to be executed.

Deprecated:

ufotest.util.execute_script(name: str, prefix: str = '', verbose: bool = False)

ufotest.util.force_aspect(ax, aspect: float = 1)

Sets the aspect ratio of the given matplotlib Axes object to the given float.

Parameters:

• ax – A matplotlib Axes object which describes a plot
• aspect (float) – The aspect ratio which to force on the graph

ufotest.util.format_byte_size(value: int, unit: str = 'B', factor: int = 1024, include_unit=True) → str

Given an integer value, this function formats the number which is assumed to be a size in bytes into a different byte related unit such as kilobytes, megabytes etc.

Parameters:

• value (int) – The actual integer amount of bytes to be converted into another unit
• factor (int) – The factor which seperates two denominations of units. By default this is 1024 which is correct when working with bytes. Should not be changed.
• unit (str) – The string identifier of the unit to be converted to. May only be one of the following strings: B, KB, MB, GB, TB. Default is B (Bytes)
• include_unit (bool) – A boolean flag which determines whether or not the unit string should be included in the final returned string value.

Returns:

The string representation of the formatted value, now (potentially) in the new unit. The numeric value within the string will have two decimals accuracy

ufotest.util.get_build_reports() → List[dict]

Returns a list of all build reports, which are represented as dicts.

The build reports are represented as dicts and not as actual BuildReport instances because they are loaded from memory where they are represented as JSON files. Loading these JSON files will yield dicts. The test reports will be sorted by the time they where created, where the most recent one will be the first element of the list.

Returns:A list of dicts

ufotest.util.get_command_output(command: str, cwd: Optional[str] = None)

Executes the given “command” and returns the output of the command as string

The additional “cwd” option can be used to pass a string path description, which is supposed to be used as the current working directory for the command execution.

Parameters:

• command (str) – The command which is to be executed
• cwd (Optional[str]) – The path string of the current working directory to be assumed for the command execution

ufotest.util.get_folder_size(folder_path: str) → int

Returns the size of an entire recursive directory tree as an integer in bytes. https://www.thepythoncode.com/article/get-directory-size-in-bytes-using-python

Parameters:folder_path (str) – The absolute path to the folder which is to be measured Returns:The integer amount of bytes which all contents of the folder collectively take up

ufotest.util.get_repository_name(repository_url: str) → str

Returns the name of a git repository if the repository_url for it is given.

Parameters:repository_url – the string url of the remote repository Returns:the string name of the repo, which does NOT include the username it is just the name of the top level

folder of that repo.

ufotest.util.get_template(name: str) → jinja2.environment.Template

Returns the jinja2 template with the given file name name within the static templates folder of this project

Parameters:name – The file name of the template to be loaded Returns:The template object for the specified template

ufotest.util.get_test_reports() → List[dict]

Returns a list of all test reports, which are represented as dicts.

The test reports are represented as dicts and not as actual TestReport instances because they are loaded from memory where they are represented as JSON files. Loading these JSON files will yield dicts. The test reports will be sorted by the time they where created, where the most recent one will be the first element of the list.

Returns:A list of dicts

ufotest.util.get_version() → str

Returns the version string for the ufotest project. The version scheme of ufotest loosely follows the technique of Semantic Versioning. Where a minor version change may introduce backward incompatible changes, due to the project still being in active development with many features being subject to change.

The return value of this function is subject to the “get_version” filter hook, which is able to modify the version string after it has been loaded from the file and sanitized.

EXAMPLE

version = get_version() # "1.2.1"

Returns:The version string without any additional characters or whitespaces.

ufotest.util.init_install(verbose=False) → str

Initializes the installation folder for the ufotest app.

Parameters:verbose – whether or not to print additional info to the console. Returns:The string absolute path to the top level installation folder which has been created.

ufotest.util.markdown_to_html(input_path: str, output_path: str, header_lines: List[str] = [])

Converts the markdown file at input_path to html file at output_path

The functionality of this function is provided by the command line tool ‘md-to-html’ provided by the python package of the same name. This function also offers the additional possibility to add custom string lines to the head section of the created html file to potentially insert additional styles.

Parameters:

• input_path – The path to the original markdown file
• output_path – The path to be created for the html file
• header_lines – A list of strings to be included as separate lines into the head of the html

ufotest.util.random_string(length: int, additional_letters: str = ' ')

ufotest.util.run_command(command: str, cwd: Optional[str] = None) → Tuple[int, str]

Runs a terminal command and returns it’s exit code and console output

Parameters:

• command – The command string to execute
• cwd – Optionally a string path, which is to be used as the current working directory for the execution of the command.

Returns:

A tuple, where the first value ist the int exit code of the command execution and the second value is the string of all the output the command produced on the stdout pipe.

ufotest.util.run_script(script_name: str, prefix: str = '') → Tuple[int, str]

Runs the script which is identified by script_name.

Parameters:

• script_name – The string name, which identifies the script within the ufotest application.
• prefix – A string which is prepended to the command string for the execution of the script. This mainly exists to possibly supply the “sudo” prefix to the execution of a script, if it needs it. Default is empty str.

Raises:

FileNotFoundError – If the given script name does not refer to a valid script, which is registered with the ufotest application.

Returns:

A tuple, where the first element is the integer exit code of the script command and the second the str of the output, which the command generated on stdout.

ufotest.util.setup_environment()

Sets up all the environmental variables defined in the config file

ufotest.util.update_install(folder_path: str) → None

Module contents

Top-level package for ufotest.