bsb.storage package¶
Subpackages¶
Submodules¶
bsb.storage.interfaces module¶
- class bsb.storage.interfaces.ConnectivityIterator(cs: ConnectivitySet, direction, lchunks=None, gchunks=None, scoped=True)[source]¶
Bases:
object- Parameters:
cs (ConnectivitySet)
- chunk_iter()[source]¶
Iterate over the connection data chunk by chunk.
- Returns:
The presynaptic chunk, presynaptic locations, postsynaptic chunk, and postsynaptic locations.
- Return type:
Tuple[Chunk, numpy.ndarray, Chunk, numpy.ndarray]
- class bsb.storage.interfaces.ConnectivitySet(engine)[source]¶
Bases:
InterfaceStores the connections between 2 types of cell as
localandgloballocations. A location is a cell id, referring to the n-th cell in the chunk, a branch id, and a point id, to specify the location on the morphology. Local locations refer to cells on this chunk, while global locations can come from any chunk and is associated to a certain chunk id as well.Locations are either placement-context or chunk dependent: You may form connections between the n-th cells of a placement set (using
connect()), or of the n-th cells of 2 chunks (usingchunk_connect()).A cell has both incoming and outgoing connections; when speaking of incoming connections, the local locations are the postsynaptic cells, and when speaking of outgoing connections they are the presynaptic cells. Vice versa for the global connections.
- abstractmethod chunk_connect(src_chunk, dst_chunk, src_locs, dst_locs)[source]¶
Must connect the
src_locsto thedest_locs, interpreting the cell ids (first column of the locs) as the cell rank in the chunk.
- abstractmethod connect(pre_set, post_set, src_locs, dest_locs)[source]¶
Must connect the
src_locsto thedest_locs, interpreting the cell ids (first column of the locs) as the cell rank in the placement set.
- abstractmethod static exists(engine, tag)[source]¶
Must check the existence of the connectivity set.
- abstractmethod flat_iter_connections(direction=None, local_=None, global_=None)[source]¶
Must iterate over the connectivity data, yielding the direction, local chunk, global chunk, and data:
for dir, lchunk, gchunk, data in self.flat_iter_connections(): print(f"Flat {dir} block between {lchunk} and {gchunk}")
If a keyword argument is given, that axis is not iterated over, and the value is fixed in each iteration.
- abstractmethod get_global_chunks(direction, local_)[source]¶
Must list all the global chunks that contain data coming from a
localchunk in the givendirection
- abstractmethod get_local_chunks(direction)[source]¶
Must list all the local chunks that contain data in the given
direction("inc"or"out").
- abstractmethod classmethod get_tags(engine)[source]¶
Must return the tags of all existing connectivity sets.
- Parameters:
engine – Storage engine to inspect.
- abstractmethod load_block_connections(direction, local_, global_)[source]¶
Must load the connections from
directionperspective betweenlocal_andglobal_.- Returns:
The local and global connections locations
- Return type:
Tuple[numpy.ndarray, numpy.ndarray]
- load_connections()[source]¶
Loads connections as a
CSIterator.- Returns:
A connectivity set iterator, that will load data
- abstractmethod load_local_connections(direction, local_)[source]¶
Must load all the connections from
directionperspective inlocal_.- Returns:
The local connection locations, a vector of the global connection chunks (1 chunk id per connection), and the global connections locations. To identify a cell in the global connections, use the corresponding chunk id from the second return value.
- Return type:
Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]
- abstractmethod nested_iter_connections(direction=None, local_=None, global_=None)[source]¶
Must iterate over the connectivity data, leaving room for the end-user to set up nested for loops:
for dir, itr in self.nested_iter_connections(): for lchunk, itr in itr: for gchunk, data in itr: print(f"Nested {dir} block between {lchunk} and {gchunk}")
If a keyword argument is given, that axis is not iterated over, and the amount of nested loops is reduced.
- class bsb.storage.interfaces.Engine(root, comm)[source]¶
Bases:
InterfaceEngines perform the transactions that come from the storage object, and read/write data in a specific format. They can perform collective or individual actions.
Warning
Collective actions can only be performed from all nodes, or deadlocks occur. This means in particular that they may not be called from component code.
- property comm¶
The communicator in charge of collective operations.
- property format¶
Name of the type of engine.
Automatically set through the plugin system.
- abstractmethod get_chunk_stats()[source]¶
readonly Must return a dictionary with all chunk statistics.
- classmethod peek_exists(root)[source]¶
Must peek at the existence of the given root, without instantiating anything.
- read_only()[source]¶
A context manager that enters the engine into readonly mode.
In readonly mode the engine does not perform any locking, write-operations or network synchronization, and errors out if a write operation is attempted.
- abstractmethod static recognizes(root, comm)[source]¶
Must return whether the given root argument is recognized as a valid storage object.
- Parameters:
root – The unique identifier for the storage
comm (mpi4py.MPI.Comm) – MPI communicator that shares control over the Storage.
- property root¶
The unique identifier for the storage.
Usually pathlike, but can be anything.
- abstract property root_slug¶
Must return a pathlike unique identifier for the root of the storage object.
- abstract property versions¶
Must return a dictionary containing the version of the engine package, and bsb package, used to last write to this storage object.
- class bsb.storage.interfaces.FileStore(engine)[source]¶
Bases:
InterfaceInterface for the storage and retrieval of files essential to the network description.
- get(id) StoredFile[source]¶
Return a StoredFile wrapper.
- Return type:
- abstractmethod get_encoding(id)[source]¶
Must return the encoding of the file with the given id, or None if it is unspecified binary data.
- abstractmethod get_mtime(id)[source]¶
Must return the last modified timestamp of file with the given id.
- abstractmethod load(id)[source]¶
Load the content of an object in the file store.
- Parameters:
id (str) – id of the content to be loaded.
- Returns:
The content of the stored object
- Return type:
- Raises:
FileNotFoundError – The given id doesn’t exist in the file store.
- abstractmethod load_active_config()[source]¶
Load the active configuration stored in the file store.
- Returns:
The active configuration
- Return type:
- Raises:
Exception – When there’s no active configuration in the file store.
- abstractmethod remove(id)[source]¶
Remove the content of an object in the file store.
- Parameters:
id (str) – id of the content to be removed.
- Raises:
FileNotFoundError – The given id doesn’t exist in the file store.
- abstractmethod store(content, id=None, meta=None, encoding=None, overwrite=False)[source]¶
Store content in the file store. Should also store the current timestamp as mtime meta.
- abstractmethod store_active_config(config)[source]¶
Store configuration in the file store and mark it as the active configuration of the stored network.
- Parameters:
config (
Configuration) – Configuration to be stored- Returns:
The id the config was stored under
- Return type:
- class bsb.storage.interfaces.GeneratedMorphology(name, generated, meta)[source]¶
Bases:
StoredMorphology
- class bsb.storage.interfaces.MorphologyRepository(engine)[source]¶
Bases:
Interface- abstractmethod all()[source]¶
Fetch all the stored morphologies.
- Returns:
List of the stored morphologies.
- Return type:
- abstractmethod get_all_meta()[source]¶
Get the metadata of all stored morphologies.
- Returns:
Metadata dictionary
- Return type:
- abstractmethod load(name)[source]¶
Load a stored morphology as a constructed morphology object.
- Parameters:
name (str) – Key of the stored morphology.
- Returns:
A morphology
- Return type:
- abstractmethod preload(name)[source]¶
Load a stored morphology as a morphology loader.
- Parameters:
name (str) – Key of the stored morphology.
- Returns:
The stored morphology
- Return type:
- abstractmethod save(name, morphology, overwrite=False)[source]¶
Store a morphology.
- Parameters:
name (str) – Key to store the morphology under.
morphology (bsb.morphologies.Morphology) – Morphology to store
overwrite (bool) – Overwrite any stored morphology that already exists under that name
- Returns:
The stored morphology
- Return type:
- abstractmethod select(*selectors)[source]¶
Select stored morphologies.
- Parameters:
selectors (list[bsb.morphologies.selector.MorphologySelector]) – Any number of morphology selectors.
- Returns:
All stored morphologies that match at least one selector.
- Return type:
- class bsb.storage.interfaces.PlacementSet(engine, cell_type)[source]¶
Bases:
InterfaceInterface for the storage of placement data of a cell type.
- abstractmethod append_additional(name, chunk, data)[source]¶
Append arbitrary user data to the placement set. The length of the data must match that of the placement set, and must be storable by the engine.
- Parameters:
name
chunk (Chunk) – The chunk to store data in.
data (numpy.ndarray) – Arbitrary user data. You decide ❤️
- abstractmethod append_data(chunk, positions=None, morphologies=None, rotations=None, additional=None, count=None)[source]¶
Append data to the placement set. If any of
positions,morphologies, orrotationsis given, the arguments to its left must also be given (e.g. passing morphologies, but no positions, is not allowed, passing just positions is allowed)- Parameters:
chunk (Chunk) – The chunk to store data in.
positions (numpy.ndarray) – Cell positions
rotations (RotationSet) – Cell rotations
morphologies (MorphologySet) – Cell morphologies
additional (dict[str, numpy.ndarray]) – Additional datasets with 1 value per cell, will be stored under its key in the dictionary
count (int) – Amount of entities to place. Excludes the use of any positional, rotational or morphological data.
- abstractmethod clear(chunks=None)[source]¶
Clear (some chunks of) the placement set.
- Parameters:
chunks (list[bsb.storage._chunks.Chunk]) – If given, the specific chunks to clear.
- abstractmethod classmethod create(engine, cell_type)[source]¶
Create a placement set.
- Parameters:
engine (bsb.storage.interfaces.Engine) – The engine that governs this PlacementSet.
cell_type (bsb.cell_types.CellType) – The cell type whose data is stored in the placement set.
- Returns:
A placement set
- Return type:
- abstractmethod static exists(engine, cell_type)[source]¶
Check existence of a placement set.
- Parameters:
engine (bsb.storage.interfaces.Engine) – The engine that governs the existence check.
cell_type (bsb.cell_types.CellType) – The cell type to look for.
- Returns:
Whether the placement set exists.
- Return type:
- abstractmethod get_all_chunks()[source]¶
Get all the chunks that exist in the placement set.
- Returns:
List of existing chunks.
- Return type:
- abstractmethod get_label_mask(labels=None)[source]¶
Should return a mask that fits the placement set for the cells with given labels. If labels are not provided, will filter non labelled cells.
- abstractmethod get_labelled(labels=None)[source]¶
Should return the ids of the cells labelled with given labels. If labels are not provided, will filter non labelled cells.
- Parameters:
- Return type:
- load_box_tree(morpho_cache=None)[source]¶
Load boxes, and form an RTree with them, for fast spatial lookup of rhomboid intersection.
- Parameters:
morpho_cache – See
load_boxes().- Returns:
A boxtree
- Return type:
- load_boxes(morpho_cache=None)[source]¶
Load the cells as axis aligned bounding box rhomboids matching the extension, orientation and position in space. This function loads morphologies, unless a morpho_cache is given, then that is used.
- Parameters:
morpho_cache (MorphologySet) – If you’ve previously loaded morphologies with soft or hard caching enabled, you can pass the resulting morphology set here to reuse it. If afterwards you need the morphology set, you best call
load_morphologies()first and reuse it here.- Returns:
An iterator with 6 coordinates per cell: 3 min and 3 max coords, the bounding box of that cell’s translated and rotated morphology.
- Return type:
- Raises:
DatasetNotFoundError if no morphologies are found.
- abstractmethod load_morphologies(allow_empty=False)[source]¶
Return a
MorphologySetassociated to the cells. Raises an error if there is no morphology data, unless allow_empty=True.- Parameters:
allow_empty (bool) – Silence missing morphology data error, and return an empty morphology set.
- Returns:
Set of morphologies
- Return type:
- abstractmethod load_positions()[source]¶
Return a dataset of cell positions.
- Returns:
An (Nx3) dataset of positions.
- Return type:
- abstractmethod load_rotations()[source]¶
Load the rotation data of the placement set.
- Returns:
A rotation set.
- Return type:
- classmethod require(engine, cell_type)[source]¶
Return and create a placement set, if it didn’t exist before.
The default implementation uses the
exists()andcreate()methods.- Parameters:
engine (bsb.storage.interfaces.Engine) – The engine that governs this PlacementSet.
cell_type (bsb.cell_types.CellType) – The cell type whose data is stored in the placement set.
- Returns:
A placement set
- Return type:
- abstractmethod set_chunk_filter(chunks)[source]¶
Should limit the scope of the placement set to the given chunks.
- Parameters:
chunks (list[bsb.storage._chunks.Chunk]) – List of chunks
- abstractmethod set_label_filter(labels)[source]¶
Should limit the scope of the placement set to the given labels.
- abstractmethod set_morphology_label_filter(morphology_labels)[source]¶
Should limit the scope of the placement set to the given sub-cellular labels. The morphologies returned by
load_morphologies()should return a filtered form of themselves ifas_filtered()is called on them.
- class bsb.storage.interfaces.StorageNode(*args, _parent=None, _key=None, **kwargs)[source]¶
Bases:
object- engine¶
Base implementation of all the different configuration attributes.
Call the factory function
attr()instead.
- get_node_name()¶
Module contents¶
This module imports all supported storage engines, objects that read and write data, which
are present as subfolders of the engine folder, and provides them transparently to the
user, as a part of the Storage factory class. The module scans
the storage.interfaces module for any class that inherits from Interface to collect all Feature Interfaces and then scans the
storage.engines.* submodules for any class that provides an implementation of those
features.
These features, because they all follow the same interface can then be passed on to consumers and can be used independent of the underlying storage engine, which is the end goal of this module.
- bsb.storage._chunks.chunklist(chunks) list[Chunk][source]¶
Convert an iterable of chunk like objects to a sorted unique chunklist.
- class bsb.storage._chunks.Chunk(chunk, chunk_size)[source]¶
Chunk identifier, consisting of chunk coordinates and size.
- class bsb.storage.NotSupported(operation)[source]¶
Bases:
objectUtility class that throws a
NotSupportederror when it is used.This is the default “implementation” of every storage feature that isn’t provided by an engine.
- class bsb.storage.Storage(engine, root, comm=None, main=0, missing_ok=True)[source]¶
Bases:
objectFactory class that produces all the features and shims the functionality of the underlying engine.
Create a Storage provider based on a specific engine uniquely identified by the root object.
- Parameters:
engine (str) – The name of the storage engine.
root (object) – An object that uniquely describes the storage, such as a filename or path. The value to be provided depends on the engine. For the hdf5 engine the filename has to be provided.
comm (mpi4py.MPI.Comm) – MPI communicator that shares control over this Storage.
main – Rank of the MPI process that executes single-node tasks.
- create()[source]¶
Create the minimal requirements at the root for other features to function and for the existence check to pass.
- property files¶
- property format¶
- get_connectivity_set(tag)[source]¶
Get a connection set.
- Parameters:
tag (str) – Connection tag
- Returns:
~bsb.storage.interfaces.ConnectivitySet
- get_connectivity_sets()[source]¶
Return a ConnectivitySet for the given type.
- Parameters:
type (CellType) – Specific cell type.
- Returns:
~bsb.storage.interfaces.ConnectivitySet
- get_placement_set(type, chunks=None, labels=None, morphology_labels=None)[source]¶
Return a PlacementSet for the given type.
- property morphologies¶
- property preexisted¶
- remove()[source]¶
Remove the storage and all data contained within.
This is an irreversible destructive action!
- require_connectivity_set(tag, pre=None, post=None)[source]¶
Get a connection set.
- Parameters:
tag (str) – Connection tag
- Returns:
~bsb.storage.interfaces.ConnectivitySet
- require_placement_set(cell_type)[source]¶
Get a placement set.
- Parameters:
cell_type (CellType) – Connection cell_type
- Returns:
~bsb.storage.interfaces.PlacementSet
- property root¶
- property root_slug¶
- bsb.storage.create_engine(name, root, comm)[source]¶
Create an engine from the engine’s Engine interface.
- Parameters:
name (str) – The name of the engine to create.
root (object) – An object that uniquely describes the storage, such as a filename or path. The value to be provided depends on the engine. For the hdf5 engine the filename has to be provided.
comm (bsb.services.mpi.MPIService) – MPI communicator that shares control over the Engine interface.
- bsb.storage.open_storage(root, comm=None)[source]¶
Load a Storage object from its root.
- Parameters:
root – Root (usually path) pointing to the storage object.
comm (mpi4py.MPI.Comm) – MPI communicator that shares control over the Storage.
- Returns:
A network scaffold
- Return type:
- class bsb.storage._files.CodeDependencyNode(*args, _parent=None, _key=None, **kwargs)[source]¶
Allow the loading of external code during network loading.
- file: FileDependency¶
- get_node_name()¶
- class bsb.storage._files.FileDependency(source: str | os.PathLike, file_store: FileStore = None, ext: str = None, cache=True)[source]¶
- Parameters:
source (str | os.PathLike)
file_store (FileStore)
ext (str)
- property uri¶
- class bsb.storage._files.FileDependencyNode(*args, _parent=None, _key=None, **kwargs)[source]¶
- file: FileDependency¶
Base implementation of all the different configuration attributes.
Call the factory function
attr()instead.
- get_node_name()¶
- class bsb.storage._files.FileScheme[source]¶
- find(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- get_content(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- get_local_path(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- get_meta(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- provide_stream(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- should_update(file: FileDependency, stored_file)[source]¶
- Parameters:
file (FileDependency)
- class bsb.storage._files.MorphologyDependencyNode(*args, _parent=None, _key=None, **kwargs)[source]¶
Configuration dependency node to load morphology files.
The content of these files will be stored in bsb.morphologies.Morphology instances.
- get_morphology_name()[source]¶
Returns morphology name provided by the user or extract it from its file name.
- Returns:
Morphology name
- Return type:
- get_node_name()¶
- load_object(parser=None, save=True) Morphology[source]¶
- Return type:
- name: str¶
Base implementation of all the different configuration attributes.
Call the factory function
attr()instead.
- parser: MorphologyParser¶
Name associated to the morphology.
If not provided, the program will use the name of the file in which the morphology is stored.
- pipeline: cfglist[MorphologyOperation]¶
- queue(pool)[source]¶
Add the loading of the current morphology to a job queue.
- Parameters:
pool (bsb.services.pool.JobPool) – Queue of jobs.
- store_object(morpho, hash_)[source]¶
Save a morphology into the circuit file under the name of this instance morphology.
- Parameters:
hash (str) – Hash key to store as metadata with the morphology
morpho (bsb.morphologies.Morphology) – Morphology to store
- class bsb.storage._files.MorphologyOperation(*args, _parent=None, _key=None, **kwargs)[source]¶
- func: MorphologyOperationCallable¶
Base implementation of all the different configuration attributes.
Call the factory function
attr()instead.
- get_node_name()¶
- class bsb.storage._files.NeuroMorphoScheme[source]¶
-
- get_meta(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- get_nm_meta(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- resolve_uri(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- class bsb.storage._files.NrrdDependencyNode(*args, _parent=None, _key=None, **kwargs)[source]¶
Configuration dependency node to load NRRD files.
- get_node_name()¶
- class bsb.storage._files.Operation(*args, _parent=None, _key=None, **kwargs)[source]¶
- func: OperationCallable¶
Base implementation of all the different configuration attributes.
Call the factory function
attr()instead.
- get_node_name()¶
- class bsb.storage._files.UriScheme[source]¶
- abstractmethod find(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- abstractmethod get_content(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- abstractmethod get_local_path(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- abstractmethod get_meta(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- abstractmethod should_update(file: FileDependency, stored_file)[source]¶
- Parameters:
file (FileDependency)
- class bsb.storage._files.UrlScheme[source]¶
-
- find(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- get_content(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- get_local_path(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- get_meta(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- resolve_uri(file: FileDependency)[source]¶
- Parameters:
file (FileDependency)
- should_update(file: FileDependency, stored_file)[source]¶
- Parameters:
file (FileDependency)