core

set of functions and classes used across this package and usable for other packages

Data structures

Classes to handle data structure more easily

with open('data-dev/jsondict-test.json', 'r') as fp:
    d =  json.load(fp)
d

d.items()

dict_items([('b', 2), ('c', 3), ('d', 4)])

---

source

JsonDict

def JsonDict(
    p2json:str | Path, # path to the json file to mirror with the dictionary
    dictionary:Optional[dict]=None, # optional dictionary to initialize the JsonDict
):

Dictionary whose current value is mirrored in a json file and can be initated from a json file

JsonDict requires a path to json file at creation. An optional dict can be passed as argument.

Behavior at creation:

• JsonDict(p2json, dict) will create a JsonDict with key-values from dict, and mirrored in p2json
• JsonDict(p2json) will create a JsonDict with empty dictionary and load json content if file exists

Once created, JsonDict instances behave exactly as a dictionary

Create a new dictionary mirrored to a JSON file:

d = {'a': 1, 'b': 2, 'c': 3}
p2json = Path('data-dev/jsondict-test.json')
jsond = JsonDict(p2json, d)
jsond

{'a': 1, 'b': 2, 'c': 3}
dict mirrored in /home/vtec/projects/ec-packages/eccore/nbs-dev/data-dev/jsondict-test.json

Once created, the JsonFile instance behaves exactly like a dictionary, with the added benefit that any change to the dictionary is automatically saved to the JSON file.

jsond['a'], jsond['b'], jsond['c']

(1, 2, 3)

for k, v in jsond.items():
    print(f"key: {k}; value: {v}")

key: a; value: 1
key: b; value: 2
key: c; value: 3

Adding or removing a value from the dictionary works in the same way as for a normal dictionary. But the json file is automatically updated.

jsond['d'] = 4
jsond

{'a': 1, 'b': 2, 'c': 3, 'd': 4}
dict mirrored in /home/vtec/projects/ec-packages/eccore/nbs-dev/data-dev/jsondict-test.json

with open(p2json, 'r') as fp:
    print(fp.read())

{
    "a": 1,
    "b": 2,
    "c": 3,
    "d": 4
}

del jsond['a']
jsond

{'b': 2, 'c': 3, 'd': 4}
dict mirrored in /home/vtec/projects/ec-packages/eccore/nbs-dev/data-dev/jsondict-test.json

with open(p2json, 'r') as fp:
    print(fp.read())

{
    "b": 2,
    "c": 3,
    "d": 4
}

Validation functions

---

source

is_type

def is_type(
    obj:Any, # object whose type to validate
    obj_type:type, # expected type for `obj`
    raise_error:bool=False, # when True, raise a ValueError is `obj` is not of the right type
)->bool: # True when `obj` is of the right type, False otherwise

Validate that obj is of type obj_type. Raise error in the negative when raise_error is True

is_type(obj='this is a string', obj_type=str)

True

is_type(obj=np.ones(shape=(2,2)), obj_type=np.ndarray)

True

Path validation

Functions to ensure path are properly formated and point to a real file or directory.

---

source

validate_path

def validate_path(
    path:str | Path, # path to validate
    path_type:str='file', # type of the target path: `'file'`, `'dir'` or `'any'`
    raise_error:bool=False, # when True, raise a ValueError is path does not point to a real file or directory
)->bool: # True when path is a valid path, False otherwise

Validate that path is a Path or str and points to a real file or directory

path_file = Path('data-dev/jsondict-test.json')
validate_path(path_file)

True

validate_path(path_file, path_type='any')

True

path_dir = Path('../data')
validate_path(path_dir, path_type='dir')

True

validate_path(path_dir, path_type='any')

True

path_error = Path('../data/img/IIIMG_001_512px.jpg')
validate_path(path_error)

False

path_error = Path('../data/img/IIIMG_001_512px.jpg')
try:
    validate_path(path_error, raise_error=True)
except ValueError as e:
    print('Raised an ValueError exception with error message:')
    print(e)

Raised an ValueError exception with error message:
No file at /home/vtec/projects/ec-packages/eccore/nbs-dev/../data/img/IIIMG_001_512px.jpg. Check the path

---

source

safe_path

def safe_path(
    path:str | Path, # path to validate
)->Path: # validated path returned as a pathlib.Path

Return a Path object when given a valid path as a str or a Path, raise error otherwise

fname_with_file = 'data-dev/jsondict-test.json'
safe_path(fname_with_file)

Path('data-dev/jsondict-test.json')

fname_without_file = 'data-dev/no-file-here.json'
try:
    safe_path(fname_without_file)
except ValueError as e:
    print('Raised an ValueError exception with error message:')
    print(e)

Raised an ValueError exception with error message:
No file or directory at /home/vtec/projects/ec-packages/eccore/nbs-dev/data-dev/no-file-here.json. Check the path

Access key files and directories

---

source

get_config_value

def get_config_value(
    section:str, # section in the configparser cfg file
    key:str, # key in the selected section
    path_to_config_file:Optional[Path | str]=None, # path to the cfg file
)->Any: # the value corresponding to section>key>value

Returns the value corresponding to the key-value pair in the configuration file (configparser format)

When no path_to_config_file is provided, the function will try to find the file in: the system’s home, the parent directory of the current directory, and the Google drive directory mounted to the Colab environment.

By defaults (path_to_config_file is None), it is assumed that the configuration file is located in: - the local package config directory (home/.eccore/) - the working directory - the folder above the working directory - the private-accross-accounts directory on google drive.

File names are expected to be either config-api-keys.cfg or config-sample.cfg.

If not, a path to the file (Path or str) must be provided.

The configuration file is expected to be in the format used by the standard module configparser documentation

    [DEFAULT]
    key = value

    [section_name]
    key = value

    [section_name]
    key = value

get_config_value(section="github", key="git_name")

found /home/vtec/projects/ec-packages/eccore/config-api-keys.cfg
Using config file at /home/vtec/projects/ec-packages/eccore/config-api-keys.cfg

'Etienne Charlier'

path2cfg = Path('../config-sample.cfg').resolve()
assert path2cfg.is_file(), f"{path2cfg} is not a file"
print(path2cfg.absolute())

with open(path2cfg, 'r') as fp:
    print(fp.read())

/home/vtec/projects/ec-packages/eccore/config-sample.cfg
[azure]
azure-api-key= dummy_api_key_for_azure

[github]
git_name = not_my_real_github_name
git_email = not_my_real_git_email
github_username = not_my_real_git_username

[kaggle]
kaggle_username = not_my_real_kaggle_name
kaggle_key = dummy_api_key_for_kaggle

[wandb]
api_key = dummy_api_key_for_wandb

value = get_config_value(section='azure', key='azure-api-key', path_to_config_file=path2cfg)
assert value == 'dummy_api_key_for_azure'

Using config file at /home/vtec/projects/ec-packages/eccore/config-sample.cfg

value = get_config_value(section='kaggle', key='kaggle_username', path_to_config_file=path2cfg)
assert value == 'not_my_real_kaggle_name'

Using config file at /home/vtec/projects/ec-packages/eccore/config-sample.cfg

value = get_config_value(section='wandb', key='api_key', path_to_config_file=path2cfg)
assert value == 'dummy_api_key_for_wandb'

Using config file at /home/vtec/projects/ec-packages/eccore/config-sample.cfg

value = get_config_value(section='dummy', key='dummy-user-id')
assert value.startswith('dummy-userID-from')

found /home/vtec/projects/ec-packages/eccore/config-api-keys.cfg
Using config file at /home/vtec/projects/ec-packages/eccore/config-api-keys.cfg

Setup utilities

---

source

CurrentMachine

def CurrentMachine(
    mount_gdrive:bool=True, # True to mount Google Drive if running on Colab
):

Callable class representing the current machine. When called, instance return a dict all attrs:

• os: the operating system running on the machine
• home: path to home on the machine
• is_local, is_colab, is_kaggle: whether the machine is running locally or not
• p2config: path to the config file
• package_root: path to the package root directory

CurrentMachine is a singleton class.

machine = CurrentMachine()
machine()

{'os': 'linux',
 'home': Path('/home/vtec'),
 'is_local': True,
 'is_colab': False,
 'is_kaggle': False,
 'p2config': Path('/home/vtec/.ecutilities/ecutilities.cfg'),
 'package_root': Path('/home/vtec/projects/ec-packages/eccore')}

machine.is_local, machine.is_colab, machine.is_kaggle

(False, False, False)

This machine is not registered a local machine, but is also not running in the cloud. We should register it as a local machine with register_as_local

---

source

CurrentMachine.register_as_local

def register_as_local():

Update the configuration file to register the machine as local machine

Use this method to register the current machine as local machine. Only needs to be used once on a machine. Do not use on cloud VMs

machine.register_as_local()
machine.is_local, machine.is_colab, machine.is_kaggle

(True, False, False)

Technical Note:

The configuration file is located at a standard location, which varies depending on the OS:

• Windows:
  • home is C:\Users\username
  • application data in C:\Users\username\AppData/Local/... or C:\Users\username\AppData\Roaming\... (see StackExchange)
  • application also can be loaded under a dedicated directory under C:\Users\username like C:\Users\username\.conda\...
• Linux:
  • home is /home/username
  • application data in a file or dedicated directory /home/username/ s.a.:
    • file in home directory, e.g. .gitconfig
    • file in an application dedicated directory, e.g. /home/username/.conda/...

ecutilities places the configuration file in a dedicated directory in the home directory: - C:\Users\username\.ecutilities\ecutilities.cfg - /home/username/.ecutilities/ecutilities.cfg

Retrieve the OS:

sys.platform

win32           with Windows
linux           with linux
darwin          with macOs

Accessing the correct path depending on the OS:

Path().home().absolute()

WindowsPath('C:/Users/username') with Windows
Path('/home/username')           with linux

---

source

ProjectFileSystem

def ProjectFileSystem(
    mount_gdrive:bool=True, # True to mount Google Drive if running on Colab
):

Class representing the project file system and key subfolders (data, nbs, src)

Set paths to key directories, according to whether the code is running locally or in the cloud. Give access to path to these key folders and information about the environment.

pfs = ProjectFileSystem()
pfs()

{'os': 'linux',
 'home': Path('/home/vtec'),
 'is_local': True,
 'is_colab': False,
 'is_kaggle': False,
 'p2config': Path('/home/vtec/.ecutilities/ecutilities.cfg'),
 'package_root': Path('/home/vtec/projects/ec-packages/eccore')}

---

source

ProjectFileSystem.create_project_file_system

def create_project_file_system(
    p2project_root, # path to project root, where all subfolder will be located
    overwrite:bool=False, # overwrite current folders if they exist when True (not implemented yet)
):

Create a standard project file system with the following structure:

    project_root
        |--- data   all data files
        |--- nbs    all notebooks for work and experiments
        |--- src    all scripts and code

pfs.create_project_file_system(Path('/home/vtec/projects/ec-packages/eccore'))

/home/vtec/projects/ec-packages/eccore/data
/home/vtec/projects/ec-packages/eccore/nbs
/home/vtec/projects/ec-packages/eccore/src
Created project file system in /home/vtec/projects/ec-packages/eccore

Logging setup and functions

---

source

setup_logging

def setup_logging(
    logfile:Path | None=None
):

Setup logging to console and to file if logfile is not None

---

source

logthis

def logthis(
    *args
)->None:

Logs all elements passed to logs

---

source

monitor_fn

def monitor_fn(
    fn
):

Highlights when function in entered to and exited from

After setting up the logging, it is easy to create log entries:

p2log = pfs.package_root / 'nbs-dev/data-dev/dev.log'
setup_logging(p2log)

Logging to console and to /home/vtec/projects/ec-packages/eccore/nbs-dev/data-dev/dev.log.
Logging setup finished

logging.info('Logging manually as info, only shows in logfile')
logging.warning('Logging manually as warning, shows in logfile and console')

logthis('Using log function, as info, only shows in the log file')

2026-07-24 10:34:27: Logging manually as warning, shows in logfile and console

See logfile content:

if p2log.exists():
    print('Log file content:')
    with open(p2log, 'r') as f:
        print(''.join(f.readlines()))

Log file content:
2026-07-24 10:34:27: Logging manually as info, only shows in logfile
2026-07-24 10:34:27: Logging manually as warning, shows in logfile and console
2026-07-24 10:34:27: Using log function, as info, only shows in the log file

Create a function decorated with @monitor_fn to monitor function calls, i.e. when function in entered and exited.

@monitor_fn
def a_function(a,b):
    """Test functions to add two numbers"""
    return a + b

print(f"function output is {a_function(1,2)}")
print(f"")

function output is 3

if p2log.exists():
    print('Log file content:')
    with open(p2log, 'r') as f:
        print(''.join(f.readlines()))

    p2log.unlink()

Log file content:
2026-07-24 10:34:27: Logging manually as info, only shows in logfile
2026-07-24 10:34:27: Logging manually as warning, shows in logfile and console
2026-07-24 10:34:27: Using log function, as info, only shows in the log file
2026-07-24 10:34:30: Entering `a_function`
2026-07-24 10:34:30: Exiting  `a_function`

File structure exploration

---

source

files_in_tree

def files_in_tree(
    path:str | Path, # path to the directory to scan
    pattern:str | None=None, # pattern (glob style) to match in file name to filter the content
):

List files in directory and its subdiretories, print tree starting from parent directory

p2dir = Path('').resolve()
print(p2dir, '\n')

files = files_in_tree(p2dir)
print(f"List of {len(files)} files when unfiltered")

/home/vtec/projects/ec-packages/eccore/nbs-dev 

eccore
  |--nbs-dev
  |    |--index.ipynb (0)
  |    |--0_02_plotting.ipynb (1)
  |    |--styles.css (2)
  |    |--9_01_dev_utils.ipynb (3)
  |    |--nbdev.yml (4)
  |    |--_quarto.yml (5)
  |    |--0_00_core.ipynb (6)
  |    |--0_01_ipython.ipynb (7)
  |    |--sidebar.yml (8)
  |    |--.last_checked (9)
  |    |--data-dev
  |    |    |--ten-blobs-6-cols-y.npy (10)
  |    |    |--ten-blobs-6-cols-X.npy (11)
  |    |    |--jsondict-test.json (12)
  |    |    |--ten-blobs-6-cols-clusters.npy (13)
List of 14 files when unfiltered

Use pattern to filter the paths to return (using glob syntax)

files = files_in_tree(p2dir, pattern='ipynb')
print(f"List of {len(files)} files when filtered")

eccore
  |--nbs-dev
  |    |--index.ipynb (0)
  |    |--0_02_plotting.ipynb (1)
  |    |--9_01_dev_utils.ipynb (2)
  |    |--0_00_core.ipynb (3)
  |    |--0_01_ipython.ipynb (4)
  |    |--data-dev
List of 5 files when filtered

---

source

path_to_parent_dir

def path_to_parent_dir(
    pattern:str, # pattern to identify the parent directory
    path:str | Path | None=None, # optional path from where to seek for parent directory
)->Path: # path of the parent directory

Climb directory tree up to a directory starting with pattern, and return its path.

• When no directory is found in the tree starting with pattern, return the current directory path.

• It is possible to pass a path as starting path to climb from.

p2dir = path_to_parent_dir('nbs')
assert 'nbs-dev' in p2dir.parts and 'nbs' not in p2dir.parts
p2dir

Path('/home/vtec/projects/ec-packages/eccore/nbs-dev')

# p2dir = path_to_parent_dir('nbs', Path('../nbs/sandbox.ipynb').resolve())
# assert 'nbs' in p2dir.parts and 'nbs-dev' not in p2dir.parts
# p2dir

p2dir = path_to_parent_dir('not-in-tree').resolve()
assert p2dir == Path().absolute()

p2project_root = path_to_parent_dir('eccore')
assert 'eccore' in p2project_root.parts and 'nbs' not in p2project_root.parts
p2project_root

Path('/home/vtec/projects/ec-packages/eccore')