Dynamically import python code Component
Introduction
In python, calling an existing code component is easy with import {$module} options, but it becomes a challenge while the new code components are introduced and code that is importing the other codes needs modification. The impacts are not limited to existing code enhancement but it introduces lots of testing too. What if we can import the new code component dynamically in existing code without impacting existing code components? Recently I have faced similar challenges and following describes how I have solved the problem.
Solution Description
On cloud or on-premise, the applications are either triggered by Event or called by scheduler on a defined time. The solution that I have chosen is independent of the platform. For the purpose of the discussion, I have taken a sample scenario of processing differently delimited files — comma delimited(‘,’) and table delimited(‘\t’). Following is high level process flow diagram which shows, how the solution will work. I have referred the diagram through out the discussion as required.
During the technical design, I have created a python package( 4 in above diagram), which shall contain only the new/dynamically added python code components as a separate *.py file.
The Main calling code ( No 3 in above diagram) are either triggered by event or called by scheduler ( №1 in above diagram) and the event or scheduler shall pass config files ( No 2 in above diagram) to the Main calling code.
How it works
For each *.py file a separate configuration file( No 2 in above diagram) is created and passed as runtime parameter. Following is a sample config file:
[configuration_items]
package_name=callableComponent
application_name=readcsv.py
src_key=csv_valIn config file “package_name” and “application_name” are python package name and module name. For the purpose of this discussion I have added “src_key”. The field is used to proof the concept that parametrs can be passed in standard way. The code is uploaded in github for reference. and link is shared at the end.
The main code is “mainProcessor.py”. It is available in Github.
First the code ingest the configuration file which is passed during run time. Following describes the same.
argList = sys.argv
srcConfig=argList[1]
configfile = os.path.join(thisfolder, os.path.join('dynamicimport',srcConfig))Following is the code snippet which reads the configuration file.
print(f"config file path is:{configfile}")
config = ConfigParser(interpolation=None)
config.read(configfile)
filename = config.get('configuration_items', 'application_name')
srckey = config.get('configuration_items', 'src_key')
packagename=config.get('configuration_items', 'package_name')
modulename=filename.split('.')[0]Once having the details of package and *.py code, next “mainProcessor.py” import the module dynamically and then call the method written inside the module. The module is the intended code component, in this case it is csv file parser or tab parser. Codes are in github.
finalpath=packagename+"."+modulename
module = importlib.import_module(finalpath, packagename)In above code ‘importlib.import_module() ‘ does the trick.
Once the module is available then call methods within the method. Following is the code snippet.
The code components can be referred in github link provided below.
module.parser(srckey)One thing to remember , entry point method in any code component(module) shall always be named same, in this case I have named it as “parser”. “srckey” is the parameter which is retrieved from configuration file and passed to the module . The method can return values for further processing.
Notes
The same approch can be taken in case the python code component has a class. However the name of the class should be same in all dynamically importable code components .
One thing to remember. if the new code component is not part of the different package then “__import__($name of the module”) shall serve the purpose. However it is always a good practice to separate the code components in different packages based on its use.
