Config manager
execution.core.config_manager
¶
Job Configuration Manager Module
This module provides centralized configuration management for ETL jobs. It handles parsing command line arguments, creating job configurations, and determining job types based on file paths.
JobConfigManager
¶
Manages job configuration and argument parsing for ETL jobs.
This class handles the parsing of command line arguments using AWS Glue's getResolvedOptions utility and creates JobConfiguration objects with proper validation and type detection.
Attributes:
| Name | Type | Description |
|---|---|---|
REQUIRED_ARGS |
List of required command line arguments |
|
args |
Dict[str, Any]
|
Dictionary containing parsed job arguments |
REQUIRED_ARGS = ['JOB_NAME', 's3_prefix', 'POWERTOOLS_LOG_LEVEL']
class-attribute
instance-attribute
¶
args: Dict[str, Any]
instance-attribute
¶
__init__()
¶
Initialize the JobConfigManager.
Sets up logging and initializes the args dictionary for storing parsed command line arguments.
parse_job_arguments() -> Dict
¶
Parse required job arguments from command line using AWS Glue utilities.
Returns:
| Type | Description |
|---|---|
Dict
|
Dictionary containing parsed job arguments |
Raises:
| Type | Description |
|---|---|
ValueError
|
If required arguments cannot be parsed |
create_job_config(args: Dict) -> JobConfiguration
¶
Create a JobConfiguration object from parsed arguments.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args
|
Dict
|
Dictionary containing parsed job arguments |
required |
Returns:
| Type | Description |
|---|---|
JobConfiguration
|
JobConfiguration object with validated parameters |
Raises:
| Type | Description |
|---|---|
MissingConfigurationError
|
If required parameters are missing |
determine_job_type(file_path: str) -> JobType
¶
Determine the job type based on the file path prefix.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path
|
str
|
S3 file path to analyze |
required |
Returns:
| Type | Description |
|---|---|
JobType
|
JobType enum value corresponding to the file path |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the file path doesn't match any known job type patterns |