Unable to Locate Credentials Aws Hooks
Am Trying to Create a Simple Dag Using Airflow. Hooks. S3Hook Orchestrates 2 Tasks the First Prints a Simple String on Bash, the Next Is Uploading a Csv File...
am trying to create a simple DAG using airflow.hooks.S3Hook orchestrates 2 tasks the first prints a simple string on bash, the next is uploading a CSV file to AWS s3 bucket. so I get this error:
unable to locate credentials
It's a credential error related to aws_access_key_id and aws_secret_access_key. I know that I can solve it using boto3, but I need it with airflow.hooks
from airflow import DAG
from airflow.hooks.S3_hook import S3Hook
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago
default_arguments = {'owner': 'airflow', 'start_date': days_ago(1)}
def upload_file_to_s3_bucket(filename, key, bucket_name, region_name):
hook = S3Hook(aws_conn_id='aws_default')
hook.create_bucket(bucket_name, region_name)
hook.load_file(filename, key, bucket_name)
with DAG('upload_to_aws',
schedule_interval='@daily',
catchup=False,
default_args=default_arguments
) as dag:
bash_task = BashOperator(task_id='bash_task',
bash_command='echo $TODAY',
env={'TODAY': '2020-11-16'})
python_task = PythonOperator(task_id='py_task',
python_callable=upload_file_to_s3_bucket,
op_kwargs={'filename': '*******.csv',
'key': 'my_s3_reasult.csv',
'bucket_name': 'tutobucket',
'region_name': 'us-east-1'}
)
bash_task >> python_task
2 Answers
You specified an aws_conn_id within the S3Hook. This connection needs to be configured, for example via the UI, see Managing Connections:
Airflow needs to know how to connect to your environment. Information such as hostname, port, login and passwords to other systems and services is handled in the Admin->Connections section of the UI. The pipeline code you will author will reference the ‘conn_id’ of the Connection objects.
There is also a dedicated description for an AWS connection:
Configuring the Connection
- Login (optional) - Specify the AWS access key ID.
- Password (optional) - Specify the AWS secret access key.
I solved it, after digging a bit I was able to make it work.
Under these fields I specified:
Conn Id: aws_default
Conn Type: Amazon Web Services
Extra:
{"aws_access_key_id":"aws_access_key_id",
"secret_access_key": "aws_secret_access_key"}
Marcus Vance is a cybersecurity auditor and technology writer dedicated to educating the public about online safety, data privacy regulations, enterprise security, and emerging cyber threats.
