Latest version
Released:
Provider package apache-airflow-providers-redis for Apache Airflow
Project description
Package apache-airflow-providers-redis
RedisPubSubSensor (channels, redisconnid,.args,.kwargs) source ¶ Bases: airflow.sensors.basesensoroperator.BaseSensorOperator Redis sensor for reading a message from pub sub channels. Echo ' airflow ALL=(ALL) NOPASSWD: ALL ' /etc/sudoers # install Airflow 1.8.0rc4 (latest as of 2017-02-18) pip install ' apache-airflows3, postgres, rabbitmq, slack, redis, crypto, celery, async ' # create a log folder for airflow: mkdir /var/log/airflow: chown airflow /var/log/airflow # create a persistent varable for AIRFLOWHOME across. Redis is an open source key-value store that functions as a data structure server.
Redis sentinel setup is done. What all changes i have to do to point my celery backend to redis-sentinel? I am using redis as a broker for celery executor. Redis celery airflow master-slave sentinel. Introduction This post uses Redis and celery to scale-out airflow. Redis is a simple caching server and scales out quite well. It can be made resilient by deploying it as a cluster. In my previous post, the airflow scale-out was done using celery with rabbitmq as the message broker. On the whole, I found the idea of maintaining a rabbitmq a bit fiddly unless you happen to be an expert in.
Release: 1.0.1
Provider package
This is a provider package for redis provider. All classes for this provider packageare in airflow.providers.redis python package.
Airflow Redis Version
You can find package information and changelog for the providerin the documentation.
Installation
NOTE!
On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolverdoes not yet work with Apache Airflow and might lead to errors in installation - depends on your choiceof extras. In order to install Airflow you need to either downgrade pip to version 20.2.4pip install --upgradepip20.2.4 or, in case you use Pip 20.3, you need to add option--use-deprecatedlegacy-resolver to your pip install command.
You can install this package on top of an existing airflow 2.* installation viapip install apache-airflow-providers-redis
PIP requirements
| PIP package | Version required |
|---|---|
| redis | ~=3.2 |
Release historyRelease notifications | RSS feed
1.0.1
1.0.1rc1 pre-release
1.0.0
1.0.0rc1 pre-release
1.0.0b2 pre-release
1.0.0b1 pre-release
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
| Filename, size | File type | Python version | Upload date | Hashes |
|---|---|---|---|---|
| Filename, size apache_airflow_providers_redis-1.0.1-py3-none-any.whl (15.7 kB) | File type Wheel | Python version py3 | Upload date | Hashes |
| Filename, size apache-airflow-providers-redis-1.0.1.tar.gz (11.1 kB) | File type Source | Python version None | Upload date | Hashes |
Hashes for apache_airflow_providers_redis-1.0.1-py3-none-any.whl
| Algorithm | Hash digest |
|---|---|
| SHA256 | 684237770ab7c0b27fcc4742a74f3103f4004ac6dfd4a929f2fcd43ca8721c06 |
| MD5 | 916db08c75e7b4bba28ccb3308bea090 |
| BLAKE2-256 | 822b2ac0782cc1dc224d92767fb92458755d7bf384c552dd218d1af4ab19d262 |
Hashes for apache-airflow-providers-redis-1.0.1.tar.gz
| Algorithm | Hash digest |
|---|---|
| SHA256 | c74517032682961ed37f0deb97b6dd0b5e7967a3939151ef6cb644a57aa12f22 |
| MD5 | 1bff6f70f7d24bb79a9b17ca50f4b75f |
| BLAKE2-256 | 5c111d65bebec7ebc1d4b95201e4b7ac26d547c93e07684a18554aea6cbb6f0f |
This is going to be a quick post on Airflow. We realized that in one of our environments, Airflow scheduler picks up old task instances that were already a success (whether marked as success or completed successfully). You can verify this is actually your issue by ssh into your Airflow workers, and run:
ps -ef | grep airflow
And check the DAG Run IDs: most of them are for old runs.
This happens when Celery's Backend, in our case Redis, has old keys (or duplicate keys) of task runs. So the solution would be to clear Celery queue. And here are the steps to do it when Celery runs on Redis:
1- Stop Airflow Scheduler:
sudo initctl status airflow-scheduler
sudo initctl stop airflow-scheduler
2- Stop webserver:
sudo initctl status airflow-webserver
sudo initctl stop airflow-webserver
3- Stop Celery Flower:
cd /var/lib/airflow/bin
./airflow.sh flower status
./airflow.sh flower stop
4- Stop workers:
cd /var/lib/airflow/bin
./airflow.sh worker status
./airflow.sh worker stop
Now ssh into the server where Redis is running and type 'redis-cli' and press enter to get into Redis CLI. Follow steps below to flush Redis DB:
- INFO keyspace — List keyspaces
a. You should get only 1 result back
- SELECT 0 — Select Database
- config get dir — Get database file location to take backup
- Copy file 'xxxx.db' from above location to your home directory
- FLUSHDB — Flush database
Now you can start all Airflow services:
1- Scheduler commands
sudo initctl start airflow-scheduler
sudo initctl status airflow-scheduler
2- Webserver commands
sudo initctl start airflow-webserver
sudo initctl status airflow-webserver
3- Flower commands
cd /var/lib/airflow/prd/bin
nohup ./airflow.sh flower start &
./airflow.sh flower status
4- Worker commands
cd /var/lib/airflow/prd/bin
Airflow Redis Cluster
nohup ./airflow.sh worker start &
./airflow.sh worker status
Airflow Redis Hook
Go back to Airflow and validate all DAGs are starting and completing successfully.
Airflow Redis Hook Example
And he happy ever after! 🙂
