Custom Management Command 8: Importing Large Data Using Celery & Reddis

1. After we installed REDIS, we installed REDIS as a package in our VIRTUAL ENVIRONMENT.

2. You installed CELERY.

$ pip install celery

3. Run this to create a Celery WOrker:

windows:

$ celery -A autocommontasks_main worker --loglevel=info --pool=solo

macos:

$ celery -A autocommontasks_main worker --loglevel=info

This results to this with an error:

4.  To correct this, we have to update our SETTINGS.PY

In the bash terminal, press CTRL + C to terminate the process and try again. Your celery should be ready.

5. To use CELERY for testing in our project, create a new URL in our root project's URLS.PY

We run our django server and att the url in our browser: http://127.0.0.1:8000/celery-test/

6. Now update our CELERY_TEST FUNCTION to perform time-consuming operations.

So, when you reload your page, in the background, it will load 10 seconds (Sleep)

7. We want to pass then the SLEEP(10) TASK to celery, so that our project or user can do or see other things while Celery is working on something else. To do this, in the DATAENTRY folder, create a new file, TASKS.PY and update as

8. To test all this, LOAD your URL http://127.0.0.1:8000/celery-test/ again, click on your CELERY TERMINAL, and it should RECEIVE and DISPLAY the message 'TASK EXECUTED SUCCESSFULLY' while we wee our webpage with H3 tag showing the message right away.

IMPORTANT REMINDER:

So, when you run your django project with celery and redis, THERE SHOULD BE 3 BASH TERMINALS AND YOU NEED TO NAME THEM APPROPRIATELY:

1. DJANGO-SERVER- this is where you will run your python server to run your Django project:

$ python manage.py runserver

2. REDIS - this is where you run your REDIS-SERVER. Every time you start this, it will create a new process of REDIS, so you can simply kill it or ignore it. As long as when you PING your REDIS SERVER, it returns a PONG message, then you are fine. You have multiple processes (TCP) here because you have started your REDIS-SERVER multiple times.

$ redis-server

$ netstat -ano | findstr :6379 (this is to see what process is listening to the port 6379 (Redis)

$ redis-cli ping (Run this before you run the REDIS-SERVER so you wont have several TCPs)

3. CELERY- this is where you will run this code whenever you need to launch a new task or when you reload your Django page. Meaning, you have to launch this command again if you have made new changes to your CELERY TASKS like changing 10 seconds to 5 seconds to see the effect.

 *autocommontasks_main    - this is your Django Project Name

$ celery -A autocommontasks_main   worker --loglevel=info --pool=solo

 

No PDF file attached.