Я пытаюсь ограничить число облачных задач Google не более чем одной обработанной задачей в секунду.
Я создал свою очередь с помощью:
gcloud tasks queues create my-queue \
--max-dispatches-per-second=1 \
--max-concurrent-dispatches=1 \
--max-attempts=2 \
--min-backoff=60s
Описание этого дает мне:
name: projects/my-project/locations/us-central1/queues/my-queue
rateLimits:
maxBurstSize: 10
maxConcurrentDispatches: 1
maxDispatchesPerSecond: 1.0
retryConfig:
maxAttempts: 2
maxBackoff: 3600s
maxDoublings: 16
minBackoff: 60s
state: RUNNING
После создания группы задач в журналах я вижу, что многие из них нежелательно обрабатываются в течение 1 секунды:
2019-07-27 02:37:48 default[20190727t043306] Received task with payload: {'id': 51}
2019-07-27 02:37:48 default[20190727t043306] "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306] Received task with payload: {'id': 52}
2019-07-27 02:37:49 default[20190727t043306] "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306] Received task with payload: {'id': 53}
2019-07-27 02:37:49 default[20190727t043306] "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306] Received task with payload: {'id': 54}
2019-07-27 02:37:49 default[20190727t043306] "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306] Received task with payload: {'id': 55}
2019-07-27 02:37:49 default[20190727t043306] "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306] Received task with payload: {'id': 56}
2019-07-27 02:37:49 default[20190727t043306] "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306] Received task with payload: {'id': 57}
2019-07-27 02:37:49 default[20190727t043306] "POST /my_handler HTTP/1.1" 200
2019-07-27 02:37:49 default[20190727t043306] Received task with payload: {'id': 58}
Как правильно заставить его запускать не более 1 задачи в течение этого 1-секундного интервала времени?
Обновление 30/06:
Я попробовал это снова с базовой настройкой, та же проблема.
Подробнее о настройке и процессе:
- Исходный код https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/flexible/tasks, без изменений
- Разверните app.yaml, а не app.f flex.yaml
- Запускать задачу несколько раз: python create_app_engine_queue_task.py --project = $ PROJECT_ID --queue = $ QUEUE_ID --location = $ LOCATION_ID --payload = привет
- Проверьте журналы: прочитайте журналы приложений gcloud
На этот раз им потребовалось некоторое время, чтобы начать обработку, но после этого кажется, что все они были обработаны более или менее одновременно:
Полные журналы:
2019-07-30 00:22:37 default[20190730t021951] [2019-07-30 00:22:37 +0000] [9] [INFO] Starting gunicorn 19.9.0
2019-07-30 00:22:37 default[20190730t021951] [2019-07-30 00:22:37 +0000] [9] [INFO] Listening at: http://0.0.0.0:8081 (9)
2019-07-30 00:22:37 default[20190730t021951] [2019-07-30 00:22:37 +0000] [9] [INFO] Using worker: threads
2019-07-30 00:22:37 default[20190730t021951] [2019-07-30 00:22:37 +0000] [23] [INFO] Booting worker with pid: 23
2019-07-30 00:22:37 default[20190730t021951] [2019-07-30 00:22:37 +0000] [26] [INFO] Booting worker with pid: 26
2019-07-30 00:27:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:27:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:27:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:27:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:27:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:27:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:41 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:41 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:42 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:42 default[20190730t021951] Received task with payload: hello
2019-07-30 00:37:43 default[20190730t021951] "POST /example_task_handler HTTP/1.1" 200
2019-07-30 00:37:43 default[20190730t021951] Received task with payload: hello