Feature/queue

config.yml

@@ -6,6 +6,6 @@ outputDir: build
 themeDir: theme
 
 customVariables:
-  current_version: '5.3.1'
+  current_version: '5.3.2'
   current_year: 2020
 

docs/_toc.yml

@@ -204,6 +204,18 @@
       href: add-ons/moblog.md
     - name: Pages
       href: add-ons/pages.md
+    - name: Queue
+      items:
+        - name: Overview
+          href: add-ons/queue/index.md
+        - name: Creating A Job
+          href: add-ons/queue/creating-a-job.md
+        - name: Running the Job
+          href: add-ons/queue/running-queue.md
+        - name: Monitoring the Queue
+          href: add-ons/queue/monitoring-queue.md
+        - name: Important Notes
+          href: add-ons/queue/notes.md
     - name: Query
       href: add-ons/query.md
     - name: RSS

docs/add-ons/queue/creating-a-job.md

@@ -0,0 +1,115 @@
+# Creating a Job
+
+Jobs are added into and managed by third party add-ons.
+
+## addon.setup.php
+
+Jobs are registered and initialized in the `addon.setup.php` file of an addon. A job belonging to an add-on is registed by creating an associative array called `jobs`. The key should be the name of the class, and the value should be the fully qualified namespace of the Job class.
+
+```php
+'jobs' => [
+	'SampleJob' => AwesomeAddon\Jobs\SampleJob::class,
+],
+```
+
+## Class Structure
+
+Job classes must use the `ExpressionEngine\Addons\Queue\Traits\Queueable` trait and contain both a `__construct` and `handle` function.
+
+```php
+namespace AwesomeAddon\Jobs;
+
+use ExpressionEngine\Addons\Queue\Traits\Queueable;
+
+class SampleJob {
+
+	use Queueable;
+
+	public function __construct()
+	{
+		$this->construct();
+	}
+
+	public function handle()
+	{
+		echo 'Hello world';
+	}
+}
+```
+
+The job class constructor must call the `construct` method from the `Queueable` trait in order to set it up for proper serialization.
+
+When the job is created, all public and protected class variables will be serialized to the database. It is important that any information you want to be processed in the job is included in a public or protected class variable. For example:
+
+```php
+class SampleJob {
+
+	use Queueable;
+
+	protected $name;
+
+	public function __construct(string $name)
+	{
+		$this->construct();
+		$this->name = $name;
+	}
+
+	public function handle()
+	{
+		echo "Hello {$this->name}";
+	}
+}
+```
+
+There are also a number of class variables that can be set to manage the job on the queue.
+
+### Attempts
+
+This is the number of times a job will be attempted if it fails. Default: 1
+
+```php
+protected $attempts = 3;
+```
+
+### Sleep
+
+This is the number of seconds that the server will pause after running the job. Default: 1
+
+```php
+protected $sleep = 10;
+```
+
+### Run At
+
+This is the time that the job should be run. It is a MySQL datetime stamp format Y-m-d H:i:s.
+
+```php
+protected $runAt;
+
+public function __construct()
+{
+	$this->construct();
+
+	// Add 10 minutes
+	$time = new DateTime;
+	$time->add(new DateInterval('PT10M'));
+	$this->runAt = $time->format('Y-m-d H:i');
+}
+```
+
+## Creating Job
+
+Once you have your Job class created and registered in the add-on, you can initiate it using EE's `queue` helper.
+
+```php
+use AwesomeAddon\Jobs\SampleJob;
+
+public function do_stuff()
+{
+
+	queue(new SampleJob('Derek'));
+
+}
+```
+
+This will automatically serialize the job and add it to the queue.

docs/add-ons/queue/index.md

@@ -0,0 +1,35 @@
+# QUEUE
+
+## Overview
+
+This queue system allows for asynchronous execution of a time-consuming task, like sending an email, by saving the information to be executed when system resources allow. It utilizes the CLI in order to process the queue.
+
+## Connection
+
+Currently, this uses the current primary database connection. Upon installation of the Queue add-on, it will create a `jobs` and `failed_jobs` table to manage jobs.
+
+## Installation
+
+The queue can be installed on the add-on panel by clicking Install on the Queue add-on.
+
+### Testing Queue
+
+There is a test job that comes out of the box with ExpressionEngine in order to help you test the queue, practice running it on the CLI, and give a solid example of the structure needed.
+
+In order to create a test job, you can run `php eecli queue:test`. This command will ask you for an email address, and send that email address an inspirational quote.
+
+Upon successful creation, the command will instruct you to run the queue by running `php eecli queue:work`. The CLI will indicate successful processing by letting you know the job has processed
+
+```
+$ php eecli queue:test
+
+Choose an email, we'll send them an inspirational quote:  hello@expressionengine.com
+Job is queued! Run `php eecli queue:work` to process
+
+$ php eecli queue:work
+
+Processing 1
+Processed ExpressionEngine\Addons\Queue\Jobs\SampleJob
+```
+
+As long as your email settings are correct, you should receive an email with a quote in the appropriate inbox.

docs/add-ons/queue/monitoring-queue.md

@@ -0,0 +1,47 @@
+# Monitoring Queue
+
+The queue system comes with a GUI monitor out of the box, and can be viewed by going to `Add-ons > Queue`.
+
+## Jobs Table
+
+The jobs table shows you current jobs in queue along with expected payload and assigned run at date. From there you will also have opportunity to cancel a job, if indicated prior to the expected start of the job.
+
+Jobs that are in the jobs table are removed after successful completion of the job.
+
+## Failed Jobs
+
+Failed jobs are jobs that did not complete as expected. This could be whether there was a problem creating the job, or whether a job failed in execution.
+
+The failed jobs table shows you all of the failed jobs in queue along with expected payload, assigned run at date, and error that occurred. You can also attempt to retry a failed job, even if the number of attempts have passed.
+
+You can also permanently delete a failed job.
+
+## Flushing Queue
+
+Flushing the queue allows for all jobs to be cleared, both the current and failed.
+
+### From CLI
+
+The queue can be flushed from the CLI by running `php eecli queue:flush`:
+
+```
+$ php eecli queue:flush
+Deleting all current jobs
+Deleting all failed jobs
+Queue flushed
+```
+
+You can also choose the jobs to flush by adding the `--failed-only` and `--fresh-only` options.
+
+### From Jump Menu
+
+The queues can be flushed EE6 jump menu by accessing the Queue: Flush commands.
+
+## Jump Menu
+
+A number of commands are available in the jump menu, including:
+
+- View Current Jobs: View the current queue of jobs
+- View Failed Jobs: View the list of failed jobs
+- Flush Queue: Clear out the entire queue, both current and failed jobs
+- Flush Failed Jobs: Clear out all failed jobs from the database

docs/add-ons/queue/notes.md

@@ -0,0 +1,11 @@
+# Important Notes
+
+## Resources
+
+Queue workers are meant to handle large jobs that can happen asynchronously. It is important to make sure that you set proper time out settings so as not to have your job ignored when it times out.
+
+Daemon queue workers do not "reboot" the framework before processing each job. Therefore, you should free any heavy resources after each job completes. For example, if you are doing image manipulation with the GD library, you should free the memory with imagedestroy when you are done.
+
+## Deploying
+
+Queues save classes at a specific point in time. So, any code changes that are made once a job is queued may not be applied. In order to avoid this, after deploying new code, make sure to restart Supervisord.

docs/add-ons/queue/running-queue.md

@@ -0,0 +1,83 @@
+# Running Queue
+
+The queue is run and managed through a CLI command: `php eecli queue:work` This will process jobs in the queue by their consecutive order received and assigned runAt date.
+
+If no jobs are found, the process will end with no output
+
+## Limiting Jobs
+
+The queue worker is assigned to take 3 jobs at a time and process them, so it can keep server resources free and avoid timing out. You can add the `--take` parameter to the command in order to adjust this.
+
+```bash
+php eecli queue:work --take=10
+```
+
+You can also use the `--once` option in order to take the first job in queue.
+
+```bash
+php eecli queue:work --once
+```
+
+The `once` option will override the `take` option.
+
+## Keeping the Queue Running
+
+A queue isn't much of a queue if you have to manually run it. It is simple to set up a few methods in order to keep the queue running.
+
+### Managing with Supervisor
+
+Supervisord or Supervisor daemon is an open source process management system. In a nutshell: if a process crashes for any reason, Supervisor restarts it. Full documentation for Supervisor is found here: [http://supervisord.org/index.html](http://supervisord.org/index.html)
+
+#### Installing Supervisor
+
+To install Supervisor on Ubuntu, you may use the following command:
+
+`sudo apt-get install supervisor`
+
+#### Configuring Supervisor
+
+Supervisor configuration files are typically stored in the /etc/supervisor/conf.d directory. Within this directory, you may create any number of configuration files that instruct supervisor how your processes should be monitored. For example, let's create a expressionengine-worker.conf file that starts and monitors a `queue:work` process:
+
+```bash
+[program:expressionengine-worker]
+process_name=%(program_name)s_%(process_num)02d
+command=php /path/to/my/site eecli queue:work --take=3
+autostart=true
+autorestart=true
+user=root
+numprocs=2
+redirect_stderr=true
+stdout_logfile=/path/to/my/site/logs/worker.log
+stopwaitsecs=3600
+```
+
+In this example, the numprocs directive will instruct Supervisor to run 2 `queue:work` processes and monitor all of them, automatically restarting them if they fail. Make sure that the value of `stopwaitsecs` in your configuration file is greater than the number of seconds consumed by your longest running job. Otherwise, Supervisor may kill the job before it is finished processing.
+
+#### Starting Supervisor
+
+Once your configuration file has been created, you may update the Supervisor configuration and start the processes using the following commands:
+
+```bash
+sudo supervisorctl reread
+
+sudo supervisorctl update
+
+sudo supervisorctl start expressionengine-worker:*
+```
+
+This will be processing your queue immediately, and restart it if it crashes.
+
+### Managing with CRON
+
+If Supervisor is not an option, you can also utilize CRON to run the queue for you.
+
+#### Set Up Cron Job
+
+Edit your crontab in your server text editor, and add a job that runs according to the schedule you would like:
+
+```bash
+# This will run every minute
+* * * * * php /path/to/my/site eecli queue:work --take=3 > /dev/null 2>&1
+```
+
+More information can be found here: [https://crontab.guru/](https://crontab.guru/)