Sending

Methods of Sending (Browser, Cron, Command-line)

Setting up phpList for sending is a complex topic, there are many (many) variables, most of which which are out of the control of phpList developers, such as your hosting provider and server configuration. 

In our early chapters, we have assumed that you are sending from the browser, to a small list, and that your phpList has been configured already. 

How sending works 

Once your campaign is ready, you need to tell phpList to send the campaign. There are two steps to this:

1. Submit the message to the queue.

When creating the campaign, the last tab of the workflow is the 'SEND' tab. Pressing the 'start campaign' button submits the campaign to the sending queue. 

2. Process the queue, either manually via the browser or command line, or automatically via a cron job or remote queue processing system. 

Manual sending using the browser

This is the simplest way to send. You must:

This method of sending works essentially as described in the chapter on Sending your first campaign. Using the default configuration file settings, either click the Process Queue button which is presented as the final step of the campaign workflow, or use the System > Process queue option from the main menu. 

Once you begin sending you must leave the browser open: if the browser is closed while the message is sending, the queue processing will stop. To continue sending where you dropped off, you will need to open the browser, log into phpList, and "System"/"Process queue" again, leaving the browser open.

The system will send the queue using the settings in your config file (config.php), which controls the send rate, the envelope header, and so forth. You should talk to your hosting provider about these settings, they may want you to send in small batches or have a cap on the number of mails you can send each hour.

phpList.com remote queue processing

A wide explanation of how the phpList remote queue processing works is on the Resources page. Additionally, here you can can find some of the most frequently asked questions

One way to bypass the issue of leaving your browser open without creating a cron job is to use the phpList.com remote queue processing system.

This means that once you click to process the queue, phpList.com will keep your system sending after you close your browser: you are still sending from your server, and according to the settings in config.php - phplist.com just keeps your system "awake" for you.

You will be prompted to consider this process the first time you send. If you have previously sent messages then visit http://yourdomain/lists/admin/?page=hostedprocessqueuesetup to get started.

You should see the following page.

Click Create Account.

IMG_20190628_151900.png

This opens a window with a phpList.com page enabling you to create a special phpList.com account. Fill out your details and click Create account.

Screenshot-from-2019-06-28-14-56-19.png

You then need to go to your email account and click the link in the confirmation email.

Screenshot-from-2019-06-28-15-04-50.png

Click Continue to your account to go to your new account.

Screenshot-from-2019-06-28-15-12-22.png

Click Generate new key to get your API key.

Screenshot-from-2019-06-28-15-12-46.png

Copy the API key.

 

Screenshot-from-2019-06-28-15-13-26.png

And paste it into your phpList installation, into the Enter the API key here box on http://yourdomain/lists/admin/?page=hostedprocessqueuesetup

Screenshot-from-2019-06-28-15-13-53.png

Click Continue setup.

IMG_20190628_152502.png

The process is now complete. When you add a message to the queue, it will be processed according to your settings.

Queue processing from a remote computer using a secret (Config > Settings > Secret for remote processing)

This is essentially the same system as above, but you don't use the phplist.com service, you use another server of your choosing. Documentation for this is here

Automatic queue processing using the command line and Cron jobs

Background

A more advanced phpList configuration is to configure automatic queue processing from the command line of the computer (shell terminal). These are the most powerful ways of sending and they have many advantages. However, setting them up can be complicated for a variety of reasons including:

If you don't understand, or give up trying to understand all these details, look for qualified third party support to configure this and/or offer hosted solutions that include automatic queue processing.

The Command line

The core of setting up the automatic processing is the command line that will 'do the deed'.

For queue processing, the general format is:

/usr/bin/php /home/mydomain/lists/admin/index.php -pprocessqueue -c/home/mydomain/lists/config/config.php

Breaking this down:

/usr/bin/php  This is the command that invokes php. Your path may vary, as well as the exact php command ( php-cli or php-cgi). The exact path to the php command will need to be changed to match your system.


/home/mydomain/lists/admin/index.php  This is the fully qualified path to the index.php file in the /lists/admin directory of your phpList installation, as seen by the php command (above). Getting this right usually requires some experimenting, and/or asking your hosting provider. The '/home/mydomain' part will need to be changed to match the layout of your system.


-pprocessqueue  This specifies the action that you want to perform (processqueue)


-c/home/mydomain/lists/config/config.php This is the fully qualified path to the config file with your settings. The '/home/mydomain' part will need to be changed to match the layout of your system.

Once you determine the exact command line that will process the queue on your machine, then you need to run that command periodically via a cron job (also called a 'scheduled task' in the cpanel interface).

Setting up a cron job to periodically run the Command Line

See the chapter "Setting up your Cron".

Feedback

Discuss this chapter here.

Setting up your Cron

What is cron

Cron is a time based task scheduler on Unix-like operating systems. For more information check the Wikipedia page. This chapter will describe how to set this up on Linux. On other Unix systems the set up will be similar.

Why set up cron

When you set up the cron, two of the most time consuming parts of phpList will become automated. These are processing the queue and processing the bounces. Once you set up the cron, you will not have to do these in your browser, but they will happen automatically. Sending a campaign will become as easy as sending a normal email from your desktop.

Caveats

The method described in this chapter requires the CLI version of PHP.  This may not be available on all systems. There are some "hacks" to make it work with other versions of PHP, but this is non-intentional functionality and therefore not described here.

Step 1: set up commandline

The easiest way to set up the cron is to set up a commandline script that will handle all phpList commands. There is an example in the "bin" directory in the phpList archive file.

Let's call the commandline script "phplist" and edit it with your favourite text editor:

$ nano phplist

and type (or copy-paste) the content (note the line breaks; the following command should be pasted literally as two lines):

#!/bin/bash
/usr/bin/php /home/website/public_html/lists/admin/index.php -c /home/website/public_html/lists/config/config.php $*

You will need to change the above content to fit your system:

#!/bin/bash -> the shell that you want to use

/usr/bin/php -> the path to the PHP-cli commandline interpreter, this can vary per system. Debian based systems (including Ubuntu) should install the "php-cli" package.

/home/website/public_html/lists/admin/index.php -> the path to your phpList installation, pointing to the index.php file in the admin directory of phpList

/home/website/public_html/lists/config/config.php -> the path to your config file

Once you have constructed your commandline file, place it somewhere in your path, e.g. in /usr/local/bin and make it executable.

chmod 755 /usr/local/bin/phplist

 From then on, you can process phpList on commandline, for example with

phplist -pprocessqueue

Step 2: set up the crons

Once you have the commandline script, you can set up the crons with your favourite cron editor. On Linux, you type "crontab -e" in order to edit your cron entries.

0-59/5 * * * * phplist -pprocessqueue > /dev/null 2>&1
0 3 * * * phplist -pprocessbounces > /dev/null 2>&1

The above example will process the queue once every 5 minutes and process the bounces once a day. That will be sufficient for most systems.

The example also discards any output, which you will want to do. You can always check the output by running the above commands manually from the shell-prompt without

> /dev/null 2>&1

On CentOS 7 you need to specify the full path to the bash script. Your cron entries should look like this:

0-59/5 * * * * /usr/local/bin/phplist -pprocessqueue > /dev/null 2>&1
0 3 * * * /usr/local/bin/phplist -pprocessbounces > /dev/null 2>&1

Step 3: tell phpList

Once you've set up the cron and it is working, you can tell phpList and some links will disappear. Put the following two lines in your config file:

define ("MANUALLY_PROCESS_BOUNCES",0);
define ("MANUALLY_PROCESS_QUEUE",0);

This will hide the links to process the queue and bounces from the phpList interface.

Remote queue processing

The remote queue processing is often called “phpList cron job” due to its similarity of other automatic cron jobs. Before version 3.1 there was not a standard option to process the queue from a different location than the same server your phpList instance is installed. From version 3.1.1 onwards there is an option to remotely process the queue. It is always highly recommended to use the latest phpList version for security updates and for your own convenience as latest versions offer more functionalities.

In Config -> Settings under the Security section, there will be an option called "Secret for remote processing". The first time, it will generate a random code, which you can use, or you can set your own. To run the queue remotely, all you need to do is load the following page:

http(s)://yoursite.com/lists/admin/?page=processqueue&secret=XXXX

Where the XXXX is the value that is set for the "Secret for remote processing"

Use the Hosted Service

phpList.com now has a “secret” way to get our servers to process your queue for you. There are no payment plans and if you use this, you will help us to tweak the way it works, and in return we will process your queues without charge.
We offer this, because the last thing we'd want is that the queue processing is an obstacle for using phpList. From any point of view, but mostly performance, we advise the use of commandline processing because that is the most powerful way of sending. phpList Hosted is using commandline for everything. But in some cases this may not be an option. This is where our service hopefully fulfills a need.
To use the phpList.com queue processing service sign up at https://www.phplist.com/.
Once you signed up, get your API key.
The first time you try to manually process your queue, you will get the option to set up processing with phpList.com. You can use that to set up the API key.
Alternatively go to:

http://www.yoursite.com/lists/admin/?page=hostedprocessqueuesetup

From then on, when you place a campaign in the queue, or requeue an existing one, it will activate the processing from our servers. We will run the queue very often and make sure your campaigns are sent.

If you place an embargo on a campaign, the hosted queue processing service will detect this and wait until your embargo has passed to send your campaigns.

We do keep track of the statistics, and we will give you a summary in your account pages. The phpList.com service uses the exact same system as the normal remote processing outlined above and we have no other access to your system.

Notes

There's no danger in overloading your server. When a second queue processing command is activated when the previous one is still active, phpList will detect this and bail out. Therefore there will always be one active queue sending process at any time. This is also important to ensure that the sending limits, set with Batch processing, will be honoured.

Appendix

How safe is this?

Is it not easy for someone to steal my secret? Particularly when you call the parameter “secret”.
It is quite safe, but you need to determine this for yourself. The output of the page when using the remote secret is minimised to only the statistics. If an “attacker” gets hold of your secret, all they would do is help you process your queue. In fact, if you run the queue this way yourself, you will be logged out from phpList. The access with the secret is restricted to processing the queue and nothing else. We will later on extend this to processing bounces as well. In comparison, using a “username=X&password=Y” processing URL would allow anyone snooping that data to have access to your entire phpList system.Also, if you use HTTPS, chances of someone stealing your secret are way lower.

What about timeouts, how long will the request last?

When run from a webpage, phpList, by default, will only run the queue processing for one minute. It will try to send as many mails as possible in that time. Then you will need to run it again.

Feedback

Discuss this chapter here.

Scheduling

In the campaign workflow you will find the scheduling tab.

If you want to send a campaign straight away, and there is no need to stop sending before a specific time, then you can leave these details unchanged and your campaign will simply send.

If your needs are more complex, read on.

Important information 

Scheduling settings interact with your sending method. If you are only able to process your queue from within the browser because you have no cron or command-line access, then the embargo time feature will not work for you. To send a campaign from the browser you must always open your browser to process the queue and keep it open until you have finished, even if you have used an embargo time.

Scheduling tab

Below is an image of the scheduling tab. We we will work though it from top to bottom:

Screenshot-from-2019-06-28-15-29-35.png

Server time 

The server time may not be the same as your local time. All of the times in the scheduling page are relative to server time. Changing this time is an advanced feature. 

Embargoed until

The campaign will not be sent until this time has passed. If you have a campaign which you wish to send at a later date, perhaps when you out of the office or away from your desk, then this is the feature you should use.

Embargo time is initially set to the nearest hour before you started your new campaign: this effectively switches off the embargo feature by default. To use the feature you must set the time in the future. Simply set the date and time to the time you want to send, relative to your server time which is stated above. Then place the campaign in the queue as normal. If you are using a cron job then the campaign will automatically start sending at the desired time.

IMG_20190629_204617.png

Stop Sending After

This feature is used when the topic of a campaign is time sensitive, for example, a special offer with an expiration date or an event with a specific time. If the campaign would become irrelevant to anyone who received it after a particular time, then this is the feature you should use. 

The primary use of this feature is when your list size exceeds your sending capacity within the timeframe you need to send your campaign in. For example, if you have a list of 10,000 subscribers and can send 1000 emails in an hour but the campaign becomes irrelevant in 8 hours time, you need to stop sending before everyone on your list has received a copy of the mail.  

Re-Queue

The re-queue setting is used to send a campaign to subscribers who join a list after the campaign is sent. Remember, a campaign will never be sent to the same subscriber twice.

IMG_20190629_204640.png

As an example, you create a monthly newsletter on the first of January, and send it to a list. Over the month you expect another 50 people to sign up, or be added to your list manually, and you would also like them to automatically receive a copy of the campaign. You would set up your campaign to re-queue every day and to re-queue until 31st of January.

Feedback

Discuss this chapter here.

Sending a web-page

To send a web page rather than a message to your subscribers, log in to your phpList admin pages, then choose to Send a Campaign Admin>Campaigns>Send a Campaign

From this page, click on Send a New Campaign (or edit an existing Campaign). In this example, I'll use Send a New Campaign.

Screenshot-from-2019-06-28-15-35-06.png

This results in:

Screenshot-from-2019-06-28-15-35-54.png

Choose Send a Webpage rather than Compose message and you should see this:

Screenshot-from-2019-06-28-15-36-35.png

In the send a web page - URL box, enter the path of the webpage you wish to send. Click outside the box and the URL is validated.

If you get this message, then check and double check that the page you have entered actually exists. The URL does look correct, but the page does not in fact exist.

Screenshot-from-2019-06-28-15-38-45.png

Once the page to be sent has been validated, simply select the template for your message (optional), the list to which you wish to send to (required), check that the embargo time is correct and send the campaign out.

 Display Issues

Be aware that most email readers, especially those web-based ones (Hotmail/Outlook, Gmail etc) will not show images by default, it will be up to your subscribers to decide if they wish to receive images in emails by (usually) clicking on a link in the email reader. Don't assume that your images will automatically appear, so make sure your webpage includes ALT Text for each image in your page when designed.

Feedback

Discuss this chapter here.

Sending plain text

phpList always generates a plain text version of your campaign, which is then included in the html campaigns as a matter of course. This is part of the normal working of email functionality. phpList also gives you two additional features:

 

You can see the plain text version embedded in the source of your email by clicking View Source in your email client (exact instructions vary wildly)

IMG_20190730_140501.png

As you can see there is a plain text version in and amongst the html and other code.

Screenshot-from-2019-07-03-13-37-58.png 

Why edit the plain text version?

 

The automatically generated plain text version may not be very attractive. It is good practice to provide a quality plain text version because it:

To assess the state of your default plain text version, you can click to view your emails only in plain text. In a desktop email client this is usually View > Message Body As > Plain text.

IMG_20190730_140446.png 

You can then see your email as it would be displayed to someone who is only able to receive plain text messages.

Screenshot-from-2019-07-03-13-39-21.png 

To improve this email and send a version formatted especially for these subscribers only takes a few extra moments and, as you can see, provides a much better quality of message.Screenshot-from-2019-07-03-13-37-58.png

Enabling plain text editing

Some installations do not show the Text tab, which is used to manually configure your text-only email content. If your tabs look like this, you'll need to change the configuration:

Screenshot-from-2019-07-30-14-16-28.png

To activate the Text tab in your installation, add the following lines to your config.php file:

# Manual text part, will give you an input box for the text version of the message

# instead of trying to create it by parsing the HTML version into plain text

define('USE_MANUAL_TEXT_PART',1);

An extra tab will now display in your campaign workflow. 

activate_manual_text_tab.png

Create your plain text version

Once you've created your HTML version in the Content tab (tab 1) click the Next button.manual_text_tab_actiaved.png

You should now be in the second tab, Text.