Managed Forms

Managed Forms are a streamlined approach to website forms that reduce the many steps needed to process forms manually (TODO link to Enhanced Forms). It is designed for a form submission workflow where:

• the website user is anonymous (i.e. not logged in)
• the data collected is easily presented in a single page form
• an optional collection of files may be included
• the data may be validated against simple rules
• the submitter may be automatically verified as a non-robot using a CAPTCHA service (reCAPTCHA or hCAPTCHA)
• upon submission the form data, and any associated files, are stored in an encrypted file within the Managed Forms Vault
• a record of the submission is stored in the messaging table (dcmThread) along with minimal identifying information*
• a notification may be automatically sent to the website owner
• a confirmation may, optionally, be sent to the person making the submission**

*When building the managed form the developer selects which identifying information to leave unencrypted - typically just the name of the submitter.
**If an email is sent to the submitter it should be done using a From address containing the website's domain. This requires additional configuration not given on this page, see details below.

Although Managed Forms are considerably easier to code than Enhanced Forms, the list above illuminates that this is not necessarily a simple process.

Basic Example

Let's start with a simple example which covers the primary needs of most website forms.

The Form

The first step is to plan the form's UI. A simple example follows:

<p> We'd love to host your next event! </p> <dcf.ManagedForm TitleFields="Name,Date" Name="Event"> <dcf.Text Label="Name" Name="Name" /> <dcf.Text Label="Date" placeholder="" Name="Date" /> <dcf.Text Label="Time" placeholder="" Name="Time" /> <dcf.Text Label="Email" Name="Email" /> <dcf.Text Label="Phone" Name="Phone" /> <dcf.Text Label="Number of Attendees" placeholder="" Name="Attendees" /> <dcf.TextArea Label="Event" placeholder="describe your event" Name="Event" /> <dcf.FormButtons> <dcf.SubmitButton Label="Submit" /> </dcf.FormButtons> </dcf.ManagedForm>

This form presents six simple single line text input fields and one multiple line text input field, along with a Submit button. There are many additional input types, review the list of Enhanced Form Tags for some details, and we'll cover a few more examples later in this page. The key pieces to notice at first are:

• each input field needs a Label and a Name , the latter is the internal name of the field and should not contain spaces or special characters. These two attributes are enhanced - not part of HTML.
• Regular attributes such as placeholder , class and id keep their normal casing and may be used
• the dcf.FormButtons tag right aligns and adds padding around any buttons included
• no additional code is required for a SubmitButton, it just works (no additional script or POST location needed)

We'll cover a few more complex examples later in this document.

Form Configuration

In order for a form submission to work at all it needs to be configured in the website's config.xml file. The naming convention for the configuration Id is CMS-ManagedForm- then the form name (in this case Event ) and finally followed by the scope. The scope will almost always be -Both . For additional details on how Catalog works see here TODO.

<Catalog Id="CMS-ManagedForm-Event-Both"> <Settings Type="harEventForm" Script="/har/event-event-form-submitted.dcs.xml" /> </Catalog>

Since our form is named Event that links it to this Catalog entry and the Settings within. There are only two settings:

• Type: the name of the data type for validating the form
• Script: the script to run for notifications

Data Validation

First let's review the data type. Data types are defined in the schema.xml file, so go to the same folder as the config.xml file is in (unless this is a subsite - in which case go to the root site - TODO see schema-belongs-in-root). If there is no schema file then create it and use the following as a template:

<Schema> <Shared> <Record Id="harEventForm"> <Field Name="Name" Type="dcSmallString" Required="true" /> <Field Name="Date" Type="dcSmallString" Required="true" /> <Field Name="Time" Type="dcSmallString" Required="true" /> <Field Name="Email" Type="dcEmail" Required="true" /> <Field Name="Phone" Type="dcSmallString" Required="true" /> <Field Name="Attendees" Type="dcSmallString" /> <Field Name="Event" Type="String" Required="true" /> </Record> </Shared> </Schema>

What we see here is that harEventForm is a Shared (website wide) datatype and that it is a Record. Records consist of fields, and each field is listed here by Name . Note that the field name here must match the name of the input field from the form. Each Field has two key features - is it Required or not (simply leave off if not, such as with Attendees), and Type which is the data type. For these examples we'll stick to a few string data types.

• String: an (effectively) unlimited free text field
• dcSmallString: also a free text text field but limited to no more than 250 characters
• dcString: also a free text text field but limited to no more than 4000 characters (this may have been an appropriate option for the Event field)
• dcEmail: this must match a specific pattern of characters
• Integer: limits the field to whole numbers (this may have been appropriate for Attendees)
• Boolean: true or false fields (the YesNo or Checkbox inputs)

There are many other field (scalar) data types, and it is also easy to build your own, but these data types cover almost all the typical fields used with Managed Forms.

Note that you must list each field from the form and use the same Name. If the form contains names not in the schema then it will fail to submit. On the other hand, the form does not need to list all the fields from the schema - though it must list all the required fields. It is recommended that you do not remove fields from the schema if a field is no longer needed, instead simply make the field optional in the schema and then remove it from the form. The reason for this is that any data previously submitted (and stored in those encrypted files) will still have that old field. Keeping the field in the schema will help preserve the ability to validate the old data and that may come in handy with future enhancements to display previously submitted data via the CMS.

Submitted Data Access

While we have the data structures needed, there is currently no standard UI for presenting the submitted data via the CMS. Adding this ability would not be much effort and it would certainly make sense with more sensitive information. When submitting sensitive information, ideally the notice (see below) would contain minimal identifying information and then the website owner would then login and review the details through the secure website. However, currently, there is no standardized approach to this - either website owner receives all the information (and files) via the notice or they use a custom UI to view the submission details.

TODO expand - how to see message details

Notifications

An script is executed after the form has been securely stored in the vault (encrypted file). The script provides an opportunity for any or all of the following:

• sending an appropriately formatted notice to the website owner (or a representative thereof)
• sending a notice to the submitter
• running some automated data processing on the data submitted

Recall from above that we have this in our config file:

<Catalog Id="CMS-ManagedForm-Event-Both"> <Settings Type="harEventForm" Script="/har/event-event-form-submitted.dcs.xml" /> </Catalog>

The path for the Script assumes the folder script - so the full path of the script within the website folder is /script/har/event-event-form-submitted.dcs.xml . We'll review the script soon, but first note that in the config file we also defined our mailing list destinations. The naming convention for the mailing list Id is Email-List-ManagedForm- then the list name (in this case Event to match the form name - but it need not match the form name) and finally followed by the scope. The scope will almost always be split to -Test and -Production so that test emails can be generated during website development. For additional details on how Catalog works see here TODO.

<Catalog Id="Email-List-ManagedForm-Event-Production"> <Settings To="nardiharmony@gmail.com;andy@designcraftadvertising.com" /> </Catalog> <Catalog Id="Email-List-ManagedForm-Event-Test"> <Settings To="andy@designcraftadvertising.com" /> </Catalog>

Given the above, we see that when in development emails will go to just andy, whereas in production emails go to andy and nardi (note the semicolon delimiter is required for each additional destination). The email list is a handy place to list the addresses, but it is the script that builds and sends the email, so let's review that now.

<dcs.Script Title="Managed Event Form Submitted"> <dcs.Info>Just got an event form submission: {$_Param.Id}!</dcs.Info> <dcdb.LoadRecord Table="dcmThread" Id="{$_Param.Id}" Result="Thread"> <Select Field="dcmTitle" As="Title" /> <Select Field="dcmManagedFormName" As="Form" /> <Select Field="dcmUuid" As="Uuid" /> <Select Field="dcmManagedFormToken" As="Token" /> </dcdb.LoadRecord> <dcs.File Name="datafile" Path="/{$_Param.Id}/data.json" In="ManagedForms" /> <dcs.With Target="$datafile"> <ReadText Result="datastr" /> </dcs.With> <dcs.Var Name="Form" Type="Record"> <Set>{$datastr}</Set> </dcs.Var> <text Name="textEmail"><![CDATA[## Event Request Form **Name:** {$Form.Data.Name} **Date:** {$Form.Data.Date} **Time:** {$Form.Data.Time} **Email:** {$Form.Data.Email} **Phone:** {$Form.Data.Phone} **Attendees:** {$Form.Data.Attendees} **Event Info:** {$Form.Data.Event} ]]></text> <dcs.SendEmail ToList="ManagedForm-Event" Subject="Harmony Bar {$Thread.Title}" TextMessage="$textEmail" ReplyTo="{$Form.Data.Email}" /> </dcs.Script>

For the most part when creating a script it is reasonable to just copy this script, or one like it. In that situation the code you are most interested in is the <text> tag and the <dcs.SendEmail> tag. As the script runs it loads the data file from the vault, which also decrypts it, and then it parses the data file into JSON. By the time we get to the <text> instruction there is the $Form.Data variable which holds all the data fields from the form.

In the <text> instruction simply build a markdown document, containing all necessary form data, within the CDATA section. This markdown is then converted to HTML when the email is sent. However, since all emails should contain both a text and a HTML option - the markdown is sent as the text option (even though almost no one will ever see it). You should always build a text option (as we do above), but if you want it to be formatted differently from the HTML there is a way to specify the HTML option separately. TODO link to building emails in scripts

The <dcs.SendEmail> command specifies (using the ToList attribute) which email list to use (it adds Email-List- to the start and the scope to the end), so in our example ManagedForm-Event matches the mailing lists discussed above. The TextMessage attribute indicates the source of the message content. The Subject attribute indicates the subject line of the email. And, finally, the ReplyTo attribute indicates the recipient if one clicks reply. Since this example notice is going to the website owner (Nardi) the reply value makes it easy for her to reply to requests submitted online through her email software.

TODO - note that notices may also be texts to a phone instead of emails

Notifications To Submitter

If the website also wishes to send a confirmation email to the submitter, just add these lines to the script above:

<text Name="textEmailConfirm"><![CDATA[ Dear {$Form.Data.Name}, Thank you for contacting us about your event on {$Form.Data.Date} at {$Form.Data.Time} - we will review the request and respond within two business days. Your neighbors, Harmony Bar and Grill ]]></text> <dcs.SendEmail To="{$Form.Data.Email}" Subject="Harmony Bar Event Received" TextMessage="$textEmailConfirm" />

The copy for the confirmation could say anything, including listing all the variables again, however it may include only a summary as this example does.

Instead of using ToList we use To and point it to the submitter's email. The ReplyTo has been removed, for although it could be hard coded to the webmaster's email, we'll rely instead on the default From email which is part of the website email setup (TODO link to article on email setup).

Even though there isn't much additional code needed to send a confirmation, I recommend not sending a this confirmation. For one, the user is told that the submission was successful via the website - they don't need another confirmation. But, furthermore, if notices are only being sent to the website owner and their staff then it is okay for the email to come from the system's default webmaster (the agency that coded the website). But if the email is going to customers then the From field should be from the website owner. In order to support that the domain name must first be verified in AWS, see email setup link above. The verification and activation process is time consuming and therefore should only be done if confirmation emails are important.

Example with Complex Data Types

Form

Expanding on the previous example we'll add in a YesNo field, a RadioGroup, a CheckGroup and a Checkbox:

<dcf.ManagedForm TitleFields="Name,Date" Name="Event"> <dcf.Text Label="Name" Name="Name" /> <dcf.Text Label="Date" placeholder="" Name="Date" /> <dcf.Text Label="Time" placeholder="" Name="Time" /> <dcf.Text Label="Email" Name="Email" /> <dcf.Text Label="Phone" Name="Phone" /> <dcf.Text Label="Number of Attendees" placeholder="" Name="Attendees" /> <dcf.YesNo Label="Open Bar" Name="OpenBar" /> <dcf.HorizRadioGroup Label="Type of Event" Name="EventType" Instructions="We provide special pricing on the pool table for birthdays and on the stage for anniversaries."> <RadioButton Value="Birthday" Label="Birthday" /> <RadioButton Value="Anniversary" Label="Anniversary" /> <RadioButton Value="Other" Label="Other" /> </dcf.HorizRadioGroup> <dcf.CheckGroup Label="Pizza Types" Name="PizzaTypes" Instructions="We make enough cheese pizza to feed 1/2 of your attendees, the other 1/2 are split equally across your selections below:"> <Checkbox Label="Pepperoni" Value="Pepperoni" /> <Checkbox Label="Sausage" Value="Sausage" /> <Checkbox Label="Veggie" Value="Veggie" /> <Checkbox Label="Gluten Sensitive" Value="GlutenSensitive" /> </dcf.CheckGroup> <dcf.TextArea Label="Event" placeholder="describe your event" Name="Event" /> <dcf.Checkbox Name="ReadGuide" LongLabel="I have read the events guideline PDF." /> <dcf.FormButtons> <dcf.SubmitButton Label="Submit" /> </dcf.FormButtons> </dcf.ManagedForm>

The CheckGroup and RadioGroup both have instructions added to them. This is a feature available to all field types. Note that the RadioGroup could also have been a Select element. Many form building options are available under the Enhanced Forms section.

Data type changes:

• the YesNo field (OpenBar) and the Checkbox (ReadGuide) will be considered a Boolean (true/false) type
• the RadioGroup (EventType) and the CheckGroup (PizzaTypes) will both be specialized enumerated types

First let's make the data types for the enumerated fields. These would both be added to the <Shared> section of the schema.

<StringType Id="harEventTypeEnum"> <StringRestriction Enum="Birthday,Anniversary,Other" /> </StringType> <StringType Id="harPizzaTypeEnum"> <StringRestriction Enum="Pepperoni,Sausage,Veggie,GlutenSensitive" /> </StringType>

Now add the fields to the event form data type:

<Record Id="harEventForm"> <Field Name="Name" Type="dcSmallString" Required="true" /> <Field Name="Date" Type="dcSmallString" Required="true" /> <Field Name="Time" Type="dcSmallString" Required="true" /> <Field Name="Email" Type="dcEmail" Required="true" /> <Field Name="Phone" Type="dcSmallString" Required="true" /> <Field Name="Attendees" Type="Integer" /> <Field Name="OpenBar" Type="Boolean" /> <Field Name="EventType" Type="harEventTypeEnum" /> <Field Name="PizzaTypes"> <List Type="harPizzaTypeEnum" /> </Field> <Field Name="Event" Type="String" Required="true" /> <Field Name="ReadGuide" Type="Boolean" Required="true" /> </Record>

Notice:

• the two enumerated types use the special data type (in Type )
• switched Attendees to Integer so the user must enter a number
• the order in which fields appear in the data type doesn't really matter, though listing in the same order as on the form helps with readability
• the pizza field can have more than one response, check all that apply, so note how that field contains a List of a special type unlike the other fields which have only one type.

Now in the email notice add these new fields:

<text Name="textEmail"><![CDATA[## Event Request Form **Name:** {$Form.Data.Name} **Date:** {$Form.Data.Date} **Time:** {$Form.Data.Time} **Email:** {$Form.Data.Email} **Phone:** {$Form.Data.Phone} **Attendees:** {$Form.Data.Attendees} **Open Bar:** {$Form.Data.OpenBar|yn:} **Event Type:** {$Form.Data.EventType} **Pizza Types:** {$Form.Data.PizzaTypes|join:, |ifempty:[none selected]} **Guidelines Read:** {$Form.Data.ReadGuide|yn:} **Event Info:** {$Form.Data.Event} ]]></text>

Notice:

• both of the Boolean data types have been formatted above using |yn: which converts true/false to yes/no for human readability.
• event type is unformatted and so will show it's value only - through enumeration may have translations if dictionaries are used (TODO link to translations)
• pizza types is a list so it can be joined into a comma list using the |join: formatter - in this case adding a comma and space between each. Furthermore, if the list is empty then the |ifempty: formatter adds “[none selected]”

Along with more advanced field types, it is relatively easy to expand on the features of Managed or Enhanced Forms.

Example of Forms with Files

Managed Forms have the built-in option to support a single file upload prompt. It does allow for multiple files, but only one prompt or field. Extending on the topics above, the additions would be:

<dcf.ManagedForm Name="Estimate" TitleFields="Name"> <dcf.Text Label="Name" Name="Name" /> <dcf.Text Label="Email" Name="Email" /> <dcf.Text Label="Phone" Name="Phone" /> <dcf.TextArea Label="Description" Name="Description" placeholder="Description of your project" /> <dcf.Uploader Label="Estimate Files" Name="Attachments" data-max-size="7340032" Instructions="Please limit total file size to 7MB." /> <dcf.FormButtons> <dcf.SubmitButton Label="Submit" /> </dcf.FormButtons> </dcf.ManagedForm>

The addition of dcf.Uploader and the Name of “Attachments”. The Label and Instructions can be whatever you like, and as always the Instructions are optional. A managed form with an Uploader named “Attachments” is the key to the automated process of a managed form.

No special addition is needed in the config.xml for a form with Attachments. For example, the above form would be exactly the same with or without the Attachments:

<Catalog Id="CMS-ManagedForm-Estimate-Both"> <Settings Type="ttrEstimateForm" Script="/ttr/event-estimate-form-submitted.dcs.xml" /> </Catalog>

The list of file names will be sent to the server along with the other information from the form (Name, Email, Phone, Description above). So to accommodate those names you'll need to add Attachments to your schema as well. Set it to be a list of strings. For example:

<Schema> <Shared> <Record Id="ttrEstimateForm"> <Field Name="Name" Type="dcSmallString" Required="true" /> <Field Name="Email" Type="dcEmail" Required="true" /> <Field Name="Phone" Type="dcMetaString" Required="true" /> <Field Name="Description" Type="String" Required="true" /> <Field Name="Attachments"> <List Type="dcMetaString" /> </Field> </Record> </Shared> </Schema>

The addition of the Attachments to the email notification is easy to code but can be unreliable. Many email services limit attachment sizes - sometimes the total size of all the attachments is limited and sometimes the size of individual attachments is limited. If you noticed the form above has data-max-size , the primary reason for using that with that website is to make sure the attachments can be emailed. In general, as noted before, it is preferable not to email details or files in the notification. Rather it would be best for the notification to simply indicate new action is required and for the client to sign into the (secure) website to view the content and the files.

However, most websites will prefer the attachments to be in the email. To support that simply add this code before the dcs.SendEmail tag.

<dcs.Var Name="Attachments" Type="List" /> <dcs.If Target="$Form.Data.Attachments" IsEmpty="false"> <dcs.ForEach Name="Attachment" In="$Form.Data.Attachments"> <dcs.File Name="addfile" Path="/{$_Param.Id}/{$Attachment}" In="ManagedForms" /> <dcs.Var Name="AddRec" Type="Record"> <SetField Name="Name" Value="$Attachment" /> <SetField Name="File" Value="$addfile" /> </dcs.Var> <dcs.With Target="$Attachments"> <AddItem Value="$AddRec" /> </dcs.With> </dcs.ForEach> </dcs.If>

While it may look complex, there is no need to understand or edit the code - just use it as is. There is a further addition required, the addition of the Attachments attribute to SendEmail. Example:

<dcs.SendEmail ToList="ManagedForm-Estimate" Subject="{$Thread.Title}" TextMessage="$textEmail" ReplyTo="{$Form.Data.Email}" Attachments="$Attachments" />

Note that $Attachments is the variable defined in the code block above.

With those two easy adjustments your email notice is now sending the uploaded files as attachments to the email.

Example with CAPTCHA Services

I no longer recommend using reCAPTCHA on the dcServer forms. We find that very few forms are spammed, since we do not use the POST back feature of forms. And clients are always concerned about interfering with, or confusing, clients. But following is an overview if you have to support it.

Quick summary of set up (TODO move to own page and link to that).

Technically we can support any of the following options:

• reCAPTCHA v2 (“I'm not a robot” Checkbox)
• reCAPTCHA v2 (Invisible reCAPTCHA badge)
• reCAPTCHA v3

However, most of the time it is fine to use the reCAPTCHA v2 (Invisible reCAPTCHA badge) option, and it is the easiest to set up. reCAPTCHA v3 is simply to unreliable for form verification.

First go and get a new site key. Learn about how to do so here:

https://developers.google.com/recaptcha/docs/versions

Make sure you list the website domain as well as the developer domain, subdomains don't matter. For example:

tonytrappllc.com
designcraftadvertising.com

Would along either domain to be used when testing the form.

Once you have the Site Key and Secret Key edit the config.xml for your website:

<Catalog Id="Google-Both"> <Settings TrackingCode="UA-2478599-00"> <reCAPTCHA SiteKey="6Ldk9ocUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" SecretKey="6Ldk9ocUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" /> </Settings> </Catalog>

Place the reCAPTCHA tag inside the Settings tag you normally use for the website Google Analytics code.

After this, the only remaining step is to replace the submit button:

<dcf.SubmitButton Label="Submit" />

becomes:

<dcf.SubmitCaptchaButton Label="Submit" Action="submit_contact" />

Provide an appropriate Action label (attribute) such as “contact” or “contest” or “pledge”, no spaces or special characters besides dash or underscore.

With the submit button changes and the key in place, the form should now automatically use reCAPTCHA to verify the human user. With the Invisible option the prompt will not always happen, but behind the scenes Google is checking for a real user.

hCAPTCHA Alternative

hCAPTCHA provides an alternative to Google's reCAPTCHA v2 services. The only difference needed on the website is to the config.xml file, use a “CAPTCHA” catalog instead of the “Google” catalog, like so:

<Catalog Id="CAPTCHA-Both"> <Settings Service="hCAPTCHA"> <hCAPTCHA SiteKey="00000000-0000-0000-0000-000000000" SecretKey="0x00000000000000000000000000000000000000" /> </Settings> </Catalog>

To get your key and secret, you'll need to sign up for the service:
https://www.hcaptcha.com/