GroupServer 12.11 — Absinthe acquired arbitrarily

Authors:Michael JasonSmith; Richard Waid; Dan Randow
Contact:Michael JasonSmith <mpj17@onlinegroups.net>
Date:2012-11-12
Organization:GroupServer.org
Copyright:This document is licensed under a Creative Commons Attribution-Share Alike 3.0 New Zealand License by OnlineGroups.Net.

Introduction

There are fourteen major changes to GroupServer in the Absinthe release — making the system more useful, usable and extensible for both group members, administrators, and developers.

The changes visible to participants are mostly subtle improvements to existing features, rather than new features. In contrast, the changes to the underlying system are complex and extensive.

You can get Absinthe immediately.

Acknowledgements

Thanks to Bill Bushey, with the support of E-Democracy.org, for the improvements to the SVG thumbnails. Thanks to Bill and Marek Kuziel for testing some early versions of Absinthe.

Changes visible to participants

The most visible change for the participants is a new search system. Some better error pages have also been created. To help participants and administrators there is a new help system, while an in-context administration guide will help administrators get their new group going more easily. Administrators now have the ability to create closed groups, and the new profile search will help administrators find the profiles of participants. Finally, SVG thumbnails are now shown.

New search system

The new search system is the most visible change in Absinthe.

  • The Topics tab on both the site homepage and the group page have a Search entry [1].
  • The Posts tab in a group has as Search entry also, allowing posts to be searched [2].
  • Clicking a topic keyword will search for other topics with that keyword [3].
  • Searching now uses stems; searching for search will match topics that contain the words searches, searching or searched, for example.
  • Searching is faster, as the full-text retrieval features of PostgreSQL are used [4].
  • The display of topics in a site or group is much faster, as the keywords are now generated when someone posts, rather than when the topic is loaded.
  • Searching using non-ASCII characters now works [5].

Better error pages

The Permission Denied page has been improved to add some suggestions about what the participant should do [6]. In addition the mailto that is embedded in the page now includes the URL of the problem page in the body of the email message to the Support address.

The Unexpected Error (500) page and the Not Found (404) page now work with infrae.wsgi, for systems that run GroupServer behind a WSGI front-end. Neither of these error-pages redirect to display the error.

New help system

The old monolithic manuals have been replaced. The new more dynamic system will automatically show help for the features that are installed in GroupServer, including the custom features. Some of the old pages have been retained and updated.

In-context administration guide

To help get groups started there is a new system to encourage the group administrator [7]. The current advice includes:

  • Start a topic
  • Invite people,
  • Write some text in the About tab, and
  • Make a group Private (rather than Secret).

Closed groups

Administrators can now close groups [8]. There are two reasons for needing to do this.

  1. The group may be starting, and the administrator does not want the members to post until everyone is in the group.
  2. The group may be finished, and the administrator does not want any more posts to be made, but he or she still wants the archive available.

While there is no front-end user interface that allows the group-type to be changed [9], an administrator can change the type of the group by making a change in the ZMI.

SVG thumbnails

GroupServer now correctly displays a thumbnail of an SVG image at the bottom of each post [11]. This item was picked from the list of low hanging fruit, where there are other (relatively) strait forward tasks listed.

Changes to the underlying system

We have made significant changes to the underlying GroupServer system in the Absinthe release. The system will be easier to maintain because of a new configuration system, improved email handling, and a new authentication system. Installation is simpler because relstorage is used by default. There have also been some significant performance improvements. Developers will notice the SQLAlchemy update, and a more flexible group page.

The changes to the underlying system have been so extensive that we have decided to change the naming scheme for the GroupServer releases. The new releases belong to the Awesome Aperitifs series. Hence this Absinthe Acquired Arbitrarily release. Internally, the eggs in this series are given the version 2.0 (which you can see by looking at the versions.cfg file in the build directory). The version-number of the release will continue to use the month.day format. The old series was known as Frozen Treats; Faloodeh Consumed with an Eye on History was the aptly-named last release in that series. (Eggs in the Frozen Treats series were given the 1.0 version.)

Configuration system

Administration is now simpler, especially for production systems, as the configuration for important parts of GroupServer are now in a file that is external to the ZODB. The new configuration system handles the database, the improved email handling, and the new authentication system. It is based on a INI file, located in parts/instance/gsconfig.ini.

Improved email handling

The email-handling subsystem of GroupServer has been completely rewritten. Changes have been made to both the handling of outgoing mail and incoming mail.

The setup for the outgoing SMTP system has moved from the ZODB (accessed through the ZMI) to an INI file [12], thanks to the new configuration system. Documentation for the new outgoing SMTP system can be found in the README for the gs.email product.

The script that is used to add email messages to a group, the incoming SMTP system, has been rewritten [13]. It is now easier to use, better documented, and works. Documentation for the new script can be found by running ./bin/smtp2gs -h or reading the README for the gs.group.messages.add.smtp2gs product.

New authentication system

A new authentication system has been created, for the server-side scripts [14]. These scripts, such as the those involved in the improved email handling, now pass a token to the server when they carry out tasks. This eliminates the need to store the password of the administrator in various plain-text files.

Relstorage

By default the Relstorage product is now used to store the ZODB. This system stores the pickled objects in a relational database, rather than in the file-system. (The PostgreSQL database is used by GroupServer.) This allows greater scalability, without the need to separately install Zope Enterprise Objects (ZEO).

Performance improvements

There have been some major performance improvements made to GroupServer in the Absinthe release. This includes the removal of some old poorly performing code [15], and altering some of the member management code [16].

SQLAlchemy update

The entire interface between GroupServer and the PostgreSQL relational database has been rewritten [17]. This has allowed GroupServer to update its SQLAlchemy dependency from the ancient 0.3 release to the current 0.7 release.

More flexible group page

The group page was refactored to make it more flexible [18]. This allows the addition of the in-context administration guide, and for other features to be added by skins.

Get Absinthe

To get Absinthe go to the Downloads page for GroupServer and follow the GroupServer Installation documentation. Those who already have a functioning installation can update an existing GroupServer system.

Update an existing GroupServer system

Updating a system running the Faloodeh release of GroupServer (12.06) to Absinthe is a three-step process, which includes updating the relational database, the products, and the scripts.

Relational Database

The biggest change that is needed to update GroupServer to the Absinthe release is to update the rational database, to support the new search system and some of the performance improvements. The tables that are used to store the posts, topics and topic keywords all need to be updated.

First, create a backup. While every effort has been made to crate a upgrade path that is smooth and with low risk, there is still a chance that something can go wrong. As such it is prudent to create a backup. First, create a backup of the relational database:

$ pg_dump -U gsadmin groupserver > gs-backup.sql

Where gsadmin is the PostgreSQL user that you set up when installing GroupServer, and groupserver is the name of the database.

If you use relstorage, create a backup of the ZODB:

$ pg_dump -U gszodbadmin groupserverzodb > gs-zodb-backup.sql

Where gszodbadmin is the PostgreSQL user for relstorage that you set up when installing GroupServer, and groupserverzodb is the name of the database.

Posts

Note:Update the posts table after you create a backup.

Begin by updating the table that stores the posts.

  1. Log in to the PostgreSQL command line:

    $ psql -hlocahlost -Ugsadmin groupserver
    

    Where gsadmin is the PostgreSQL user that you set up when installing GroupServer, and groupserver is the name of the database.

  2. Alter the post table to add the full-text retrieval (FTR, or full-text search, FTS) column, by executing the following SQL:

    ALTER TABLE post ADD COLUMN fts_vectors tsvector;
    
  3. Update the rows of the post table to add the FTR data. This may take some time:

    UPDATE post
    SET fts_vectors = to_tsvector('english',
                                  left(coalesce(subject,'') || ' ' || coalesce(body, ''),
                                       1048575));
    
  4. Create an index for the FTR data. This may take some time:

    CREATE INDEX post_fts_vectors ON post USING gin(fts_vectors);
    
  5. Create an index for the posts, sorted by the last post date:

    CREATE INDEX post_last_post_date_idx ON post (date DESC);
    
  6. Create a trigger to update the FTR data whenever a new post is made:

    CREATE TRIGGER fts_vectors_update
      BEFORE INSERT or UPDATE ON post
      FOR EACH ROW EXECUTE PROCEDURE
        tsvector_update_trigger(fts_vectors, 'pg_catalog.english', subject,
                                body);
    

Topics

Because people search topics as well as posts the FTR information needs to be present in both tables.

  1. Add the FTR column to the topic table:

    ALTER TABLE topic ADD COLUMN fts_vectors tsvector;
    
  2. Drop the old trigger:

    DROP TRIGGER count_word_count_rows ON word_count;
    
  3. Drop the old tables that were used for searching:

    DROP TABLE topic_word_count;
    DROP TABLE word_count;
    

    Even with the FTR data duplicated in the post and topic table, there is a nett saving of space once these two tables are dropped.

  4. Add the function that is used to create the body of the topic:

    CREATE OR REPLACE FUNCTION topic_body (topic_id TEXT)
      RETURNS TEXT AS $$
      DECLARE
          topic_text TEXT;
          subject TEXT;
          retval TEXT;
      BEGIN
        SELECT string_agg(post.body, ' ') INTO topic_text
          FROM post
          WHERE post.topic_id = topic_body.topic_id
            AND post.hidden IS NULL;
        SELECT COALESCE(post.subject, '') INTO subject
          FROM post WHERE post.topic_id = topic_body.topic_id LIMIT 1;
        retval := left(subject || ' ' || topic_text, 1048575);
        RETURN retval;
      END;
    $$ LANGUAGE 'plpgsql';
    
  5. Create the function that will topic table with FTR data:

    CREATE OR REPLACE FUNCTION topic_ftr_populate ()
      RETURNS void AS $$
        DECLARE
          total_topics REAL;
          trecord RECORD;
          topic_vector tsvector;
          topic_text TEXT;
          i REAL DEFAULT 0;
          p REAL;
        BEGIN
          SELECT CAST(total_rows AS REAL) INTO total_topics
            FROM rowcount WHERE table_name = 'topic';
          FOR trecord IN SELECT * FROM topic WHERE fts_vectors IS NULL LOOP
            RAISE NOTICE 'Topic %', trecord.topic_id;
            topic_vector := to_tsvector('english', topic_body(trecord.topic_id));
            UPDATE topic SET fts_vectors = topic_vector
              WHERE topic.topic_id = trecord.topic_id;
            i := i + 1;
            p := (i / total_topics) * 100;
            RAISE NOTICE '  Progress % %%', p;
          END LOOP;
        END;
    $$ LANGUAGE 'plpgsql';
    
  6. Populate the topic table with FTR data. This may take some time:

    SELECT topic_ftr_populate();
    
  7. Create an index for the FTR data. This may take some time:

    CREATE INDEX topic_fts_vectors ON topic USING gin(fts_vectors);
    
  8. Create an index for the topics, sorted by the last post date:

    CREATE INDEX topic_last_post_date_idx ON topic (last_post_date DESC);
    
  9. Create a trigger to update the FTR data:

    CREATE OR REPLACE FUNCTION topic_fts_update ()
      RETURNS TRIGGER AS $$
        DECLARE
          topic_text TEXT;
        BEGIN
          topic_text := topic_body(NEW.topic_id);
          NEW.fts_vectors := to_tsvector('english', topic_text);
          RETURN NEW;
        END;
    $$ LANGUAGE 'plpgsql';
    CREATE TRIGGER topic_update_trigger_01
      BEFORE INSERT OR UPDATE ON topic
      FOR EACH ROW EXECUTE PROCEDURE topic_fts_update ();
    

Topic Keywords

Finally, the system that displays the topic keywords has been changed. The keywords are now calculated when someone posts, and are stored in the topic_keywords table. Previously they were calculated when the list of topics was displayed.

  1. Download the file 03-keywords.sql:

    wget --no-check-certificate https://source.iopen.net/groupserver/gs.group.messages.topic/rawfile/tip/gs/group/messages/topic/sql/03-keywords.sql
    

    This contains the SQL that is normally executed when Absinthe is installed.

  2. Interpret (execute) the file in PostgreSQL:

    \i /path/to/the/download/03-keywords.sql
    

    Where /path/to/the/download is the full path to where the file 03-keywords.sql is stored.

  3. Create the function to populate the new topic_keywords table:

    CREATE OR REPLACE FUNCTION topic_keywords_populate ()
      RETURNS void AS $$
        DECLARE
          total_topics REAL;
          trecord RECORD;
          topic_text TEXT;
          new_keywords TEXT[];
          i REAL DEFAULT 0;
          p REAL;
        BEGIN
          SELECT CAST(total_rows AS REAL) INTO total_topics
            FROM rowcount WHERE table_name = 'topic';
          FOR trecord IN SELECT * FROM topic LOOP
            RAISE NOTICE 'Topic %', trecord.topic_id;
            topic_text = topic_body(trecord.topic_id);
            SELECT ARRAY(SELECT word
                           FROM topic_keywords(trecord.topic_id, topic_text))
              INTO new_keywords;
            INSERT INTO topic_keywords VALUES(trecord.topic_id, new_keywords);
            i := i + 1;
            p := (i / total_topics) * 100;
            RAISE NOTICE '  Progress % %%', p;
          END LOOP;
        END;
    $$ LANGUAGE 'plpgsql';
    
  4. Populate the topic_keywords table. This may take some time:

    SELECT topic_keywords_populate();
    

Products

To update an existing GroupServer installation to Absinthe carry out the following steps.

  1. Download the Absinthe tar-ball from the GroupServer download page.

  2. Uncompress the tar-ball:

    $ tar cfz groupserver-12.11.tar.gz
    
  3. Change to the directory that contains your existing GroupServer installation.

  4. Make a backup of your custom configuration:

    $ cp custom.cfg custom-bk.cfg
    $ cp config.cfg config-bk.cfg
    
  5. Copy the new configuration files to your existing GroupServer installation:

    $ cp ../groupserver-12.11/*.cfg .
    
  6. Restore your custom configuration:

    $ mv custom-bk.cfg custom.cfg
    $ mv config-bk.cfg config.cfg
    
  7. Disable the creation of the database tables:

    $ echo 1 > var/create-tables.cfg
    
  8. Disable the creation of a new GroupServer site:

    $ echo 1 > var/setup-gs.cfg
    
  9. In your existing GroupServer installation run:

    $ ./bin/buildout -n
    
  10. Restart your GroupServer instance.

Scripts

Some external scripts have changed in the Absinthe release of GroupServer, and need to be changed. In addition some ZMI scripts should also be updated.

The script smtp2zope used to be used to marshal an email message from Postfix into GroupServer. With the improved email handling this script should be deleted. The replacement script is called smtp2gs. It will be created when you update the products. The command is simpler to use than the old script; the options for the script are shown by running:

$ ./bin/smtp2gs --help

Alternatively, the README for the gs.group.messages.add.smtp2gs product documents the options.

The directory potfix_config in your GroupServer installation will contain an example aliases file for Postfix that uses smtp2gs. This can be used to replace the old calls to GroupServer from Postfix.

ZMI Scripts

Two scripts in the ZMI have to be replaced to gain some of the significant performance improvements.

  1. Visit the ZMI for your site. By default it is at <http://localhost:8080/manage>.

  2. Go to the folder /example/ListManager.

  3. Select the xwf_email_header script.

  4. Replace the contents of the script with the following:

    groupId = list_object.listId()
    siteId = list_object.siteId
    site = getattr(context.Content, siteId)
    group_object = getattr(site.groups, groupId)
    
    # we copy the propertysheet, because we won't be able to access it
    # in the lower layer
    group_properties = {}
    for p in group_object.propertyItems():
        group_properties[p[0]] = p[1]
    group_properties['id'] = group_object.getId()
    
    xmailer = getValueFor('xmailer')
    mailto = getValueFor('mailto')
    replyToProp = list_object.getProperty('replyto','')
    if replyToProp == 'sender':
        replyto = None
    else:
        replyto = getValueFor('mailto')
    
    try:
        group_properties['division_id'] = site.aq_explicit.getId()
        group_properties['division_title'] = site.aq_explicit.title
    except:
        group_properties['division_id'] = ''
        group_properties['division_title'] = ''
    
    files = []
    try:
        storage = context.FileLibrary2.get_fileStorage()
        for file_id in file_ids:
            file = storage.get_file(file_id)
            if file:
          files.append(file)
    except:
        pass
    
    return context.email_header(REQUEST, list_object=list_object,
                          group_properties=group_properties,
                          getValueFor=getValueFor, title=title, mail=mail,
                          body=body, files=files, post_id=post_id,
                          mailto=mailto, replyto=replyto,
                          xmailer=xmailer).strip()
    
  5. Click the Save Changes button.

  6. Click ListManager (at the top of the page) to return to the List Manager folder.

  7. Select the xwf_email_footer script.

  8. Replace the contents of the script with the following:

    groupId = list_object.listId()
    siteId = list_object.siteId
    site = getattr(context.Content, siteId)
    group_object = getattr(site.groups, groupId)
    
    # we copy the propertysheet, because we won't be able to access it
    # in the lower layer
    group_properties = {}
    for p in group_object.propertyItems():
        group_properties[p[0]] = p[1]
    group_properties['id'] = group_object.getId()
    
    try:
        group_properties['division_id'] = site.aq_explicit.getId()
        group_properties['division_title'] = site.aq_explicit.title
    except:
        group_properties['division_id'] = ''
        group_properties['division_title'] = ''
    
    # group_properties['canonical_host'] = \
    #   group_object.Scripts.get.option('canonicalHost')
    divConfig = site.DivisionConfiguration
    group_properties['canonical_host'] = divConfig.getProperty('canonicalHost', '')
    
    try:
        from_addr = context.parseaddr(mail.get('from',''))[1]
    except:
        from_addr = ''
    
    if from_addr:
        user = context.acl_users.get_userByEmail(from_addr)
    else:
        user = None
    
    files = []
    try:
        storage = context.FileLibrary2.get_fileStorage()
        for file_id in file_ids:
            file = storage.get_file(file_id)
            if file:
                files.append(file)
    except:
        pass
    
    # Get the virtual file folder "files" from the group.
    
    # Get the public_access_period from "files"
    pap = int(getattr(group_object.files, 'public_access_period', 0))
    # Turn the public_access_period to a Boolean
    pap_set = bool(pap)
    # Pass the Boolean to the "email_footer" template
    mailto = getValueFor('mailto')
    
    return context.email_footer(REQUEST, list_object=list_object,
                          group_properties=group_properties,
                          getValueFor=getValueFor, title=title,
                          mailto=mailto, mail=mail, body=body,
                          user_object=user, from_addr=from_addr,
                          files=files, post_id=post_id, pap_set=pap_set)
    
  9. Click the Save Changes button.

Resources

[1]The Search box that used to appear on every page has now been removed, as it is easier to use the Search entry in the Topics tab. This closes Bug 3434.
[2]Searching in posts closes Feature 3497.
[3]Creating topic keywords that can be clicked closes Feature 878.
[4]Using the full-text retrieval feature of PostgreSQL closes Feature 224.
[5]Being able to search for non-ASCII characters closes Bug 603.
[6]Updating the Permission Denied page closes Bug 646.
[7]Creating the encouragement closes Feature 3501 and Feature 177.
[8]Creating the closed-group closes Feature 449.
[9]The issue for creating a selectable group type is Feature 702.
[10]The creation of a basic profile search closes Feature 3486.
[11]Handling SVG Thumbnails closes Bug 635.
[12]Moving the SMTP configuration to an INI file means that the two MailHost instances from the ZODB can be removed, which closes Bug 365.
[13]Rewriting the script that is used to add email messages to a group closes Feature 687 and Feature 3536.
[14]The creation of a new authentication system closes Bug 3416.
[15]The removal of the old division_object getter closes Bug 279.
[16]The optimisation of the member-handling code closes Bug 3659.
[17]An unintentional site-effect of rewriting the database interface was a fix for Bug 203.
[18]As part of the update to the group, Feature 419 was closed.