Preview: Mattermost Release Process

We’re working on making internal processes in the Mattermost core team more transparent for the community. Below is a working draft of our software development process, which will be updating live as refine our process.

Questions, feedback, comments always welcome,


Mattermost core team plans to work on a monthly release process by end of 2015. Ideally we’ll be shipping on the 16th of each month, however our v0.7.0 release is scheduled for September 4, 2015 and we’ll need to adjust a few of our fall 2015 releases to move gracefully to a release cycle on the 16th.

This document outlines the development process for the Mattermost core team, which draws from what we find works best for us from Agile, Scrum and Software Development Lifecycle approaches.

This is a working document that will update as our process evolves.

Principles

Our development process centers about three principles:

  1. Make something people love – Talk to users constantly. Shape and adapt plans based on feedback and data.
  2. Big rocks first – Queue large, important, potentially destabilizing features early in a release cycle. Leave plenty of time for stabilization and testing.
  3. Slow is smooth, smooth is fast – Thinking carefully and building predictably at high quality are more important than jamming in any one feature.
    When we’re not sure on a decision, these principles are our tie-breakers.

Roles

Responsibilities across the development process are divided into roles. Team members may take on one or more roles depending on skills, interests and product priorities. For example, some developers might also serve as part time product managers, or some product managers may be part-time designers.

Here’s how responsibilities lay out across roles:

  1. Everyone
  • Talk to users over community channels
  • Use and test the product, and file clear tickets
  1. Developers
  • Partner with product management in defining roadmap and strategy
  • Lead, analyze, and discuss technical decisions
  • Write, test and review code
  • Manage build and technical release process
  • Support and engage technical community
  • Author technical guidance and documentation
  1. Product Managers
  • Partner with developers in defining roadmap and strategy
  • Manage product planning, issue tracking, triage and release processes
  • Write and manage specifications and tickets
  • Support technical community
  • Author general guidance and documentation
  • Designers
  • Write and manage specifications and tickets
  • Create wireframes, full fidelity UI renderings, front end code
  • Develop and author design and branding standards
  1. Operations
  • Manage day-to-day team processes and checklists
  • Lead testing and monitoring of product quality
  • Review resolved issues and close or re-open as appropriate

Process

With this in mind, below is the Mattermost development process.

Sprints

  • Sprints start and end on Fridays at 10am PST and last two weeks. Sprints continue regardless of release process.
  • (Daily)
  • PM triages incoming tickets
  • Bugs are added to active sprint. Feature are added to the backlog, assigned to a dev, component and fix version
  • Devs add estimates to tickets assigned to them
  • Operations reviews resolved tickets.
  • Tickets are closed if they are complete, re-opened if there is something significantly wrong, or opens new tickets if there are minor issues that need to be addressed in a resolved ticket.
  • (Weekly)
  • Tuesday: Team meeting
  • Wednesday: Team Testing (Announce in Platform Discussion channel what test is being done this week)
  • Friday: Team meeting (Demo features from past week and have “Kaizen” where each team member suggests one potential improvement for the team process)
  • (End of Sprint)
  • PM organize the next two-weeks of sprint tickets based on dev estimates

Release (cut-off times based on 10am PDT)

###- (10 weekdays before release date) Cut-off for major features

  • No major features can be committed to the current release after this date
  • Backlog is reviewed and major features that won’t make it are moved to next release
  • Dev joins PM in triage meetings to prioritize the final tickets of the release.
  • After stand-up on the cutoff day, devs should prioritize review, updating and merging of all pull requests that are going to make it into the release.

- (8 weekdays before release date) Feature Complete and Stabilization

  • After the cut-off time for Feature Complete, Dev prioritizes reviewing PRs and committing to master so Stabilization period can begin, with testing and high priority bug fixes.
  • During Stabilization period only BUGS can be committed to master, non-bug tickets are tagged for next version and wait until after a release candidate is cut to be added to master. Exceptions can be made by triage team consensus across PM and Dev. List of approved changes for release candidate 1 here: https://mattermost.atlassian.net/issues/?filter=10204
    • (PM) Documentation
    • (PM) Make Changelog PR with updates for latest feature additions and changes
    • (PM) Make Changelog PR with updates to contributors
    • (PM) Make NOTICE.txt PR for any new libraries added from dev, if not added already
    • (PM) Prioritize any developer documentation tickets
    • (PM and devs) Sign-off testing of their feature areas (i.e. PM/dev either signs-off that their area is well tested, or they flag that potential quality issues may exist)
    • (Ops) Mail out mugs to any new contributors
    • (Team) Select “Top Contributor” for the release from external contributions to be mentioned in release announcement
    • (Marketing) Decides announce date (discuss in meeting)
    • (Ops) Post Announce Date in Release channel + update the channel header to reflect date
    • (Marketing) Communicates checklist of items needed by specific dates to write the blog post announce (eg screenshots, GIFs, documentation) and begins to write the blog post, tweet, and email for the release announcement
  • (PM) Works with Ops to check Quality Gate for feature complete

- (5 weekdays before release date) Code Compete and Release Candidate Cut

  • (Team) Meets to discuss release at 10am PST
    • (PM) Each area changed in latest release is assigned a PM owner to lead testing.
    • (Ops) Walks through each item of the Code Complete and Release Candidate Cut checklist
    • (Dev) Last check of tickets that need to be merged before RC1
  • After 10am PST meeting the release is considered “Code Complete”.
    • (Dev) Completes final reviews and updates of PRs marked for the release version
    • Master is tagged and branched and “Release Candidate 1″ is cut (e.g. 1.1.0-RC1)
      • (Dev) Create new release listed under GitHub Releases, use description:
        For Release Notes please see [Changelog](https://github.com/mattermost/platform/blob/master/CHANGELOG.md)
      • Following the branch, devs can go back to committing to master for tickets scheduled for the next release.
      • (Dev) RC1 is pushed to an acceptance server by a dev following the PRODUCTION installation instructions step-by-step
      • (Dev) RC1 is pushed to an acceptance server by a dev following the PREVIEW AWS EBS installation instructions step-by-step
      • (Dev) RC1 is pushed to GITLAB.MATTERMOST.COM
      • (Dev) Create a “Release v1.1.0-RC1” channel on team site + add team members. Links to Preview and Production, plus invite links, added in header of channel
      • (Dev) Post in Town Square to announce Release Candidate is ready
    • (PM) Create meta issue for regressions in GitHub (see example)

- (4 weekdays before release date) Release Candidate Testing

  • Final testing is conducted by the team on the acceptance server and any issues found are filed.
  • (Ops) Posts copy of the Release Candidate Testing checklist into Town Square in PRODUCTION
    • (Ops) Moves meeting, test and community channels over to the production version of RC, and posts in Town Square asking everyone to move communication over to the new team for testing purposes
    • (PM) Test feature areas and post bugs to Bugs/Issues in PRODUCTION
    • (Ops) Runs through general testing checklist on RC1 and post bugs to Bugs/Issues in PRODUCTION
    • (PM & DEV) Add #p1 tag to any “Blocking” issue that looks like a hotfix to the RC is needed, and create a public ticket in Jira. Blocking issues are considered to be security issues, data loss issues, issues that break core functionality, or significantly impact aesthetics.
  • (PM) Updates the GitHub meta issue
    • (PM) Posts links to all issues found in RC as comments on the meta issue
    • (PM) Updates description to include approved fixes
    • (PM) Posts screenshot and link to final tickets for next RC to the Release room
  • (PM & DEV leads) Triage hotfix candidates and decide on whether and when to cut next RC or final
  • (Dev) PRs for hotfixes made to release branch, and changes from release branch are merged into master
  • (Ops) Tests approved fixes on master
  • (Dev) Pushes next RC to acceptance after testing is complete and approved fixes merged
  • (Dev) pushes next RC to acceptance and announces in Town Square on PRODUCTION
    • (PM) closes the meta issue after the next RC is cut, and opens another ticket for new RC
  • (Ops) verifies each of the issues in meta ticket is fixed
  • (PM) If no blocking issues are found, PM, Dev and Ops signs off on the release

- (2 weekdays before release date) Release

  • (Dev) Tags a new release (e.g. 1.1.0) and runs an official build which should be essentially identical to the last RC
  • (PM) Any significant issues that were found and not fixed for the final release are noted in the release notes
  • If an urgent and important issue needs to be addressed between major releases, a hotfix release (e.g. 1.1.1) may be released, however this should be very rare, given a monthly cadence
  • (PM) Update the mattermost.org/download page
  • (Dev) Delete RCs after final version is shipped
  • (PM) Close final GitHub RC meta ticket

- (0 weekdays before release date) End of Release

  • (PM) Makes sure marketing has been posted (animated GIFs, screenshots, mail announcement, Tweets, blog posts)
  • (PM) Close the release in Jira
  • (Dev) Check if any libraries need to be updated for the next release, and if so bring up in weekly team meeting
  • (Ops) Post important dates for the next release in the header of the Release channel
  • (Ops) Queue an agenda item for next team meeting for “Stepping Back” Q&A
  • (Ops) Queue an agenda item for next team meeting for Roadmap review

Quality Gate for Feature Complete

Quality Gates are used to gradually raise standards as a product matures. New quality gates may or may not be added each release as criteria to declare “Feature Complete” status. Examples of new quality gate criteria for a given release may include:

  • All install instructions reference tested versions of dependencies and tools
  • Draft IT admin and developer documentation of all new features is included under /doc and a links provided from /doc/README.md
  • Draft of end user help for each new feature area is included in /doc/help
2 Likes