Channels ▼

Open Source

Workflow Automation on Web Projects: One Team's Experience

When we started working on Cloudify's new version, 3.0 — which is a multi-module project based on Python and YAML that will have an entirely new UI, our dev team needed to interact with to many new technologies, which required entirely new skills. Before setting out on this project, we realized that with all this coming-to-life of many new modules in the product, the constant flow of new features and requirements, and the accumulating set of specs, we would need to come up with solutions to simplify the project management. So, we decided to align our development workflows so that we could better work across teams and environments. This workflow automation, essentially would enable us to occupy ourselves with the work at hand, rather than the ramp up of getting acquainted with new methodologies each time a new project rolls around, eventually letting us do what we do best: programming.

Since we found ourselves writing new projects for nearly every component in the new product, we had many configurations. Before aligning our workflows, we spent too much time just becoming familiar with new environments and configurations--sometimes spending more time on that than the actual task. This cost us valuable development time, so all these activities really needed to be standardized to help us remain in focus as a team. In this short article, I'm going to dive into the process, the workflows we installed, and the tools we use to get the job done, ultimately to make our whole dev teams as portable as possible.

Getting Started

To overcome complexities, we decided to create a unified stack, meaning that every developer on the UI team uses some set of common tools and frameworks (which does not mean we don't get to learn and implement new and exciting technologies). For us, this basically meant we wanted to use familiar languages that we know well (yes, we love JavaScript), while automating away the nitty gritty involved with bootstrapping projects, boilerplate code, etc.

We unified the stack by choosing popular technologies that are tried and well tested, but most importantly, are also active — active projects mean stable code. For the purpose of aligning our project structures and automatically generating code, we chose to work with Yeoman. For building and packaging, we went with Grunt. Bower and NPM then provide ready-made components for the front end and back end, respectively. As for the actual development of the product, we use AngularJS, NodeJS, and Express almost exclusively, although some N-Tier applications (like the reimplemented Cloudify Player) are even going further, utilizing a full MEAN stack. Since the motivation is to use familiar technology, and because MongoDB accepts JSON, the use of MongoDB allows us to go JavaScript all the way.

Working the Workflow

Having chosen this stack, the first thing a developer does to create a new project is go to the command line and type yo angular:app. This tells Yeoman to auto-generate the full project structure for an AngularJS client, including in-place configuration files for build and package management. We do the same for other layers, and we rely on out-of-the-box code generators to eliminate friction when updating these generation tools. With this common boilerplate, we can know that all of our projects will look the same; therefore, on top of providing us the benefit of familiarity with JavaScript-based technologies, there is also the added benefit of familiarity with the project structure. This is very important for context and a quick ramp up.

After a project is set up, a developer typically needs to do two things: fetch external dependencies and assemble it all together. To handle dependencies in our environment, we use the NodeJS Package Manager (NPM) to pull in third-party libraries for the backend, and Bower for client-side components. As Bower, NPM, and Grunt already have their own configuration files included in each project, this means that after we've decided on our components and defined them in these configuration files, we then simply call an NPM install (as simple as running npm install), a Bower install (similarly easy), and then call a Grunt build — and just like that, all our components are included into the project automagically. This enables us to start working immediately.

Ready-Made Reuse

At our site, we hate to reinvent the wheel. So we use what other developers, like us, have written in the past to solve issues just like ours. For instance, if a developer needs some sort of a slider component to be used in a Web page, she can just look up the Bower repository with Bower search, find an appropriate slider (let's say, angular-slider), and then all she'd have to do is Bower install angular-slider — and it's integrated into the project. Of course, some of the time, the developers who write those components are ourselves. We have our very own repository of reusable components we built for generic use. We introduce them into a project in the exact same manner — they're all published into Bower!

Running the Client

While developing in the client, we use preview servers — provided by Grunt's LiveReload and Watch plugins, in our case, so we really don't have to refresh the browser page at any point, which is annoying for developers. Once Grunt's development server is up and running, it will watch for any change in the project files and reload that specific resource in the Web app using websockets. This allows us to work continuously. Grunt rewires the app into the browser as we go along, and there's no clicking the refresh button. Ever. It's all very tidy.


Once we finish developing a given feature, we commit it to the source control. But before the check-in, it needs to be validated and tested. Again, we use Grunt, which provides some helpers out of the box. To verify our code, we go to the command line, type grunt jshint, and it runs validation on the code. To run tests, just run grunt test. We could do all of this with a single command, grunt, which runs all these steps for us. Once the code passes locally, we're ready to commit to Github and have the continuous integration (CI) tools do their job. Needless to say, the CI tools also use Grunt to create the distribution (dist is the Grunt task responsible for that).

We write tests with Jasmine, which is a behavior-driven development framework for testing JavaScript code, (and the nice thing is that an entire test suite scaffold is written for us the moment we add some auto-generated code. Yeoman does that for us, and all we have to do is fill in the test-methods bodies).

There's a lot to be said about continuous integration that I won't dive into here, but I'll just share that we use TravisCI and QuickBuild to manage our tailored/nightly builds.

Just a Bit More

We also use profiles. A profile colors an environment with certain configuration flavor for a specific need. When developing, our environment has two flavors of configuration: one for production, and the other for development, (the latter is also used for testing). This separation provides us with convenient mocking abilities when testing or developing (Why use a full-blown backend server when all you need is a fixed response? Mock it up with Jasmine!), and it also protects us from incorporating any sensitive data into the production code and possibly having it checked-in into the repository. Such data items might include passwords, secret keys, API keys, and so forth. Those all reside in the development profile; the production profile will be built with the real credentials. Again, once developers form a habit of working with profiles, it becomes second nature. Just agree on a private place for personal configuration files, switch the dev profile on, and roll with it.

A Final Note

When we started using unified workflows and tools, we ourselves became more portable: we were able to hop from one team to another easily and work seamlessly across diverse projects. From developing a UI component, to working on a feature at the backend, shifting projects was easy. This means we can always concentrate on the important things, increasing productivity and speeding up delivery of projects significantly. Aligning workflows means less friction. Less friction means fewer worries. Fewer worries means more coding. And more coding means happy developers. Well, at least that's been the case for us.

Eliran Malka is a Web developer at GigaSpaces working on Cloudify. He's passionate about problem solving and always looking for new ways to do so.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.