Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Personas: Nuts and Bolts Without Navel Gazing

Two completely different personas using the same system simultaneously.

Two completely different personas using the same system simultaneously.

Personas are often used in software development to generate and group requirements. For example, one of the classic constructions of user stories is: <Persona> <Goal> <Benefit>. The use of personas has become nearly ubiquitous. A persona represents one of the archetypical users that interacts with the system or product. Every system or product will have at least one archetypical user that can be identified. The concept of personas was originally developed by Alan Cooper in his book The Inmates Are Running the Asylum. Personas represent different behaviors based on research into the jobs, lifestyles, spare time activities, attitudes, motivations, behaviors and social roles. I am often asked where a team should get a set of personas given the level of usage (as if they were a set of playing cards), and once a team gathers the data, how that data should be captured.

In most internal development enhancement and maintenance projects, personas are developed using some form of brainstorming process. Brainstorming with affinity diagraming is a great tool to develop personas. The Software Process and Measurement Blog has described the process of Affinity diagraming. The people involved in generating personas are critical. The team you use to generate personas needs to have a broad experiential base. It needs to be cross-functional and diverse. The team needs to be able to take systems views of who uses the systems or product, otherwise the list of personas and their definitions will be incomplete. Second, when generating ideas in the brainstorming session someone needs to act as a facilitator and provide seed questions to the team. The seed questions will help the team to uncover groups of users and potential users, their needs, behaviors, attitudes and both work and non-work activities. In the grouping portion of the affinity diagraming process, the team should first think about personas when grouping and then when naming the groups ask the team to give the groups persona names. A further twist on the tried and true affinity session mechanism that I have recently found to be effective is to have the team break up in to smaller sub-teams to review what was done and re-brainstorm entries that could fill the gaps after the grouping and naming exercise. I generally time box the second step to fifteen minutes, although time-boxing may constrain innovation. When the sub-teams return integrate the new entries into the affinity diagram. The named and grouped entries from the affinity diagram will form the basis for the documentation needed to capture the personas.

Documenting personas might be the most contentious part of the process of establishing and using personas. Documentation smacks of overhead to many in the development community. I have found that if you don’t take the time to document a persona, it is far less useful. Documenting personas does a number of things. First the act of documentation allows the team to flesh out their ideas and drive out gaps and inconsistencies. Second, documentation helps the team to establish collective memory. Third, fleshing out the nuances fleshed out and establishing institutional memory makes it more likely that the team will refer to the persona after development because the investment in time and effort generates a perception of value. When documenting personas, ensure that you keep the documentation to the absolute minimum. I use a single sheet of flip chart paper per persona; not only does the single sheet feel lean, it also allows teams to post the personas around the room. Templates are good to for maintaining repeatability.

In Agile Story Maps: Personas we discussed and referenced a template defined by Roman Pitchler. There are a wide variety of templates for capturing information about a persona. In general they all include the following information:

  1. Persona Name
  2. Picture: a picture gives the persona visual substance and personality.
  3. Job: What does a person in this archetypical group do for a living (as it relates to the product or system).
  4. Goal: A definition of why the persona would want to use the product or system. Support of the goal can be found in the more personality and lifestyle (often written as backstory) of the persona.
  5. Personality: How does the group that the persona represents think, act, or react. This attribute includes motivation, attitudes, behaviors and social roles.
  6. Lifestyle: How does the group that the persona represents actually act? What does the persona do both at work and in their spare time?

Every template that I have seen (or created) is nuanced based on the needs of the team and the project. For example, the personas generated to guide developing requirements for an enhancement to a data entry application would be different from the personas needed to plan a major Agile conference. In the latter case more emphasis might be needed around motivation and lifestyle which would influence the speakers that might be accepted, social events planned and when the speakers and events might be slotted in the schedule. Teams use personas to help them think conceptually about the problem they are trying to solve without getting bogged down by having to think about and address each individual that uses the system or product as a special, individual case.

Personas are a tool help teams to understand a class of users by creating an archetypical user. Personas need to be identifiable and have back-story that includes needs and goals. The use of fictional people allows the team to dissociate themselves from the large number of individuals in the user base so they don’t get trapped in paralyzing detail. Personas need to be written down or you will not remember the nuances of personalities and lifestyles that make any two personas different.


Categories: Process Management

Game Performance: Layout Qualifiers

Android Developers Blog - 9 hours 21 min ago
Today, we want to share some best practices on using the OpenGL Shading Language (GLSL) that can optimize the performance of your game and simplify your workflow. Specifically, Layout qualifiers make your code more deterministic and increase performance by reducing your work.

Let’s start with a simple vertex shader and change it as we go along.

This basic vertex shader takes position and texture coordinates, transforms the position and outputs the data to the fragment shader:
attribute vec4 vertexPosition;
attribute vec2 vertexUV;

uniform mat4 matWorldViewProjection;

varying vec2 outTexCoord;

void main()
{
  outTexCoord = vertexUV;
  gl_Position = matWorldViewProjection * vertexPosition;
}
Vertex Attribute Index To draw a mesh on to the screen, you need to create a vertex buffer and fill it with vertex data, including positions and texture coordinates for this example.

In our sample shader, the vertex data may be laid out like this:
struct Vertex
{
  Vector4 Position;
  Vector2 TexCoords;
};
Therefore, we defined our vertex shader attributes like this:
attribute vec4 vertexPosition;
attribute vec2  vertexUV;
To associate the vertex data with the shader attributes, a call to glGetAttribLocation will get the handle of the named attribute. The attribute format is then detailed with a call to glVertexAttribPointer.
GLint handleVertexPos = glGetAttribLocation( myShaderProgram, "vertexPosition" );
glVertexAttribPointer( handleVertexPos, 4, GL_FLOAT, GL_FALSE, 0, 0 );

GLint handleVertexUV = glGetAttribLocation( myShaderProgram, "vertexUV" );
glVertexAttribPointer( handleVertexUV, 2, GL_FLOAT, GL_FALSE, 0, 0 );
But you may have multiple shaders with the vertexPosition attribute and calling glGetAttribLocation for every shader is a waste of performance which increases the loading time of your game.

Using layout qualifiers you can change your vertex shader attributes declaration like this:
layout(location = 0) in vec4 vertexPosition;
layout(location = 1) in vec2 vertexUV;
To do so you also need to tell the shader compiler that your shader is aimed at GL ES version 3.1. This is done by adding a version declaration:
#version 300 es
Let’s see how this affects our shader, changes are marked in bold:
#version 300 es

layout(location = 0) in vec4 vertexPosition;
layout(location = 1) in vec2 vertexUV;

uniform mat4 matWorldViewProjection;

out vec2 outTexCoord;

void main()
{
  outTexCoord = vertexUV;
  gl_Position = matWorldViewProjection * vertexPosition;
}
Note that we also changed outTexCoord from varying to out. The varying keyword is deprecated from version 300 es and requires changing for the shader to work.

Note that Vertex Attribute qualifiers and #version 300 es are supported from OpenGL ES 3.0. The desktop equivalent is supported on OpenGL 3.3 and using #version 330.

Now you know your position attributes always at 0 and your texture coordinates will be at 1 and you can now bind your shader format without using glGetAttribLocation:
const int ATTRIB_POS = 0;
const int ATTRIB_UV   = 1;

glVertexAttribPointer( ATTRIB_POS, 4, GL_FLOAT, GL_FALSE, 0, 0 );
glVertexAttribPointer( ATTRIB_UV, 2, GL_FLOAT, GL_FALSE, 0, 0 );
This simple change leads to a cleaner pipeline, simpler code and saved performance during loading time.

To learn more about performance on Android, check out the Android Performance Patterns series.

Posted by Shanee Nishry, Games Developer Advocate Join the discussion on

+Android Developers
Categories: Programming

An interview with O'Reilly

While at the O'Reilly Software Architecture conference in Boston last week, I was interviewed by O'Reilly about a number of things, including the software architecture role and the tension between software architecture and code.

This interview originally appeared in Signals from the O’Reilly Software Architecture Conference 2015 that looks at some of the key insights from the event. The slides from my talk titled Software architecture vs code are available to view online/download.

Categories: Architecture

Why Do I Write Emails the Way That I Do?

In this video I respond to a somewhat negative email asking why I write emails the way that I do. The answer is simple: It’s effective.

The post Why Do I Write Emails the Way That I Do? appeared first on Simple Programmer.

Categories: Programming

Incremental Delivery of Features May Not Be Desirable

Herding Cats - Glen Alleman - 15 hours 29 min ago

There is a popular notion in agile that incremental delivery of Features allows the project to stop at any time and deliver value to those paying for the work. This is another concept of agile, in the absence of a domain and context, is pretty much meaningless in terms of actionable decision making.

We need to know ahead of time what collection of features produce what business value to the customer. Arbitrarly stopping the project may be too soon or too late. If too soon, we'll leave the current features orphaned. They won't have what is needed to provide value to the customer.

Stopping to late leaves the delivered features with partially completed work that either pollutes the functionality or prevents complete use of that functionality.

What we need is a PLAN. A business capability deployment plan like the one below for a health insurance provider network system. This system integrates legacy systems with a new Insurance ERP system.

Project Maturity Flow is the Incremental Delivery of Business Capabilities on the Planned Need Date.

Incremental Capabilities

Partial completion may provide value to the customer. Or it may not, depending on what is provided. The pilot with demo grade data conversion doesn't do the providers much good as it's not real data and it's not real enrollment. Getting the shared group matrix up and running can test the integration across the provider network. The real value starts when real data is in the Data Mart and accessible by the new ERP system.

So Once Again

When we hear the  platitudes of deliver early deliver often or incremental delivery can be stopped at any time and value delivered to the customer please ask for a picture like this and have that person point to where and what on the picture can be stopped at what time. 

Building this picture is part of Project Governance. Part of the business management planning process  of how and when to spend the customers money in an agile emerging domain where requirements certainly do change. But it's not about coding and the needs of the coders. It's about business management and managing in the presence of MicroEconomics of Software Development. This project governance paradigm should be in place for any non-trivial effort.

Principles of program governance from Glen Alleman

And the Final Notion

Of course to do this successfully we need estimates of everything in order to assure the business plan is going to be met with some degree of confidence. Time to reach a useful capability, cost to reach that capability, risks in reaching that capability as planned, resource demands, and a myriad of other random variables. With this information and a statistical model (Monte Carlo, COD, Method of Moments, Design Structure Matrix, Probabilistic Value Stream Map), we can make decsisions about the business choices.

Without these processes, any non-trivial project is just coding until time and money runs out.

Related articles Release Early and Release Often Risk Management is How Adults Manage Projects Capabilities Based Planning First Then Requirements
Categories: Project Management

Why Managers Ask for Estimates and What They Need to Know

In many of my transitioning to agile clients, the managers want to know when the project will be done. Or, they want to know how much the project will cost. (I have a new book about this, Predicting the Unpredictable: Pragmatic Approaches to Estimating Cost or Schedule.)

Managers ask for estimates because they want to know something about their ability to recognize revenue this year. How many projects can they release? What is the projected effect on revenue; customer acquisition and retention; and on service revenue (training, support, all that kind of service). We pay managers big bucks so they can project out for “a while” and plan for the business.

You need to know this in your life, too. If you are an employee, you know approximately how much money you will make in a year. You might make more if you get a bonus. You might make less if you get laid off. But, you have an idea, which allows you to budget for mortgages, vacations, and kid’s braces.

Remember, in waterfall, there was no benefit until the end of the project. You couldn’t realize any benefit from a project until it was complete: not revenue, not capitalization, not any effect on what customers saw. Nothing.

When you use agile, you have options if you can release early. Remember the potential for release frequency?

If you can do continuous deployment or even release something more often, you can realize the benefits of the project before the end.

If you are agile, you don’t need to estimate a lot to tell them when they can first receive value from your work. You can capitalize software early. Your customers can see the benefits early. You might be able to acquire more customers early.

Agile changes the benefits equation for projects.

Agile is about the ability to change. We see this at the team level clearly. When the team finishes a feature, the team is ready for the next feature. It doesn’t matter if you work in flow or timeboxes, you can change the features either for the next feature (flow) or at the next timebox. You can change what the team does.

Agile is most successful when teams finish features every day (or more often). The faster you finish a feature, the faster the team gets feedback on the feature. The more flexibility the product owners has to update/change the backlog for the team (either for flow or the next timebox). The teams do have to complete their work on a feature in a reasonable amount of time. If your cycle time gets too high, no one sees the flow of features. If you don’t get to done at the end of the iteration, you don’t get the benefit of agile. Teams need to learn how to get to done quickly on small features, so they can demo and get feedback on their progress.

What does this fast delivery/fast feedback cycle do for senior managers?

It allows senior managers to change their questions. Instead of “When will this be done?” or “How much will it cost?” senior managers can ask, “When will I see the first bit of value? Can we turn that value into revenue? When can we capitalize the work?”

Those questions change the way teams and senior management work together.

When teams do agile/lean, and they have a constant flow of features, managers don’t need “assurances” or “commitments to estimates” from the teams. Instead, the team estimates a much smaller chunk of work–time to first delivery value.

You might not know precisely when you can deliver that first value. But, as soon as the team works together if they understand agile/lean, they can create a reasonable estimate. They can update that estimate if necessary.

What else can teams do?

  • Work to a target. If the teams and the product owners know that management has a desired release date, they can work to it. Teams can track their feature flow through their systems, understanding their cycle time. They can use timeboxes for focus. They can measure how close to done they are with a product backlog burnup chart.
  • Demo as you proceed. Always demo to the product owners. Depending on the pressure for revenue/whatever, ask the senior managers to participate in the demo. That way, they can see the product’s progress as you add more features.
  • Keep the backlog item size small. It doesn’t matter how much is in the backlog if the size of every item is small. The smaller the backlog items, the easier it is for teams to estimate. It’s also easier for teams to maintain a flow of features into the ever-evolving system. Who knows? You might be done earlier than you expect.

With agile, you don’t have to set the strategy for a year, fund the projects, and expect that the projects will complete within that year. A year is a long time in almost every market. Managers might want the ability to change their strategy, and still get to a first “delivery of value” date.

Our metrics need to change. Cost to release or time to release is inadequate. They are inadequate because we can change the backlog at any time.

Instead, consider these metrics:

  • Time to release value: How long will it take us to release something for revenue? (The smaller the number, the better.)
  • Frequency of release: How often can we release? (The higher the frequency, the better.)
  • Run rate (What the team costs per unit time)
  • When you capitalize software. I will admit too much ignorance here to provide you guidance.

I have other measurement suggestions for programs in Organizing An Agile Program, Part 5: Measurements That Might Mean Something to a Program.

It’s not about #noestimates. It’s about which estimates your managers need. Managers have a fiduciary responsibility to the organization. You have the responsibility to release often, at least internally. The more often you release, the fewer time/cost estimates your managers need. The more your managers can take advantage of capitalizing software and what the software can do for the organization and the customers.

Your managers need estimates. And, they need to change the estimates they request. It’s all about your organization’s transition to agile.

Categories: Project Management

Python: matplotlib hangs and shows nothing (Mac OS X)

Mark Needham - Thu, 03/26/2015 - 01:02

I’ve been playing around with some of the matplotlib demos recently and discovered that simply copying one of the examples didn’t actually work for me.

I was following the bar chart example and had the following code:

import numpy as np
import matplotlib.pyplot as plt
 
N = 5
ind = np.arange(N)
fig, ax = plt.subplots()
menMeans = (20, 35, 30, 35, 27)
menStd =   (2, 3, 4, 1, 2)
width = 0.35       # the width of the bars
rects1 = ax.bar(ind, menMeans, width, color='r', yerr=menStd)
 
plt.show()

When I execute this script from the command line it just hangs and I don’t see anything at all.

Via a combination of different blog posts (which all suggested different things!) I ended up with the following variation of imports which seems to do the job:

import numpy as np
import matplotlib
matplotlib.use('TkAgg')
import matplotlib.pyplot as plt
 
N = 5
ind = np.arange(N)
fig, ax = plt.subplots()
menMeans = (20, 35, 30, 35, 27)
menStd =   (2, 3, 4, 1, 2)
width = 0.35       # the width of the bars
rects1 = ax.bar(ind, menMeans, width, color='r', yerr=menStd)
 
plt.show()

If I run this script a Python window pops up and contains the following image which is what I expected to happen in the first place!

2015 03 25 23 56 08

The thing to notice is that we’ve had to change the backend in order to use matplotlib from the shell:

With the TkAgg backend, which uses the Tkinter user interface toolkit, you can use matplotlib from an arbitrary non-gui python shell.

Current state: Wishing for ggplot!

Categories: Programming

Developing audio apps for Android Auto

Android Developers Blog - Wed, 03/25/2015 - 20:56

Posted by Joshua Gordon, Developer Advocate

Have you ever wanted to develop apps for the car, but found the variety of OEMs and proprietary platforms too big of a hurdle? Now with Android Auto, you can target a single platform supported by vehicles coming soon from 28 manufacturers.

Using familiar Android APIs, you can easily add a great in-car user experience to your existing audio apps, with just a small amount of code. If you’re new to developing for Auto, watch this DevByte for an overview of the APIs, and check out the training docs for an end-to-end tutorial.


Playback and custom controls
table, th, td { border: 1px solid black; border-collapse: collapse; }

Custom playback controls on NPR One and iHeartRadio.

The first thing to understand about developing audio apps on Auto is that you don’t draw your user interface directly. Instead, the framework has two well-defined UIs (one for playback, one for browsing) that are created automatically. This ensures consistent behavior across audio apps for drivers, and frees you from dealing with any car specific functionalities or layouts. Although the layout is predefined, you can customize it with artwork, color themes, and custom controls.

Both NPR One and iHeartRadio customize their UI. NPR One adds controls to mark a story as interesting, to view a list of upcoming stories, and to skip to the next story. iHeartRadio adds controls to favorite stations and to like songs. Both apps store user preferences across form factors.

Because the UI is drawn by the framework, playback commands need to be relayed to your app. This is accomplished with the MediaSession callback, which has methods like onPlay() and onPause(). All car specific functionality is handled behind the scenes. For example, you don’t need to be aware if a command came from the touch screen, the steering wheel buttons, or the user’s voice.

Browsing and recommendations
table, th, td { border: 1px solid black; border-collapse: collapse; }

Browsing content on NPR One and iHeartRadio.

The browsing UI is likewise drawn by the framework. You implement the MediaBrowserService to share your content hierarchy with the framework. A content hierarchy is a collection of MediaItems that are either playable (e.g., a song, audio book, or radio station) or browsable (e.g., a favorites folder). Together, these form a tree used to display a browsable menu of your content.

With both apps, recommendations are key. NPR One recommends a short list of in-depth stories that can be selected from the browsing menu. These improve over time based on user feedback. iHeartRadio’s browsing menu lets you pick from favorites and recommended stations, and their “For You” feature gives recommendations based on user location. The app also provides the ability create custom stations, from the browsing menu. Doing so is efficient and requires only three taps (“Create Station” -> “Rock” -> “Foo Fighters”).

When developing for the car, it’s important to quickly connect users with content to minimize distractions while driving. It’s important to note that design considerations on Android Auto are different than on a mobile device. If you imagine a typical media player on a phone, you may picture a browsable menus of “all tracks” or “all artists”. These are not ideal in the car, where the primary focus should be on the road. Both NPR One and iHeartRadio provide good examples of this, because they avoid deep menu hierarchies and lengthy browsable lists.

Voice actions for hands free operation

Voice actions (e.g., “Play KQED”) are an important part of Android Auto. You can support voice actions in your app by implementing onPlayFromSearch() in the MediaSession.Callback. Voice actions may also be used to start your app from the home screen (e.g., “Play KQED on iHeartRadio”). To enable this functionality, declare the MEDIA_PLAY_FROM_SEARCH intent filter in your manifest. For an example, see this sample app.

Next steps

NPR One and iHeartRadio are just two examples of great apps for Android Auto today. They feel like a part of the car, and look and sound great. You can extend your apps to the car today, too, and developing for Auto is easy. The framework handles the car specific functionalities for you, so you’re free to focus on making your app special. Join the discussion at http://g.co/androidautodev if you have questions or ideas to share. To get started on your app, visit developer.android.com/auto.

Join the discussion on

+Android Developers
Categories: Programming

The Convergence that Changes Everything: Data Gravity + Containers + Microservices

What will change everything this time?

  • Data Gravity (move compute to data) + 
  • Containers (lightweight computation) +
  • Microservices (composable compute modules)

That’s according to John Willis and Damon Edwards on a highly recommended recent DevOps Cafe Episode.

The high concept goes something like:

  • Containers are the killer app for microservices.

  • Data Gravity is the killer app for containers.

The logic goes something like:

Categories: Architecture

Agile Results Refresher

We live in amazing times. 

The world is full of opportunity at your fingertips.

You can inspire your mind and the art of the possible with TED talks.

You can learn anything with all of the Open Courseware from MIT or Wharton, or Coursera, or you can build your skills with The Great Courses or Udemy.

You can read about anything and fill your kindle with more books than you can read in this lifetime.

You can invest in yourself.  You can develop your intellectual horsepower, your emotional intelligence, your personal effectiveness, your communication skills, your relationship skills, and your financial intelligence.

You can develop your career, expand your experience, build your network, and grow your skills and abilities.  You can take on big hairy audacious goals.  You can learn your limits, build your strengths, and reduce your liabilities.

You can develop your body and your physical intelligence, with 4-minute work outs, P90x3 routines, Fit Bits, Microsoft Band, etc.

You can expand your network and connect with people around the world, all four corners of the globe, from all walks of life, for all sorts of reasons.

You can explore the world, either virtually through Google Earth, or take real-world epic adventures.

You can fund your next big idea and bring it to the world with Kickstarter.

You can explore new hobbies and develop your talents, your art, your music, you name it.

But where in the world will you get time?

And how will you manage your competing priorities?

And how will you find and keep your motivation?

How will you wake up strong, with a spring in your step, where all the things you want to achieve in this lifetime, pull you forward, and help you rise above the noise of every day living?

That's not how I planned on starting this post, but it's a reminder of how the world is full of possibility, and how amazing your life can be when you come alive and you begin the journey to become all that you're capable of.

How I planned to start the post was this.  It's Spring.  It's time for a refresher in the art of Agile Results to help you bring out your best.

Agile Results is a simple system for meaningful results.  It combines proven practices for productivity, time management, and personal effectiveness to help you achieve more in less time, and enjoy the process.

It's a way to spend your best time and your best energy to get your best results.

Agile Results is a way to slow down to speed up, find more fulfillment, and put your ambition into practice.

Agile Results is a way to realize your potential, and to unleash your greatest potential.  Potential is a muscle that gets better through habits.

The way to get started with Agile Results is simple.  

  1. Daily Wins.  Each day, ask yourself what are Three Wins you want for today?   Maybe it's win a raving fan, maybe it's finish that thing that's been hanging over you, or maybe it's have a great lunch.  You can ask yourself this right here, right now -- what are Three Wins you want to accomplish today?
  2. Monday Vision.  On Mondays, ask yourself what are Three Wins you want for this week?  Imagine it was Friday and you are looking back on the week, what are Three Wins that you want under your belt.  Use these Three Wins for the Week to inspire you, all week long, and pull you forward.  Each week is a fresh start.
  3. Friday Reflection.  On Friday, ask yourself, what are three things going well, and what are three things to improve?  Use these insights to get better each week.  Each week is a chance to get better at prioritizing, focusing, and creating clarity around what you value, and what others value, and what's worth spending more time on.

For bonus, and to really find a path of fulfillment, there are three more habits you can add ...

  1. Three Wins for the Month.  At the start of each month, ask yourself, what are Three Wins you want for the month?  If you know your Three Wins for the Month, you can use these to make your months more meaningful.  In fact, a great practice is to pick a theme for the month, whatever you want your month to be about, and use that to make your month matter.  And, when you combine that with your Three Wins, not only will you enjoy the month more, but at the end of the month, you'll feel a better sense of accomplishment when you can talk about your Three Wins that you achieved, whether they are your private or public victories.  And, if the month gets off track, use your Three Wins to help you get back on track.  And, each month is a fresh start.
  2. Three Wins for the Year.  At the start of the year, ask yourself what are Three Wins you want to achieve for the year?  This could be anything from get to your fighting weight to take an epic adventure to write your first book.
  3. Ask better questions.   You can do this anytime, anywhere.  Thinking is just asking and answering questions.  If you want better answers, ask better questions.  You can exponentially improve the quality of your life by asking better questions.

A simple way that I remember this is I remember to think in Three Wins:

Think in terms of Three Wins for the Day, Three Wins for the Week, Three Wins for the Month, Three Wins for the Year

Those are the core habits of Agile Results in a nutshell. 

You can start with that and be well on your way to getting better results in work and life.

If you want to really master Agile Results, you can read the book, Getting Results the Agile Way: A Personal Results System for Work and Life.

It's been a best seller in time management, and it’s helped many people around the world create a better version of themselves.

If you want to take Agile Results to the next level, start a study group and share ways that you use Agile Results with each other around how you create work-life balance, create better energy, learn more faster, and unleash your potential, while enjoying the journey, and getting more from work and life.

Share your stories and results with me, with your friends, with your family, anyone, and everyone – help everybody live a little better.

Categories: Architecture, Programming

Flaw of Averages

Herding Cats - Glen Alleman - Wed, 03/25/2015 - 15:41

There is a popular notion in agile estimating that a time series of past performance can be used to forecast the future performance of the project. Here's a clip of that time series.It is conjectured that data like this can be used to make decisions about the future. And certainly data about the past CAN be used and IS used to make decisions (forecast based decisions), but there are serious flaws in the suggested approach.

Screen Shot 2015-03-24 at 3.27.38 PM

The first flaw is the Flaw of Averages.

Sam Savage's book The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty is a good place to start. From a 2000 newspaper article, this cartoon says it all.

FlawOfAverages

But the Flaw of Averages is deeper (pun intended) than the simple cartoon. Any time series of data, like the first chart can be used to forecast the possible outcomes from past performance. The Autoregressive Integrated Moving Average function in R, can easily shows that a time series like the first chart has a huge swing in possible outcomes. Below is an example, from a much better behaved time series. In this case the Schedule Performance Index of a software intensives system.

ARIMA

When we see Poorman's solutions to estimating future outcomes, be careful not to become that poorman by spending your customers money in ways that bankrupt the project.

Buy Savage's book, google software estimating, read up on ARIMA and time series analysis, download R and all its free books and documents. Here's some more resources on estimating software projects.

Don't fall prey to simple and many times simple minded approaches to managing other peoples money. Math is required when dealing with probabilistic processes found on ALL projects. The future is always uncertain and NEVER the same as the past. This is the case even of production processes found at Toyota. Margin and risk reduction activities are always needed. Knowing how much margin and what risk reduction activities is past of the planning process for any non-trivial project.

Related articles Empirical Data Used to Estimate Future Performance Five Estimating Pathologies and Their Corrective Actions Calculating Value from Software Projects - Estimating is a Risk Reduction Process Failure is not an Option Managing in the Presence of Uncertainty The Cost Estimating Problem Estimating Probabilistic Outcomes? Of Course We Can! Forecast, Automatic Routines vs. Experience
Categories: Project Management

How to Turn a Bad Presentation Into a Good One

NOOP.NL - Jurgen Appelo - Wed, 03/25/2015 - 13:10
How to Turn a Bad Presentation into a Good One

For three weeks, I had been working on a new presentation: New stories, new message, new visuals, new format, new jokes… everything was new. Even my hair was new. The idea was to reinvent myself as a public speaker. Raising the bar!

The post How to Turn a Bad Presentation Into a Good One appeared first on NOOP.NL.

Categories: Project Management

Quote of the Month March 2015

From the Editor of Methods & Tools - Wed, 03/25/2015 - 09:10
Competencies versus Roles: We’ve seen a positive move toward emphasizing competencies in a team rather than roles or titles. As teams make that change, we see fewer “It’s not my job” excuses and more “How can I help?” conversations. Team members will continue to have core competencies in some areas more than others, but they may not identify as strongly with a particular role. For example, saying, “I am a tester” really means, “I perform mainly testing activities because that is my primary passion and strength. I can provide leadership ...

CMMI: The Easy Way or The Hard Way? A Simple CMMI Readiness Checklist

Are you ready to build change?

Are you ready to build change?

Many times an organization or individual will start a change program because they deem it necessary for survival. But change is never easy. Survival and pain avoidance, while powerful, can lead to pursuing change as a reaction to pain rather than as pursuit of value. Pain avoidance and generation of business value both are necessary pieces of knowledge as the intellectual benefits persuade, while pain avoidance sells. In order to ensure that both sides of the change are addressed a framework can be useful to generate focus. The simple CMMI Readiness Checklist can be used for any major change initiative, but is tailored toward the testing whether the requirements for implementing a framework like the CMMI have been addressed.

I have broken this checklist into three categories: resources, plans and attitudes.  Each can be leveraged separately; however using the three components will help you to focus on the big picture.

Scale

The simple checklist can be used as a tool to evaluate how well you have prepared for you CMMI journey using the questions as evaluation criteria.  To use the checklist, evaluate each question on a scale of high, medium, low and not present (with one exception). Each question will potentially contribute points toward the total that can be used to evaluate preparation.

Section and Question Weights:

Resources: Forty-two total points. Each component contributes up to 7 points (7, 3, 1, 0).

Plans: Eighteen total points. Each component contributes up to 6 points (6, 3, 1, 0).

Attitude: Forty total points. Each component contributes up to 8 points (8, 4, 2, 0).

Resources

Resources are the raw materials that you will consume on your journey.  As with any journey having both the correct resources and correct amount of resources will make the journey easier.  Just think of trying to canoe from New York to London for a meeting; the wrong resources can make the trip difficult.

Management Support

Support from management is critical as we have discussed in past checklists, but so is support from your peers and from the teams that will be using the processes.

Score

7 – Senior management is actively involved in guiding and using the outputs of the CMMI.  Senior managers stop people in the hall to discuss progress and recently process implementations. Discussion of progress is an agenda item at all managers staff meetings.

3 – Senior and middle managers attend formal CMMI informational meetings and talk about the need to support the CMMI initiative.

1 – Senior managers attended the kick-off meeting, then relocating in mass to Aruba, leaving the middle managers in charge.

0 – The change initiative is a grass-roots effort.

Cash

Change costs money. Costs can include consultants, training, travel and an odd late-night pizza or two.

7 – A reasonable budget has been established and the implementation team can draw from the budget for planned expenditures.  Emergency funding can be attained to handle issues.

3 – A reasonable budget has been established and approved; however, access must be requested and justified for all expenditures.

1 – Any time that money is required funding must be requested and approved.

0 – Donations are sought in the organization’s lunchroom on a periodic basis (consider a PayPal donation button on your homepage).

Effort

Even if you have bales of cash, developing and implementing processes will require effort. Effort will be required from many constituencies including the process-improvement team, management and from the teams using the process, just to name a few.

7 – A reasonable staffing plan has been established and the change program is the only project the assigned resources have been committed to.

4 – A reasonable staffing plan has been established and the change initiative is the highest priority for the assigned resources.

1 – All resources are shared between the change initiative and are also assigned to other projects with high priority.

0 – You have all the effort you need after 5 PM and before 8 AM and during company holidays.

Change Specialist

Organizational change requires skills that are not generally found in an IT department. The skills needed include sales, marketing and communication.

7 – An organizational-change specialist has been assigned as a full-time resource for the project.

3 – An organizational-change specialist is available within the organization and works on many projects simultaneously. The specialist may or may not have had experience with IT change programs.

1 – Someone on the team has helped craft an organizational change plan in the past.

0 – Organizational change websites are blocked and your best bet is buying a book on Amazon using your own cash.

Projects

Change requires something to impact.  The organization needs to have a consistent flow of projects so that changes are not one-shot attempts.

7 – Projects are constantly beginning that will provide a platform for implementing process changes.

3 – There are numerous projects in the organization; however they typically begin early in the year or on some other periodic basis that makes waiting a necessity if you are not ready exactly on time.

1 – The organization does only a small number of projects every year.

0 – The organization does one large project every year.

Calendar Time

Calendar time is a resource that is as important as any other resource. Severe calendar constraints can lead to irrational or bet-the-farm behaviors which increase risk.

7 – The schedule for implementing the CMMI is in line with industry norms and includes time for tweaking the required processes before appraising.

3 – The schedule is realistic but bare bones. Any problems could cause delay.

1 – Expectations have been set that will require a compressed schedule; however, delay will only be career limiting rather than a critical impact on the business.

0 – The CMMI implantation is critical for the organization’s survival and is required on an extremely compressed schedule.

Expertise

A deep understanding of the CMMI (or any other framework for that matter) will be needed to apply the model in a dynamic environment.  Experience is generally hard won. “Doing” it once generally does not provide enough expertise to allow the level of tailoring needed to apply the model in more than one environment. Do not be afraid to get a mentor if this is a weakness.

7 – The leaders and team members working to implement the CMMI have been intimately involved in successfully implementing the framework in different environments.

3 –The leader and at least one of the team members have been involved in implementing the CMMI in the past in a similar environment.

1 – Only the leader of the CMMI program has been involved with implementing the CMMI in another environment.

0 – All of the team members have taken the basic CMMI course and can spell CMMI assuming they can buy a vowel.

Plans

Planning for the implementation of change can take many forms — from classic planning documents and schedules to backlogs.  The structure of the plan is less of a discussion point than the content.  You need several plans when changing an organization. While the term “several” is used this does not mandate many volumes of paper and schedules, rather that the activities required are thought through and recorded, the goal is known and the constraints on the program have been identified (in other words the who, what, when, why and how are known to the level required).

Scale and Scoring

Plans: Eighteen total points. Each component contributes up to 6 points (6, 3, 1, 0).

Organizational Change Plan

The Organizational Change Plan includes information on how the changes required to implement the CMMI will be communicated, marketed, reported, discussed, supported, trained and, if necessary escalated.

6 – A full change management plan has been developed, implemented and is being constantly monitored.

3 –An Organizational Change Plan is planned but is yet to be developed. .

1 – When created, the Organizational Change Plan will be referenced occasionally.

0 – No Organizational Change Plan has or will be created.

Backlog

The backlog records what needs to be changed in prioritized order. The backlog should include all changes, issues and risks. The items in the backlog will be broken down into tasks as they are selected to be worked on.  The format needs to match corporate culture and can range from an Agile backlog to in a waterfall organization, a requirements document.

6 – A prioritized backlog exists and is constantly maintained.

3 – A prioritized backlog exists and is periodically maintained.

1 – A rough list of tasks and activities is kept on whiteboard.

0 – No backlog or list of tasks exists.

Governance

Any change program requires resources, perseverance and political capital. In most corporations these types of requirements scream the need for oversight (governance is a code word for the less friendly word oversight). Governance defines who decides which changes will be made, when they will be made and who will pay for the changes. I strongly recommend that you decide how governance will be handled and write it down and make sure all of your stakeholders are comfortable on how you will get their advice, counsel, budget and in some cases permission.

6 – A full-governance plan has been developed, implemented and is being constantly monitored.

3 –A governance plan is planned, but is yet to be developed.

1 – When created, the governance plan will be used to show the process auditors.

0 – Governance . . . who needs it!

Attitude

When you talk about attitude it seems personal rather than organizational, but when it comes to large changes I believe that both the attitude of the organization and critical individuals are important.  As you prepare to address the CMMI, the onus is on you as a change leader to develop a nuanced understanding of who you need to influence within the organization. The checklist will portray an organizational view; however, you can and should replicate the exercise for specific critical influencers.

Scale and Scoring

Attitude: Forty total points. Each component contributes up to 8 points (8, 4, 2, 0).

Vision of tomorrow

Is there a belief that tomorrow will be demonstratively better based on the actions that are being taken? The organization needs to have a clear vision that tomorrow will be better than today in order to positively motivate the team to aspire to be better than they are.

8 – The organization is excited about the changes that are being implemented.  Volunteers to help or to pilot are numerous.

4 – Most of the organization is excited about most of the changes and their impact on the future.

2 – A neutral outlook (or at least undecided) is present.

0 – Active disenchantment with or dissension about the future is present.

Minimalist

The view that the simplest process change that works is the best is important in today’s lean world.  In many cases heavy processes are wearing on everyone who uses them and even when the process is okay today, entropy will add steps and reviews over time, which adds unneeded weight.  Score this attribute higher if the organization has a process to continually apply lean principles as a step in process maintenance.

8 – All processes are designed with lean principles formally applied.  Productivity and throughput are monitored to ensure that output isn’t negatively impacted.

4 – All processes are designed with lean principles formally applied; however, they are not monitored quantitatively.

2 – All processes are designed with lean principles informally applied.

0 – Processes are graded by the number of steps required, with a higher number being better.

Learner

A learner is someone that is learning understands that they don’t know everything and that mistakes will be made. They understand that when made, mistakes are to be examined and corrected rather than swept under the carpet. Another attribute of a learner is the knowledge that the synthesis of data and knowledge from other sources is required for growth.  In most organizations, an important source of process knowledge and definition are the practitioners — but not the sole source.

8 – New ideas are actively pursued and evaluated on an equal footing with any other idea or concept.

4 – New ideas are actively pursued and evaluated, but those that reflect the way work is currently done are given more weight.

2 – The “not invented here” point of view has a bit of a hold on the organization, making the introduction of new ideas difficult.

0 – There is only one way to do anything and it was invented here sometime early last century.  Introduction of new ideas is considered dangerous.

Goal Driven

The organization needs to have a real need to drive the change and must be used to pursuing longer-term goals. The Process Philosopher of Sherbrooke once told me that being goal-driven is required to be serious about change.  In many cases a good, focused near-death experience increases the probability of change, but waiting that long can create a negative atmosphere. A check-the-box goal rarely provides more than short-term localized motivation.

8 – The organization has a well-stated positive goal and that the CMMI not only supports, but is integral to attaining that goal.

2 – The pursuit of the CMMI is about checking a box on a RFP response.

0 – CMMI is being pursued for no apparent purpose.

Conviction

Belief in the underlying concepts of the CMMI (or other change framework) provides motivation to the organization and individuals.  Conviction creates a scenario where constancy of purpose (Deming) is not an after-thought but the way things are done. Implementing frameworks like the CMMI are long-term efforts — generally with levels of excitement cycling through peaks and valleys.  In the valley when despair becomes a powerful force, many times conviction is the thread that keeps things moving forward. Without a critical mass of conviction it will be easy to wander off to focus on the next new idea.

8 – We believe and have evidence that from the past that we can continue to believe over time.

4 – We believe but this is the first time we’ve attempted something this big!

2 – We believe  . . . mostly.

0 – No Organizational-Change Plan has been created.

Scoring

Sum all of the scores and apply the following criteria.

100 – 80   You have a great base; live the dream.

79 – 60   Focus on building your change infrastructure as you begin the CMMI journey.

59 – 30   Remediate your weaknesses before you start wrestling with the CMMI.

29 –   0   Run Away! Trying to implement the CMMI will be equivalent to putting your hand in the garbage disposal with it running; avoid if you absolutely can!


Categories: Process Management

Topic Modelling: Working out the optimal number of topics

Mark Needham - Tue, 03/24/2015 - 23:33

In my continued exploration of topic modelling I came across The Programming Historian blog and a post showing how to derive topics from a corpus using the Java library mallet.

The instructions on the blog make it very easy to get up and running but as with other libraries I’ve used, you have to specify how many topics the corpus consists of. I’m never sure what value to select but the authors make the following suggestion:

How do you know the number of topics to search for? Is there a natural number of topics? What we have found is that one has to run the train-topics with varying numbers of topics to see how the composition file breaks down. If we end up with the majority of our original texts all in a very limited number of topics, then we take that as a signal that we need to increase the number of topics; the settings were too coarse.

There are computational ways of searching for this, including using MALLETs hlda command, but for the reader of this tutorial, it is probably just quicker to cycle through a number of iterations (but for more see Griffiths, T. L., & Steyvers, M. (2004). Finding scientific topics. Proceedings of the National Academy of Science, 101, 5228-5235).

Since I haven’t yet had the time to dive into the paper or explore how to use the appropriate option in mallet I thought I’d do some variations on the stop words and number of topics and see how that panned out.

As I understand it, the idea is to try and get a uniform spread of topics -> documents i.e. we don’t want all the documents to have the same topic otherwise any topic similarity calculations we run won’t be that interesting.

I tried running mallet with 10,15,20 and 30 topics and also varied the stop words used. I had one version which just stripped out the main characters and the word ‘narrator’ & another where I stripped out the top 20% of words by occurrence and any words that appeared less than 10 times.

The reason for doing this was that it should identify interesting phrases across episodes better than TF/IDF can while not just selecting the most popular words across the whole corpus.

I used mallet from the command line and ran it in two parts.

  1. Generate the model
  2. Work out the allocation of topics and documents based on hyper parameters

I wrote a script to help me out:

#!/bin/sh
 
train_model() {
  ./mallet-2.0.7/bin/mallet import-dir \
    --input mallet-2.0.7/sample-data/himym \
    --output ${2} \
    --keep-sequence \
    --remove-stopwords \
    --extra-stopwords ${1}
}
 
extract_topics() {
  ./mallet-2.0.7/bin/mallet train-topics \
    --input ${2} --num-topics ${1} \
    --optimize-interval 20 \
    --output-state himym-topic-state.gz \
    --output-topic-keys output/himym_${1}_${3}_keys.txt \
    --output-doc-topics output/himym_${1}_${3}_composition.txt
}
 
train_model "stop_words.txt" "output/himym.mallet"
train_model "main-words-stop.txt" "output/himym.main.words.stop.mallet"
 
extract_topics 10 "output/himym.mallet" "all.stop.words"
extract_topics 15 "output/himym.mallet" "all.stop.words"
extract_topics 20 "output/himym.mallet" "all.stop.words"
extract_topics 30 "output/himym.mallet" "all.stop.words"
 
extract_topics 10 "output/himym.main.words.stop.mallet" "main.stop.words"
extract_topics 15 "output/himym.main.words.stop.mallet" "main.stop.words"
extract_topics 20 "output/himym.main.words.stop.mallet" "main.stop.words"
extract_topics 30 "output/himym.main.words.stop.mallet" "main.stop.words"

As you can see, this script first generates a bunch of models from text files in ‘mallet-2.0.7/sample-data/himym’ – there is one file per episode of HIMYM. We then use that model to generate differently sized topic models.

The output is two files; one containing a list of topics and another describing what percentage of the words in each document come from each topic.

$ cat output/himym_10_all.stop.words_keys.txt
 
0	0.08929	back brad natalie loretta monkey show call classroom mitch put brunch betty give shelly tyler interview cigarette mc laren
1	0.05256	zoey jerry arthur back randy arcadian gael simon blauman blitz call boats becky appartment amy gary made steve boat
2	0.06338	back claudia trudy doug int abby call carl stuart voix rachel stacy jenkins cindy vo katie waitress holly front
3	0.06792	tony wendy royce back jersey jed waitress bluntly lucy made subtitle film curt mosley put laura baggage officer bell
4	0.21609	back give patrice put find show made bilson nick call sam shannon appartment fire robots top basketball wrestlers jinx
5	0.07385	blah bob back thanksgiving ericksen maggie judy pj valentine amanda made call mickey marcus give put dishes juice int
6	0.04638	druthers karen back jen punchy jeanette lewis show jim give pr dah made cougar call jessica sparkles find glitter
7	0.05751	nora mike pete scooter back magazine tiffany cootes garrison kevin halloween henrietta pumpkin slutty made call bottles gruber give
8	0.07321	ranjit back sandy mary burger call find mall moby heather give goat truck made put duck found stangel penelope
9	0.31692	back give call made find put move found quinn part ten original side ellen chicago italy locket mine show
$ head -n 10 output/himym_10_all.stop.words_composition.txt
#doc name topic proportion ...
0	file:/Users/markneedham/projects/mallet/mallet-2.0.7/sample-data/himym/1.txt	0	0.70961794636687	9	0.1294699168584466	8	0.07950442338871108	2	0.07192178481473664	4	0.008360809510263838	5	2.7862560133367015E-4	3	2.562409242784946E-4	7	2.1697378721335337E-4	1	1.982849604752168E-4	6	1.749937876710496E-4
1	file:/Users/markneedham/projects/mallet/mallet-2.0.7/sample-data/himym/10.txt	2	0.9811551470820473	9	0.016716882136209997	4	6.794128563082893E-4	0	2.807350575301132E-4	5	2.3219634098530471E-4	8	2.3018997315244256E-4	3	2.1354177341696056E-4	7	1.8081798384467614E-4	1	1.6524340216541808E-4	6	1.4583339433951297E-4
2	file:/Users/markneedham/projects/mallet/mallet-2.0.7/sample-data/himym/100.txt	2	0.724061485807234	4	0.13624729774423758	0	0.13546964196228636	9	0.0019436342339785994	5	4.5291919356563914E-4	8	4.490055982996677E-4	3	4.1653183421485213E-4	7	3.5270123154213927E-4	1	3.2232165301666123E-4	6	2.8446074162457316E-4
3	file:/Users/markneedham/projects/mallet/mallet-2.0.7/sample-data/himym/101.txt	2	0.7815231689893246	0	0.14798271520316794	9	0.023582384458063092	8	0.022251052243582908	1	0.022138209217973336	4	0.0011804626661380394	5	4.0343527385745457E-4	3	3.7102343418895774E-4	7	3.1416667687862693E-4	6	2.533818368250992E-
4	file:/Users/markneedham/projects/mallet/mallet-2.0.7/sample-data/himym/102.txt	6	0.6448245189567259	4	0.18612146979166502	3	0.16624873439661025	9	0.0012233726722317548	0	3.4467218590717303E-4	5	2.850788252495599E-4	8	2.8261550915084904E-4	2	2.446611421432842E-4	7	2.2199909869250053E-4	1	2.028774216237081E-
5	file:/Users/markneedham/projects/mallet/mallet-2.0.7/sample-data/himym/103.txt	8	0.7531586740033047	5	0.17839539108961253	0	0.06512376460651902	9	0.001282794040111701	4	8.746645156304241E-4	3	2.749100345664577E-4	2	2.5654476523149865E-4	7	2.327819863700214E-4	1	2.1273153572848481E-4	6	1.8774342292520802E-4
6	file:/Users/markneedham/projects/mallet/mallet-2.0.7/sample-data/himym/104.txt	7	0.9489502365148181	8	0.030091466847852504	4	0.017936457663121977	9	0.0013482824985091328	0	3.7986419553884905E-4	5	3.141861834124008E-4	3	2.889445824352445E-4	2	2.6964174000656E-4	1	2.2359178288566958E-4	6	1.9732799141958482E-4
7	file:/Users/markneedham/projects/mallet/mallet-2.0.7/sample-data/himym/105.txt	8	0.7339694064061175	7	0.1237041841318045	9	0.11889696041555338	0	0.02005288536233353	4	0.0014026751618923005	5	4.793786828705149E-4	3	4.408655780020889E-4	2	4.1141370625324785E-4	1	3.411516484151411E-4	6	3.0107890675777946E-4
8	file:/Users/markneedham/projects/mallet/mallet-2.0.7/sample-data/himym/106.txt	5	0.37064909999661005	9	0.3613559917055785	0	0.14857567731040344	6	0.09545466082502917	4	0.022300625744661403	8	3.8725629469313333E-4	3	3.592484711785775E-4	2	3.3524900189121E-4	7	3.041961449432886E-4	1	2.779945050112539E-4

The output is a bit tricky to understand on its own so I did a bit of post processing using pandas and then ran the results of that through matplotlib to see the distribution of documents for different topics sizes with different stop words. You can see the script here.

I ended up with the following chart:

2015 03 24 22 08 48

On the left hand side we’re using more stop words and on the right just the main ones. For most of the variations there are one or two topics which most documents belong to but interestingly the most uniform distribution seems to be when we have few topics.

These are the main words for the most popular topics on the left hand side:

15 topics

8       0.50732 back give call made put find found part move show side ten mine top abby front fire full fianc

20 topics

12      0.61545 back give call made put find show found part move side mine top front ten full cry fire fianc

30 topics

22      0.713   back call give made put find show part found side move front ten full top mine fire cry bottom

All contain more or less the same words which at first glance seem like quite generic words so I’m surprised they weren’t excluded.

On the right hand side we haven’t removed many words so we’d expect common words in the English language to dominate. Let’s see if they do:

10 topics

1       3.79451 don yeah ll hey ve back time guys good gonna love god night wait uh thing guy great make

15 topics

5       2.81543 good time love ll great man guy ve night make girl day back wait god life yeah years thing
 
10      1.52295 don yeah hey gonna uh guys didn back ve ll um kids give wow doesn thing totally god fine

20 topics

1       3.06732 good time love wait great man make day back ve god life years thought big give apartment people work
 
13      1.68795 don yeah hey gonna ll uh guys night didn back ve girl um kids wow guy kind thing baby

30 topics

14      1.42509 don yeah hey gonna uh guys didn back ve um thing ll kids wow time doesn totally kind wasn
 
24      2.19053 guy love man girl wait god ll back great yeah day call night people guys years home room phone
 
29      1.84685 good make ve ll stop time made nice put feel love friends big long talk baby thought things happy

Again we have similar words across each run and as expected they are all quite generic words.

My take away from this exploration is that I should vary the stop word percentages as well and see if that leads to an improved distribution.

Taking out very common words like we do with the left hand side charts seems to make sense although I need to work out why there’s a single outlier in each group.

The authors suggest that having the majority of our texts in a small number of topics means we need to create more of them so I will investigate that too.

The code is all on github along with the transcripts so give it a try and let me know what you think.

Categories: Programming

Two Programmers Turned WordPress Entrepreneurs Profiled

Making the Complex Simple - John Sonmez - Tue, 03/24/2015 - 20:18

I’ve been involved with the WordPress community quite a bit, since I’ve launched my “How to Create a Blog to Boost Your Career” course. As I result, I’ve been very interested in what in happening in the WordPress development world. I’ve gotten a few requests for information about WordPress development, but I’m really not an […]

The post Two Programmers Turned WordPress Entrepreneurs Profiled appeared first on Simple Programmer.

Categories: Programming

Calculating Value from Software Projects - Estimating is a Risk Reduction Process

Herding Cats - Glen Alleman - Tue, 03/24/2015 - 15:41

Software can provide increased benefits to internal users, with the firm paying for the development or acquisition of that software. Software can also reduce costs to the users in exchange for the cost of development or acquisition of the software.

This exchange of benefit for cost is the basis of all business decision making where the governance of resources is in place. This governance process assumes there are limited resources - cost, time, capacity for work, available labor, equipment, and any other element that enters in the production of output of the firm. 

Benefits produced in exchange for this cost also include risk reduction, as well as increased revenue, and lowered cost. But again this risk reduction is in exchange for cost - money paid for reduced risk.

To determine benefits we can use a decision framework †

  • Business scope - measures deliverable benefits in either cost or revenue.
  • Value and Potential Impact:
    • Improvement benefits — based on existing valuation tools or what if calculations.
    • Scaling benefits — the total value of the associated business change resulting from the software acquisition and deployment.
    • Risk reduction benefits — the expected value of the risks being mitigates.
    • Future options — using Black-Scholes, Real Options, or Monte Carlo Simulation to determine future benefits.
  • Project cost - assess using standard project costing tools.
  • Direct savings — through operational cost reduction.
  • Real cash effective — netted off against direct savings.

Making Decisions in the Presence of Uncertainty ‡

It is better to be approximately right than precisely wrong - Warren Buffet

This quote is many times misused to say we can't possibly estimate so let's not estimate, let's just get started coding and we'll see what comes out. 

Since all project work is based on uncertainty — reducible and irreducible — risk is the result of this uncertainty. So first we need to determine what is the Value At Risk before we can say how we are going to manage in the presence of this uncertainty. 

Risk is a quantity that has relevance on it's own. What's the risk that if I make this next turn on a Double-Black run in Breckenridge, there will be ice and I'll fall? Hard to say on the first run of the day, so I'll slow down as I come around the tree line. On the third run in a row, I've got the experience on that run today and the experience of skiing fast and hard on that run for several years to know more about the risk.

Since statistical uncertainty drives projects, the resulting risk from that uncertainty is probabilistic. When interdependencies between work elements, capacity for work, technical and operational processes, changing understanding, and a myriad of other variables are also uncertain, we are faced with the problem. This problem is no deterministic assessment of time, cost, or capabilities can be performed for our project. We must have a probabilistic process. And of course this probabilistic process requires estimating the range of values possible for each variable that interacts with the other variables.

Sampling from the past (empirical data) is a good way to start, but those past samples tell us nothing about the future statistical process unless all work in the future is identical to the work in the past. This is naive at best and dangerous at worst. Such naive assumptions are many times the Root Cause of major cost overruns in our space and defense software intensive systems business. Those same naive assumptions are applicable across all software domains.

When there is a network of work activities - as there is on any non-trivial project - each activity is a probabilistic process. The notion of Independent work in the agile paradigm must always be confirmed before assuming simple queuing process can be put to work. So when you hear about Little's Law and Bootstrapping simulations, confirm the interdependencies of the work. The model in the chart below and the Probability Distribution Function below that are from a Monte Carlo Simulation, where the interdependencies of the work activities are modeled by the tool. In this case RiskyProject that provides a Risk Register for reducible risks, the means of modeling the Irreducible uncertainty in the duration of the work, shows the coupling between the work elements Cruciality Index and other indicators of trouble to come from the status of the past performance.

Screen Shot 2015-03-24 at 8.17.58 AM

The chart below says that the activity being modeled (and all the activities in the network of activities are modeled, I just picked this one), has a 54% chance of completing on or before Oct 18th, 2015. 

Probabilistic Finish

The End

If you're project is small enough to be able to see all the work in one place, see how to produce outcomes from all that work, and has a low value at risk, making business based estimates using risk reduction is probably not needed. 

If you project has interdependencies between the work elements, the work is of sufficient duration you can't see the end in enough detail to remove all the statistical variances, and the value at risk is sufficiently high that the business impact of not showing up on time, on budget, with the needed outcomes will be unacceptable - then the process of managing in the presence of uncertainty must be able to estimate all the interacting variables. 

It's this simple

  • No risk, low impact for being wrong, low value at risk project — no need to worry about the future, just start  and work out problems as you go.

But when we hear we don't need to estimate to make decisions and no domain and context is provided, those making that conjecture haven't considered the domain or context either. They're either unaware of the probability and statistics projects or they're intentionally ignoring the probability and statistics of projects. Since they ignore these fundamental process of all non-trivial project, ignore their advice.

‡ How to Measure Anything: Finding Value of Intangibles in Business 3rd Edition, Douglas Hubbard, 

† A New Framework for IT Investment Decisions, Anthony Barnes, 

Related articles Managing Projects By The Numbers Empirical Data Used to Estimate Future Performance Five Estimating Pathologies and Their Corrective Actions Managing in Presence of Uncertainty I Think You'll Find It's a Bit More Complicated Than That A Monte Carlo Simulation for Pi Day
Categories: Project Management

Announcing the new Azure App Service

ScottGu's Blog - Scott Guthrie - Tue, 03/24/2015 - 15:23

In a mobile first, cloud first world, every business needs to deliver great mobile and web experiences that engage and connect with their customers, and which enable their employees to be even more productive.  These apps need to work with any device, and to be able to consume and integrate with data anywhere.

I'm excited to announce the release of our new Azure App Service today - which provides a powerful new offering to deliver these solutions.  Azure App Service is an integrated service that enables you to create web and mobile apps for any platform or device, easily integrate with SaaS solutions (Office 365, Dynamics CRM, Salesforce, Twilio, etc), easily connect with on-premises applications (SAP, Oracle, Siebel, etc), and easily automate businesses processes while meeting stringent security, reliability, and scalability needs. Azure App Service

Azure App Service includes the Web App + Mobile App capabilities that we previously delivered separately (as Azure Websites + Azure Mobile Services).  It also includes powerful new Logic/Workflow App and API App capabilities that we are introducing today for the very first time - along with built-in connectors that make it super easy to build logic workflows that integrate with dozens of popular SaaS and on-premises applications (Office 365, SalesForce, Dynamics, OneDrive, Box, DropBox, Twilio, Twitter, Facebook, Marketo, and more). 

All of these features can be used together at one low price.  In fact, the new Azure App Service pricing is exactly the same price as our previous Azure Websites offering.  If you are familiar with our Websites service you now get all of the features it previously supported, plus additional new mobile support, plus additional new workflow support, plus additional new connectors to dozens of SaaS and on-premises solutions at no extra charge

Web + Mobile + Logic + API Apps

Azure App Service enables you to easily create Web + Mobile + Logic + API Apps:

image

You can run any number of these app types within a single Azure App Service deployment.  Your apps are automatically managed by Azure App Service and run in managed VMs isolated from other customers (meaning you don't have to worry about your app running in the same VM as another customer).  You can use the built-in AutoScaling support within Azure App Service to automatically increase and decrease the number of VMs that your apps use based on the actual resource consumption of them. 

This provides an incredibly cost-effective way to build and run highly scalable apps that provide both Web and Mobile experiences, and which contain automated business processes that integrate with a wide variety of apps and data sources.

Below are additional details on the different app types supported by Azure App Service.  Azure App Service is generally available starting today for Web apps, with the Mobile, Logic and API app types available in public preview:

Web Apps

The Web App support within Azure App Service includes 100% of the capabilities previously supported by Azure Websites.  This includes:

  • Support for .NET, Node.js, Java, PHP, and Python code
  • Built-in AutoScale support (automatically scale up/down based on real-world load)
  • Integrated Visual Studio publishing as well as FTP publishing
  • Continuous Integration/Deployment support with Visual Studio Online, GitHub, and BitBucket
  • Virtual networking support and hybrid connections to on-premises networks and databases
  • Staged deployment and test in production support
  • WebJob support for long running background tasks

Customers who have previously deployed an app using the Azure Website service will notice today that they these apps are now called "Web Apps" within the Azure management portals.  You can continue to run these apps exactly as before - or optionally now also add mobile + logic + API app support to your solution as well without having to pay anything more.

Mobile Apps

The Mobile App support within Azure App Service provides the core capabilities we previously delivered using Azure Mobile Services.  It also includes several new enhancements that we are introducing today including:

  • Built-in AutoScale support (automatically scale up/down based on real-world load)
  • Traffic Manager support (geographically scale your apps around the world)
  • Continuous Integration/Deployment support with Visual Studio Online, GitHub, and BitBucket
  • Virtual networking support and hybrid connections to on-premises databases
  • Staged deployment and test in production support
  • WebJob support for long running background tasks

Because we have an integrated App Service offering, you can now run both Web and Mobile Apps using a single Azure App Service deployment.  This allows you to avoid having to pay for a separate web and mobile backend - and instead optionally pool your resources to save even more money.

Logic Apps

The Logic App support within Azure App Services is brand new and enables you to automate workflows and business processes.  For example, you could configure a workflow that automatically runs every time your app calls an API, or saves data within a database, or on a timer (e.g. once a minute) - and within your workflows you can do tasks like create/retrieve a record in Dynamics CRM or Salesforce, send an email or SMS message to a sales-rep to follow up on, post a message on Facebook or Twitter or Yammer, schedule a meeting/reminder in Office 365, etc. 

Constructing such workflows is now super easy with Azure App Services.  You can define a workflow either declaratively using a JSON file (which you can check-in as source code) or using the new Logic/Workflow designer introduced today within the Azure Portal.  For example, below I've used the new Logic designer to configure an automatically recurring workflow that runs every minute, and which searches Twitter for tweets about Azure, and then automatically send SMS messages (using Twilio) to have employees follow-up on them:

image 

Creating the above workflow is super easy and takes only a minute or so to do using the new Logic App designer.  Once saved it will automatically run within the same VMs/Infrastructure that the Web Apps and Mobile Apps you've built using Azure App Service use as well.  This means you don't have to deploy or pay for anything extra - if you deploy a Web or Mobile App on Azure you can now do all of the above workflow + integration scenarios at no extra cost

Azure App Service today includes support for the following built-in connectors that you can use to construct and automate your Logic App workflows:

image

Combined the above connectors provide a super powerful way to build and orchestrate tasks that run and scale within your apps.  You can now build much richer web and mobile apps using it.

Watch this Azure Friday video about Logic Apps with Scott Hanselman and Josh Twist to learn more about how to use it.

API Apps

The API Apps support within Azure App Service provides additional support that enables you to easily create, consume and call APIs - both APIs you create (using a framework like ASP.NET Web API or the equivalent in other languages) as well as APIs from other SaaS and cloud providers.

API Apps enable simple access control and credential management within your applications, as well as automatic SDK generation support that enables you to easily expose and integrate APIs across a wide-variety of languages.  You can optionally integrate these APIs with Logic Apps. Getting Started

Getting started with Azure App Service is easy.  Simply sign-into the Azure Preview Portal and click the "New" button in the bottom left of the screen.  Select the "Web + Mobile" sub-menu and you can now create Web Apps, Mobile Apps, Logic Apps, and API Apps:

image 

You can create any number of Web, Mobile, Logic and API apps and run them on a single Azure App Service deployment at no additional cost. 

Learning More

I'll be hosting a special Azure App Service launch event online on March 24th at 11am PDT which will contain more details about Azure App Service, a great demo from Scott Hanselman, and talks by several customers and analytics talking about their experiences.  You can watch the online event for free here.

Also check out our new Azure Friday App Service videos with Scott Hanselman that go into detail about all of the new capabilities, and show off how to build Web, Mobile, Logic and API Apps using Azure App Service:

Then visit our documentation center to learn more about the service and how to get started with it today.  Pricing details are available here.

Summary

Today’s Microsoft Azure release enables a ton of great new scenarios, and makes building great web and mobile applications hosted in the cloud even easier.

If you don’t already have a Azure account, you can sign-up for a free trial and start using all of the above features today.  Then visit the Microsoft Azure Developer Center to learn more about how to build apps with it.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at:twitter.com/scottgu omni

Categories: Architecture, Programming

A High Available Docker Container Platform using CoreOS and Consul

Xebia Blog - Tue, 03/24/2015 - 11:35

Docker containers are hot, but containers in themselves are not very interesting. It needs an eco-system to make it into  24x7 production deployments. Just handing your container names to operations, does not cut it.

In the blog post, we will show you how  CoreOS can be used to provide a High Available Docker Container Platform as a Service, with a box standard way to deploy Docker containers. Consul is added to the mix to create a lightweight HTTP Router to any docker application offering a HTTP service.

We will be killing a few processes and machine on the way to prove our point...

Architecture

The basic architecture for our Docker Container Platform as a Service, consists of the following components

coreos-caas

  • CoreOS cluster
    The CoreOS cluster will provide us with a cluster of Highly Available Docker Hosts. CoreOS is an open source lightweight operating system based on the Linux kernel and provides an infrastructure for clustered deployments of applications. The interesting part of CoreOS is that you cannot install applications or packages on CoreOS itself. Any custom application has to be packed and deployed as a Docker container. At the same time CoreOS provides only basic functionality for managing these applications.
  • Etcd
    etcd is the CoreOS distributed key value store and provides a reliable mechanism to distribute data through the cluster.
  • Fleet
    Fleet is the cluster wide init system of CoreOS which allows you to schedule applications to run inside the Cluster and provides the much needed nanny system for you apps.
  • Consul
    Consul from Hashicorp is a tool that eases service discovery and configuration. Consul allows services to be discovered via DNS and HTTP and provides us with the ability to respond to changes in the service registration.
  • Registrator
    The Registrator from Gliderlabs will automatically register and deregister any Docker container as a service in Consul. The registrator runs on each Docker Host.
  • HttpRouter
    Will dynamically route HTTP traffic to any application providing a HTTP services, running anywhere in the cluster.  It listens on port 80.
  • Load Balancer
    An external load balancer which will route the HTTP traffic to any of the CoreOS node listening on port 80.
  • Apps
    These are the actual applications that may advertise HTTP services to be discovered and accessed. These will be provided by you.

 

Getting Started

In order to get your own container platform as a service running, we have created a Amazon AWS CloudFormation file which installs the basic services: Consul, Registrator, HttpRouter and the load balancer.

In the infrastructure we create two autoscaling groups: one for the Consul Servers which is limited to 3 to 5 machines and one from the Consul clients which is basically unlimited and depends on your need.

The nice thing about the autoscaling group is that it will automatically launch a new machine if the number of machines drops below the minimum or desired number.  This adds robustness to the platform.

The Amazon Elastic Load Balancer balances incoming traffic to any port machine in either autoscaling group.

We created a little script that creates your CoreOS cluster. This has prerequisite that you are running MacOS and have installed:

In addition, the CloudFormation file assumes that you have a Route53 HostedZone in which we can add Records for your domain. It may work on other Linux platforms, but that I did not test.

 

git clone https://github.com/mvanholsteijn/coreos-container-platform-as-a-service
cd coreos-container-platform-as-a-service
./bin/create-stack.sh -d cargonauts.dutchdevops.net

...
{
"StackId": "arn:aws:cloudformation:us-west-2:233211978703:stack/cargonautsdutchdevopsnet/b4c802f0-d1ff-11e4-9c9c-5088484a585d"
}
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
CoreOSServerAutoScale 54.185.55.139 10.230.14.39
CoreOSServerAutoScaleConsulServer 54.185.125.143 10.230.14.83
CoreOSServerAutoScaleConsulServer 54.203.141.124 10.221.12.109
CoreOSServerAutoScaleConsulServer 54.71.7.35 10.237.157.117

Now you are ready to look around. Use one of the external IP addresses to setup a tunnel for fleetctl.

export FLEETCTL_TUNNEL=54.203.141.124

fleetctl is the command line utility that allows you to manage the units that you deploy on CoreOS.


fleetctl list-machines
....
MACHINE		IP		METADATA
1cdadb87...	10.230.14.83	consul_role=server,region=us-west-2
2dde0d31...	10.221.12.109	consul_role=server,region=us-west-2
7f1f2982...	10.230.14.39	consul_role=client,region=us-west-2
f7257c36...	10.237.157.117	consul_role=server,region=us-west-2

will list all the machines in the platform with their private IP addresses and roles. As you can see we have tagged 3 machines for the consul server role and 1 machine for the consul client role. To see all the docker containers that have started on the individual machines, you can run the following script:

for machine in $(fleetctl list-machines -fields=machine -no-legend -full) ; do
   fleetctl ssh $machine docker ps
done
...
CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS                                                                                                                                                                                                                                      NAMES
ccd08e8b672f        cargonauts/consul-http-router:latest   "/consul-template -c   6 minutes ago       Up 6 minutes        10.221.12.109:80->80/tcp                                                                                                                                                                                                                   consul-http-router
c36a901902ca        progrium/registrator:latest            "/bin/registrator co   7 minutes ago       Up 7 minutes                                                                                                                                                                                                                                                   registrator
fd69ac671f2a        progrium/consul:latest                 "/bin/start -server    7 minutes ago       Up 7 minutes        172.17.42.1:53->53/udp, 10.221.12.109:8300->8300/tcp, 10.221.12.109:8301->8301/tcp, 10.221.12.109:8301->8301/udp, 10.221.12.109:8302->8302/udp, 10.221.12.109:8302->8302/tcp, 10.221.12.109:8400->8400/tcp, 10.221.12.109:8500->8500/tcp   consul
....

To inspect the Consul console, you need to first setup a tunnel to port 8500 on a server node in the cluster:

ssh-add stacks/cargonautsdutchdevopsnet/cargonauts.pem
ssh -A -L 8500:10.230.14.83:8500 core@54.185.125.143
open http://localhost:8500

Consul Console

You will now see that there are two services registered: consul and the consul-http-router. Consul registers itself and the http router was detected and registered by the Registrator on 4 machines.

Deploying an application

Now we can to deploy an application and we have a wonderful app to do so: the paas-monitor. It is a simple web application which continuously gets the status of a backend service and shows who is responding in a table.

In order to deploy this application we have to create a fleet unit file. which is basically a systemd unit file. It describes all the commands that it needs to execute for managing the life cycle of a unit. The paas-monitor unit file looks like this:


[Unit]
Description=paas-monitor

[Service]
Restart=always
RestartSec=15
ExecStartPre=-/usr/bin/docker kill paas-monitor-%i
ExecStartPre=-/usr/bin/docker rm paas-monitor-%i
ExecStart=/usr/bin/docker run --rm --name paas-monitor-%i --env SERVICE_NAME=paas-monitor --env SERVICE_TAGS=http -P --dns 172.17.42.1 --dns-search=service.consul mvanholsteijn/paas-monitor
ExecStop=/usr/bin/docker stop paas-monitor-%i

It states that this unit should always be restarted, with a 15 second interval. Before it starts, it stops and removes the previous container (ignoring any errors) and when it starts, it runs a docker container - non-detached. This allows systemd to detect that the process has stopped. Finally there is also a stop command.

The file also contains %i: This is a template file which means that more instances of the unit can be started.

In the environment settings of the Docker container, hints for the Registrator are set. The environment variable SERVICE_NAME indicates the name under which it would like to be registered in Consul and the SERVICE_TAGS indicates which tags should be attached to the service. These tags allow you to select the  'http' services in a domain or even from a single container.

If the container would expose more ports for instance 8080 en 8081 for  http and administrative traffic, you cloud specify environment variables.

SERVICE_8080_NAME=paas-monitor
SERVICE_8080_TAGS=http
SERVICE_8081_NAME=paas-monitor=admin
SERVICE_8081_TAGS=admin-http

Deploying the file goes in two stages: submitting the template file and starting an instance:

cd fleet-units/paas-monitor
fleetctl submit paas-monitor@.service
fleetctl start paas-monitor@1

Unit paas-monitor@1.service launched on 1cdadb87.../10.230.14.83

Now the fleet report that it is launched, but that does not mean it is running. In the background Docker has to pull the image which takes a while. You can monitor the progress using fleetctl status.

fleetctl status paas-monitor@1

paas-monitor@1.service - paas-monitor
   Loaded: loaded (/run/fleet/units/paas-monitor@1.service; linked-runtime; vendor preset: disabled)
   Active: active (running) since Tue 2015-03-24 09:01:10 UTC; 2min 48s ago
  Process: 3537 ExecStartPre=/usr/bin/docker rm paas-monitor-%i (code=exited, status=1/FAILURE)
  Process: 3529 ExecStartPre=/usr/bin/docker kill paas-monitor-%i (code=exited, status=1/FAILURE)
 Main PID: 3550 (docker)
   CGroup: /system.slice/system-paas\x2dmonitor.slice/paas-monitor@1.service
           └─3550 /usr/bin/docker run --rm --name paas-monitor-1 --env SERVICE_NAME=paas-monitor --env SERVICE_TAGS=http -P --dns 172.17.42.1 --dns-search=service.consul mvanholsteijn/paas-monitor

Mar 24 09:02:41 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 85071eb722b3: Pulling fs layer
Mar 24 09:02:43 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 85071eb722b3: Download complete
Mar 24 09:02:43 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 53a248434a87: Pulling metadata
Mar 24 09:02:44 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 53a248434a87: Pulling fs layer
Mar 24 09:02:46 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 53a248434a87: Download complete
Mar 24 09:02:46 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Pulling metadata
Mar 24 09:02:47 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Pulling fs layer
Mar 24 09:02:49 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Download complete
Mar 24 09:02:49 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Download complete
Mar 24 09:02:49 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: Status: Downloaded newer image for mvanholsteijn/paas-monitor:latest

Once it is running you can navigate to http://paas-monitor.cargonauts.dutchdevops.net and click on start.

Screen Shot 2015-03-24 at 10.05.20

 

 

You can now add new instances and watch them appear in the paas-monitor! It definitively takes a while because the docker images have to be pulled from the registry before the can be started, but in the end it they will all appear!

fleetctl start paas-monitor@{2..10}
Unit paas-monitor@2.service launched on 2dde0d31.../10.221.12.109
Unit paas-monitor@4.service launched on f7257c36.../10.237.157.117
Unit paas-monitor@3.service launched on 7f1f2982.../10.230.14.39
Unit paas-monitor@6.service launched on 2dde0d31.../10.221.12.109
Unit paas-monitor@5.service launched on 1cdadb87.../10.230.14.83
Unit paas-monitor@8.service launched on f7257c36.../10.237.157.117
Unit paas-monitor@9.service launched on 1cdadb87.../10.230.14.83
Unit paas-monitor@7.service launched on 7f1f2982.../10.230.14.39
Unit paas-monitor@10.service launched on 2dde0d31.../10.221.12.109

to see all deployed units, use the list-units command

fleetctl list-units
...
UNIT MACHINE ACTIVE SUB
paas-monitor@1.service 94d16ece.../10.90.9.78 active running
paas-monitor@2.service f7257c36.../10.237.157.117 active running
paas-monitor@3.service 7f1f2982.../10.230.14.39 active running
paas-monitor@4.service 94d16ece.../10.90.9.78 active running
paas-monitor@5.service f7257c36.../10.237.157.117 active running
paas-monitor@6.service 7f1f2982.../10.230.14.39 active running
paas-monitor@7.service 7f1f2982.../10.230.14.39 active running
paas-monitor@8.service 94d16ece.../10.90.9.78 active running
paas-monitor@9.service f7257c36.../10.237.157.117 active running
How does it work?

Whenever there is a change to the consul service registry, the consul-http-router is notified, selects all http tagged services and generates a new nginx.conf. After the configuration is generated it is reloaded by nginx so that there is little impact on the current traffic.

The consul-http-router uses the Go template language to regenerate the config. It looks like this:

events {
    worker_connections 1024;
}

http {
{{range $index, $service := services}}{{range $tag, $services := service $service.Name | byTag}}{{if eq "http" $tag}}

    upstream {{$service.Name}} {
	least_conn;
	{{range $services}}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
	{{end}}
    }
{{end}}{{end}}{{end}}

{{range $index, $service := services}}{{range $tag, $services := service $service.Name | byTag}}{{if eq "http" $tag}}
    server {
	listen 		80;
	server_name 	{{$service.Name}}.*;

	location / {
	    proxy_pass 		http://{{$service.Name}};
	    proxy_set_header 	X-Forwarded-Host	$host;
	    proxy_set_header 	X-Forwarded-For 	$proxy_add_x_forwarded_for;
	    proxy_set_header 	Host 			$host;
	    proxy_set_header 	X-Real-IP 		$remote_addr;
	}
    }
{{end}}{{end}}{{end}}

    server {
	listen		80 default_server;

	location / {
	    root /www;
	    index index.html index.htm Default.htm;
	}
    }
}

It loops through all the services and selects all services tagged 'http' and creates a virtual host for servicename.* which sends all request to the registered upstream services. Using the following two commands you can see the current configuration file.

AMACHINE=$(fleetctl list-machines -fields=machine -no-legend -full | head -1)
fleetctl ssh $AMACHINE docker exec consul-http-router cat /etc/nginx/nginx.conf
...
events {
    worker_connections 1024;
}

http {

    upstream paas-monitor {
	least_conn;
	server 10.221.12.109:49154 max_fails=3 fail_timeout=60 weight=1;
	server 10.221.12.109:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.221.12.109:49155 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.39:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.39:49154 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.83:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.83:49154 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.83:49155 max_fails=3 fail_timeout=60 weight=1;
	server 10.237.157.117:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.237.157.117:49154 max_fails=3 fail_timeout=60 weight=1;

    }

    server {
	listen 		80;
	server_name 	paas-monitor.*;

	location / {
	    proxy_pass 		http://paas-monitor;
	    proxy_set_header 	X-Forwarded-Host	$host;
	    proxy_set_header 	X-Forwarded-For 	$proxy_add_x_forwarded_for;
	    proxy_set_header 	Host 			$host;
	    proxy_set_header 	X-Real-IP 		$remote_addr;
	}
    }

    server {
	listen		80 default_server;

	location / {
	    root /www;
	    index index.html index.htm Default.htm;
	}
    }
}
[/code]

This also happens when you stop or kill and instance. Just stop an instance and watch your monitor respond.
1
fleetctl destroy paas-monitor@10
...
Destroyed paas-monitor@10.service
Killing a machine

Now let's be totally brave and stop and entire machine!

ssh core@$FLEETCTL_TUNNEL sudo shutdown -h now.
...
Connection to 54.203.141.124 closed by remote host.

Keep watching your paas-monitor. You will notice a slow down and also notice that a number of backend services are no longer responding. After a short while (1 or 2 minutes) you will see new instances appear in the list.
paas-monitor after restart
What happened is that Amazon AWS restarted a new instance into the cluster and all units that were running on the stopped node have been moved to running instances with only 6 HTTP errors!

Please note that CoreOS is not capable of automatically recovering from loss of a majority of the servers at the same time. In that case, manual recovery by operations is required.

Conclusion

CoreOS provides all the basic functionality to manage Docker containers and provided High Availability to your application with a minimum of fuss. Consul and Consul templates actually make it very easy to use custom components like NGiNX to implement dynamic service discovery.

Outlook

In the next blog we will be deploying an multi-tier application that uses Consul DNS to connect application parts to databases!

References

This blog is based on information, ideas and source code snippets from  https://coreos.com/docs/running-coreos/cloud-providers/ec2, http://cargonauts.io/mitchellh-auto-dc and https://github.com/justinclayton/coreos-and-consul-cluster-via-terraform

2015 is The Year of Your Launch

Google Code Blog - Mon, 03/23/2015 - 20:01

Posted by Amir Shevat, Google Developers Launchpad Program Manager

With new events, improved courses and an expanded mentorship network - Startup Launch is now Google Developers Launchpad. We’re changing our program name to emphasize how you can use our resources as a launch pad to scale and monetize your app business. Read on to learn about our upcoming events and how you can apply to participate.

Events: Launchpad Week goes global

Launchpad Week, Launchpad’s weeklong in-person bootcamp for early-stage apps, continues to expand, with new 2015 programs planned in Munich, Mexico City, Helsinki, Bogota, and Sydney, to name a few. We’ll also regularly host these events in Tel Aviv, London, Berlin, and Paris.

We kicked off Launchpad Week in Bengaluru, India and Bordeaux, France last month. 32 startups and 80 experts from these communities gathered at Idiom Design Center and Le Node for a week of product, UX, and technology sprints designed to help transform ideas into validated, scalable businesses.

Featured startups from Bengaluru included iReff, an app that helps pre-paid mobile users find the best recharge plan for their specific needs. In Bordeaux, Google Developer Expert David Gageot volunteered as a tech mentor, helping startups “ship early, ship often” through testing and continuous integration.

Events: Google Developers Summits

For later-stage startups, we’re providing some of the best tech experts to help optimize apps for Material Design, Android TV, and Google Cast at two-day Google Developer Summits. At an event in Buenos Aires, Argentina, last week, we had participants such as game developer Etermax, the team behind Trivia Crack. Similar events happened in Kuala Lumpur, Bangkok, and Bengaluru this month, and we’re looking forward to inviting more startups to this program in London, Tokyo, Tel Aviv, and New York in 2015.

Products: Your app, powered by Google

In 2014, we helped over 5,000 developers in 170 countries get their ideas off the ground by providing the infrastructure back-end that allows developers to build incredible products. For example, our program delivered software architecture reviews and Google Cloud Platform credits to help entrepreneurs in the program build businesses that scale with them. Check out how Fansino is using Google Cloud Platform to let artists interact with their fans.

We’ve also expanded our product offer for early-stage startups to include AdWords promotional offers for new accounts. Whatever your monetization plan, we’re making it easy to get started with tools like the new In-app Billing API and instruction from the AdMob team.

Courses: Upskilling you and your app

Starting this month, we’ll offer a virtual curriculum of how Google products can help your startup. We’re kicking things off with new Launchpad Online videos covering Google Analytics - are you observing how your users use your app? How do different promotional channels perform?

The series continues in April 2015 with AdMob products, and will expand with instruction in implementing material design and conducting user research later in the year.

If you can’t wait, we’ve also built courses together with Udacity to take your technical skills to the next level on topics, including Android, Java, Web Fundamentals, and UX.

Apply to get involved

Apply to Google Developers Launchpad program to take advantage of these offers - g.co/launchpad. Here’s to a great launch!

Categories: Programming