Geoff Dagley from Relevance

About a week ago, we had the pleasure of hearing Geoff Dagley from Relevance Development. He was a friend of Jeremy Suing's (the professor) from his years of working at the (now defunct) J.D. Edwards Company. Geoff's talk covered both agile project management and agile development.

The project management half of the talk was pretty interesting. Relevance is on the bleeding edge of not only agile development, but agile management as well. They do most of their work remotely for smaller clients. Relevance uses Mingle for project management, which looks like the best tool for the job. Some other highlights about their project management techniques:

• They work with clients and educate them on agile processes - so the client knows what their consultants are doing during an iteration.
  
• They involve the client during daily standups - Relevance teams conduct daily conference calls with their clients.
• Clients can purchase a single iteration of development from Relevance.
  

I was very intrigued by the fact that would could buy a single iteration (2ish weeks) of development from Relevance. They must work very closey with their clients to make that possible. He also showed us the end of iteration documentation and it was pretty impressive. They used Mingle to export reporting materials, which is a great way to save time.

The second half of Geoff's talk was about agile development. It was nice to hear it from Geoff because he still developed software. Relevance uses rails, github and a continous integration server (Calm down Erik). Some more highlights about thier development practices:

• They practice Behavior Driven Development (BDD) - A practice that is driven by value delivered to the client, which is perfect for a consulting organization.
• RunCodeRun, which is their continuous integration solution, is tied closely to github, so it makes sense for Relevance.
• Developers do remote XP with skype and screen sharing software, which I found to be incredible.

The remote screen sharing was very interesting - He is on the phone and sharing his screen with another developer for most of the day. That's an interesting way to get things done right, but might be a little intrusive.

His final thoughts touched on the culture of Relevance, which is a big reason for their success. If the culture of an organization fits with agile development, then that organization should build their software with agile development. This is a counter argument to the traditional "Right tool for the job" argument about using agile to build social, interface driven software. The culture of the company should dictate the methods they use for getting the job done. A sports anaology can be applied in this situation: If I have a guard oriented basketball team (culture), then I would run a guard oriented offense (software development methodology), to win the game (deliver working software).

Geoff gave a great talk - I Could really tell that he loves what he does and was very passionate about software.

Bonus Evaluations

The final system is arguably the most controversial, but accurately mimics systems in some companies. In many companies, salaries are "low", while a good portion of compensation is in yearly bonuses and/or commission. These are used to reward results and encourage improvement. In Design Studio, the PMs would be given a set amount of bonus money points. These points would be distributed to teams and team members based on achievement and improvement. PMs would need to defend these choices to the other PMs saying why their team deserves more points. It would essentially put the Design Studio grading on a sort of curve. The PM could choose to give each team member a "B", or (more likely) set distinct grades.
This would require extensive PM communication and rely on their ability to accurately justify their teams' progress and results. They would also need to know each team member and their activities very well, more than in the current system. This would still require them to evaluate their team, but on their own terms. (perhaps with a previous method?)
Pros:

• Competitiveness between teams can yield greater results.
• Forces PMs to be actively involved to properly evaluate teams.
• PMs get to defend value of their decisions.

Cons:

• Competitiveness between teams can easily become negative.
• Team grading is truly at the discretion of their PM, there is very little standardization.
• Results may take precedence over learning and goals.

This would work with a set of identical PMs and a purely positive competition between the teams. Unfortunately, students always question grading and would even more so if people (Lutz) broke the curve.

Big thanks to Nate for writing the majority (all) of this post. Please forgive me for being a day late in our post-a-day series about Design Studio evaluations. Please post any final comments below.

Interview Evaluations

Instead of having team members write responses, evals could be done face-to-face with a Program Manager. These "interviews" would happen every 2 weeks and take about 5 - 10 minutes. It's basically a "how goes it" session with your PM without any other team members. These would closely resemble the post-eval talks from the current system.

An obvious benefit of this system is the timeliness of responses. Issues with team members or processes can be acted on immediately. Another necessary piece would be to focus time on goals (if there are no major issues). This would allow a regular check on the progress of goals and suggestions for improvement. Since setting, achieving, and learning from these goals are a large part of our grade, students would be able to hear exactly how they were doing and still have time to improve.

Pros:

• Timely information.
• Informal, conversational situation.
• Elimination of evaluation deadlines.
• Good practice on open, constructive communication for both students and PMs.

Cons:

• Hard for students to verbralize difficult evaluation subjects (i.e. Criticism of a teammates work). Also difficult for the PM to respond in a timely manner to difficult issues (more on this below).
• Scheduling pains and time investment.
• Recording workload is placed on the PM.
• Difficult to quantify evaluations into a grade.
• Issues may take time away from goals.

Personal interviews would be a big change of pace for Design Studio. Communicating a specific, difficult subject verbally in the proper context (PM and student are on the same page) is a communication skill that most students would need to improve on.


PM's would also have to factor in the personality of the evaluator, as well as the information provided about the student that is being evaluated. For example, students that have more upfront personalities would provide evaluations that are biased - either very positive or very negative reviews. On the other hand, if the evaluating student's personality is more reserved, then the PM would need to be very attentive during the interview to understand the evaluating student's opinion.

Personal interviews would give PMs better feedback on the person that is providing the evaluation. This would be in addition to the reviews provided by the rest of the team. A lot can be determined about a person from the way they describe their teammates.


This is one of the more experimental ideas that we came up with for improving the evaluation system. Please post thoughts and comments about personal interview evaluations. Also, examples of a similar system being used in the industry would be great to discuss in the comments.

Private Blogs for Evaluations

Since Nate and Nick are avid bloggers (note we didn't say competent), our first idea for the improvement of the evaluation is an evaluation blog. Team members would make a well thought-out blog post about the performance of their team at any necessary time. Blog posts could be shorter (a few paragraphs) and focus on one member of the team. We would want to enforce a minimum length for each post so we don't have people tweeting their evaluations. This would reduce the number of reactionary and flamatory posts.

Pros:

• Students can report feedback at their own pace. Blogging gives the option of more frequent updates, while retaining the option of having large posts twice a semester.
• Team members would receive faster feedback on how to improve. PMs could also use comments to ask questions for clarification.
  
• Can tag based on team member, individual goal, team goal, etc. This should make PM life a bit easier.
  
• Faculty can enforce a qouta on number of posts depending on the amount of feedback required.
• Easier to remember specific behavior examples the day they happened instead of at a particular time.

Cons:

• High initial overhead if an existing cannot be adapted to fit Design Studio's needs. These need to be both secure and accessible, and only visible to the student and their PM.
  
• Easy to forget/put off because there will be more updates required (i.e. every two weeks compared to every six weeks).
• Would bring on more negative comments.
• High PM overhead because they would need to compile multiple posts to determine a grade.

An organic system like this would skip over the "evaluation period" and require more upfront planning to ensure the completion of goals. Team member goals would need to be communicated to the rest of the team, or this system would not make it easier to give feedback on teammates goals. Unless there was a public post of goals that teammates could comment on.

A system like this would either be very easy (use Wordpress or Blogger) or quite difficult (write custom application) to set up. If a blogging system could enforce all of the constraints required by an evaluation system then it would be an attractive replacement for the current system.

This example post took me less than 2 minutes to think up, write, and publish.



Please, share your thoughts about blogging evaluations in Design Studio. And remember, if you have any other ideas about improving the evaluations process, please share your thoughts below.

Also, it happened.

The Current Evaluation System

Four times a year, we do evaluations of our teammates, ourselves, and the team in general. This involves filling out multiple categories as strengths or weaknesses and elaborating on three sections for each: Current Progress, Improvement Suggestions, and Other Comments. Categories can be added or removed from the top, and you can add your own category (even "Beard"). A suggested grade is also required. It is a lot better than the previous system. It has improved since the beginning of the year; the second round of evaluations brought the addition of some good looking web 2.0 features (click the image for a better view).


There has been an added emphasis on goals this year, which is an improvement from last year. Goals are not built into the system that handles the rest of the evaluations; they are tracked separately in SharePoint.

Pros:

• Comprehensive - They cover all aspects of Design Studio.
• Choose Your Own Adventure (or categories) - Evaluate what you think is important for them or your team.
• Goal based - Meet with your PM after evals to review grades/goals.
• 4 due dates per year - there is no week to week requirement for maintaining evaluations.

Cons:

• Not Responsive - Feedback goes to your PM four times a year, hardly often enough to take corrective action in a timely manner.
• Requires People To Remember Situations - If a teammate does something negative or outstanding, you need to remember it untileval time.
• They Take Forever - Any week can be ruined by 6-10 hours of evals.
• Goals are tracked in a separate system, so there is little feedback on goals from teammates.

The length of evaluations are always up to the discretion of the evaluator. There is no set requirement on the length of each evaluation. It is suggested by faculty that each team member evaluate themselves, team members and their PM in at least 5categories. However, this guideline is rarely followed.

The current system has served Design Studio well for the past X number of years (at least two). The following 4 posts in this series will explore new ways of handling feedback to improve performance and make Design Studio a more enjoyable experience.

Please post comments related to the existing system - What you like? What you dislike? What is confusing about it? Also, share an idea if you would like us to include it as a part of this series.

Design Studio - Evaluation Series

Starting this evening and continuing throughout the week, Nate and I will be taking a closer look at the Design Studio Evaluation process and possible ways to improve it. We have to give credit where credit is due - this series is borrowing from Erik's Design a Day Series. But neither Nate or myself are very good designers, so we'll be presenting a series of examples to improve the evaluation process and help students improve during design studio.

Nate and I plan to keep in mind the following guidelines when rethinking the evaluation system:

• Design Studio is still a class - which means the new system must be able to assign a reasonable grade to each student.
• We need to focus on improvement - If a student is learning and applying course material, then they will show improvement.
• Students should work on their projects as much as possible, so there needs to be low overhead in an improved evaluation system.

Evals have come a long way since we began Design Studio, and we want to keep improving the process. Improving the evaluation system would lead to better feedback, which will help students get better and increase the probability of project success.

Later tonight, we will post a review of the current evaluation system and the road map for the rest of the week. Please leave comments - we want to hear as many different viewpoints as possible.