The Current Evaluation System

Four times a year, we do evaluations of our teammates, ourselves, and the team in general. This involves filling out multiple categories as strengths or weaknesses and elaborating on three sections for each: Current Progress, Improvement Suggestions, and Other Comments. Categories can be added or removed from the top, and you can add your own category (even "Beard"). A suggested grade is also required. It is a lot better than the previous system. It has improved since the beginning of the year; the second round of evaluations brought the addition of some good looking web 2.0 features (click the image for a better view).


There has been an added emphasis on goals this year, which is an improvement from last year. Goals are not built into the system that handles the rest of the evaluations; they are tracked separately in SharePoint.

Pros:

• Comprehensive - They cover all aspects of Design Studio.
• Choose Your Own Adventure (or categories) - Evaluate what you think is important for them or your team.
• Goal based - Meet with your PM after evals to review grades/goals.
• 4 due dates per year - there is no week to week requirement for maintaining evaluations.

Cons:

• Not Responsive - Feedback goes to your PM four times a year, hardly often enough to take corrective action in a timely manner.
• Requires People To Remember Situations - If a teammate does something negative or outstanding, you need to remember it untileval time.
• They Take Forever - Any week can be ruined by 6-10 hours of evals.
• Goals are tracked in a separate system, so there is little feedback on goals from teammates.

The length of evaluations are always up to the discretion of the evaluator. There is no set requirement on the length of each evaluation. It is suggested by faculty that each team member evaluate themselves, team members and their PM in at least 5categories. However, this guideline is rarely followed.

The current system has served Design Studio well for the past X number of years (at least two). The following 4 posts in this series will explore new ways of handling feedback to improve performance and make Design Studio a more enjoyable experience.

Please post comments related to the existing system - What you like? What you dislike? What is confusing about it? Also, share an idea if you would like us to include it as a part of this series.

Design Studio - Evaluation Series

Starting this evening and continuing throughout the week, Nate and I will be taking a closer look at the Design Studio Evaluation process and possible ways to improve it. We have to give credit where credit is due - this series is borrowing from Erik's Design a Day Series. But neither Nate or myself are very good designers, so we'll be presenting a series of examples to improve the evaluation process and help students improve during design studio.

Nate and I plan to keep in mind the following guidelines when rethinking the evaluation system:

• Design Studio is still a class - which means the new system must be able to assign a reasonable grade to each student.
• We need to focus on improvement - If a student is learning and applying course material, then they will show improvement.
• Students should work on their projects as much as possible, so there needs to be low overhead in an improved evaluation system.

Evals have come a long way since we began Design Studio, and we want to keep improving the process. Improving the evaluation system would lead to better feedback, which will help students get better and increase the probability of project success.

Later tonight, we will post a review of the current evaluation system and the road map for the rest of the week. Please leave comments - we want to hear as many different viewpoints as possible.

Demand Based Pricing

In my previous post, I discussed the general pricing strategy for airlines. Our booking of the trip to Italy for $388 was an anomaly and didn't really fit with trends of airfare - round-trip to Europe goes for $800 during May (which I still find to be incredibly cheap.) The below graph gives a great picture, and its only one-year-old.



When we booked our tickets, we booked 5 at one time. Erik and Matt followed within the hour. We originally thought that the rate we booked at (which Paul somehow found) was going to expire. This was not the case, because when Carrie went to book her ticket a few hours later, the rate adjusted itself to $750ish dollars.

All travel websites implement a back end web-service for flight information and booking online (from a company like ITA Software). Even though the travel sites look different, (Expedia vs. Traveolcity) they use the same back-end web-service to offer flights. The travel "front end" sites offer sites from the same pool, with some price variance depending on the time of day or area of the country from where the person is visiting the site.

So the front end sites and the back end provider must keep in close contact, because the front ends are all selling from the same pool of flights.

The strange thing about our example is that we are leaving on May 7th, and the rate was only adjusted for that day. The day before and the day after (6th and 8th) had the same rate that the rest of the group booked. The pricing service saw that there was high demand of people from Lincoln, Nebraska booking the same flight to Italy, so it adjusted itself hoping that the last person would pay double to travel with the same group.

The fortunate thing about the rate adjustment was that Brett (internet travel extraordinaire) was able find one Website, Orbitz.com, that did not update their prices. So Carrie booked the flight through Orbitz at $388. But Orbitz has to turn around and pay the increased wholesale rate for the $750ish flight to the back end service.

We really dodged a bullet here. Online travel (shopping as well) is difficult, and consumes lives. Booking online travel is impossible - that's why I outsource whenever I can.

Demand based pricing is an interesting subject. Since the acceptance of online shopping, retailers know much more about their customers and can offer a range of prices for the same goods. Keep your ears up for more Italy trip posts and at least one more post (this week) related to variable pricing.

Variable Pricing on Airlines

Recently, Paul convinced Brett, Nate, Matt, Erik, Kyle, Carrie, and myself to travel to Italy for a beginning of the summer vacation. The trip is shaping out to be a blast and I'll post more about the specifics of the trip in the future.

Our main motivation for this trip was not the sights or sounds of Italy, but the price of the plane ticket. We booked our round-trip flights from Chicago to Italy for $388 dollars a price. We might be flying over on the Wright brothers plane, but this was quite the accomplishment, considering that you cannot get to Phoenix for less than $300 dollars. Or can you?

Everyone has a story where they stuck it to the man, beat the system and booked air travel for a ridiculously low price. There is of course a logical explanation to these low rate flights - variable pricing. This is evident in airline pricing strategy (different classes of tickets). Some people can afford to pay $1000, some $800, $500, down to $100. This is done to meet the demand of the market while still trying to maximize revenue.



Maximizing revenue is a difficult subject, because once the fixed costs of keeping the plane in the air are covered, the marginal cost of adding another passenger to the plane is around $30 (meal, baggage, ticket handling, etc.) If the airlines' marginal cost to add another person to the plane is $30 dollars, why are there empty seats? An airline lowering its rates at the last minute to ensure that the plane is filled would make the most revenue off of that flight.

Airlines cannot conduct a last minute, everything must go fire sale for all of its flights. This practice would increase revenues for individual flights, but would hurt the brand image of the airlines in the long run. People would refuse to book their flights, hoping that they will book at the right time to get the cheap seats. Lack of early bookings would cause a reduction of flights because airlines cannot meet the fixed costs of keeping planes in the air. This would cause customer dissatisfaction, and eventually the airline would go under. Then the economy would collapse and we would revert back to primal tribes of nomads (maybe not that far, but who knows?)

Airlines can decrease rates slightly to get the final 2% of seats filled, but there are still empty seats.

Air travel is an interesting business because it is so broken. There are a multitude of problems with air travel - business and technical - that still have yet to be solved. This slide show was pretty interesting (but its from 2003).

Please, Don't Throw the Sheep

My Mentoring by the Masters post made me think of an article I wanted to share for quite some time - Tim O'Reilly urging developers to stop throwing sheep and do something worthy. HappierHour is not a sheep throwing application. But it is not preventing global warming either.

O'Reilly makes some other observations in the article. He stresses that the United States is getting surpassed in science and technology innovations - American firms are not providing enough value. He also mentions that Wall Street firms capture more value than they create.

Value is something that needs to be in the front of your mind. Always. HappierHour might not cure cancer, but we are providing value to a significant market. I still think the value we provide is valid, but would it pass O'Reilly's test? Not too confident about that one.

Then how on earth are we supposed to tackle large (checkbook-intensive) problems? And the better question is - How do we solve a large world problem with software?

We can't directly solve world problems with software. Software is not the net that stops the spread of malaria or the smart fuels powering hybrids. We need to take it some step further and think of the building blocks that go into solving these issues. Want to map the human genome? Grab your Perl book and get to work.

Through the efficient use of software - large, complicated world problems can be solved. I hope that's what O'Reilly meant when he told us to stop throwing the sheep.