Story Time: A Really Bad Situation

Starting a series of posts relating events and activities from a 25+ year Product Management career. First up, a shit-storm with a capital 'S'

Story Time: A Really Bad Situation

Herein begins a new series of posts where I tell stories about various times in my Product Management career. Over 25 years, it is inevitable that there are some interesting tales, and since product management is also a little about storytelling, it is seems like a fruitful exercise to capture some of these.

The first story actually is from just before I jumped from Applications Engineer to Product Manager. I was working at a company that made linewidth measuring microscopes that the makers of photomasks used to determine if the mask was good, or in spec. It was a computer controlled system, with an optics system, a stage to move the mask, and software to interpret the pixels off the image and return a value that was the line width.

It was a lot more complicated than that, but in essence it was an automated microscope that repeatably measured critical dimension on a chrome on glass photomask.

With that background out of the way, on to the story.

Too Clever Salespeople

When you sell an instrument to a customer that will use it to control their processes, typically there is a specification that you offer, and the customer will often counter with what they want. You will then negotiate, usually with a demonstration on their product(s) so that you can share performance, and show its suitability for purpose.

When the sale is consummated, this spec is documented, and an “Acceptance Criteria” is agreed upon. The vendor (us) delivers the tool, we do a full suite of testing, and validating that it is performing. We share the results with the customer, and they agree (or if you are not meeting it, you get engineering on the horn to fix it), and ultimately, you get sign off, the customer takes full ownership of the product, and you get the last of the payment (the payment for a small company like ours was typically 70% on shipment, and 30% on sign off. That way we could afford to build the instrument before the signoff.)

In this case, the customer was in South Korea, and they were a large semiconductor manufacturer who wanted to build a captive mask shop, thus reducing the costs of sourcing masks from one of the merchant shops (like Photronics at the time).

As such, the customer’s engineers weren’t very savvy, so our sales people scored what they thought was a coup. No acceptance criteria was codified in the sales contract.

Their view was that on delivery, they would just get paid their commission, and all would be good.

However, that is not what happened.

The Chaos

Almost immediately, the first application engineer that was sent got into an unending loop of more tests, more validation, more measurements. He was there for 2 weeks, then he had to leave (entry visa issues) only to return two weeks later.

Again, he struggled with the engineers at the customer. They weren’t super knowledgeable about making photomasks, so practically every day there was a new “ask”.

This applications engineer visited a total of 3 times, covering about 8 weeks, and at the end it was clear that we were never going to get sign off.

I got the tap on the shoulder to try to clear it up. I took a day (and 5 crisp $100 bills) up to the Korean embassy in San Francisco to get a work visa (yes, you had to submit the fee in cash, I am totally sure that money was fully accounted for *wink*) and I headed over to Inchon Korea to unravel the mess.

It took a day to survey the work and the data collected thus far. We spent a day in negotiations as to the exit criteria, and I set to work.

Two days later, I had great data, and the engineers said yep, that looks good. But they wanted another round of tests (tests that had been done literally dozens of times) and I said I would talk to the factory.

That night, I got on the phone with the owner and the COO of our company, and I told them the situation. We had:

  • Literally 100 megabytes of data (that is a lot of .csv data circa 1996)
  • All that data was great. There were no anomalies, no outliers, no fliers.
  • The instruments (they bought 2) were about as good as they could get. There were no easy alignments/adjustments to make to improve performance. In short, they were about as good as it gets, and plenty good for the likes of, oh, I don’t know, INTEL.
  • Their engineers were a bit green (new) and were uncertain about what to do.
  • We had about 5 man-weeks on the ground between my predecessor and myself. Pretty soon, there would be no profit left after the costs of getting sign off.

I proposed a radical plan. We had gotten 75% payment on shipment. We had delivered in good faith. Yes, we foolishly booked the order without acceptance conditions, but we couldn’t pfaff around endlessly on this.

My proposal was:

  • I would do one more round of our standard acceptance, crunch the numbers, and if it passed (which it had 10 or 15 times already) we would tell the customer to sign off.
  • If they chose to not sign off (and release the final 25% payment) we would give them title to the instruments, however we would keep the 75% of the money, and we would void their warranty. We would not service it, even prepaid. Ever.

The next morning, prior to the visit to the customers, I conferred with our local representative (note, they were the ones that got the purchase agreement without the acceptance defined) that I would tell this customer (a HUGE company, one of the Chaebols) this was the plan, and that I would await their decision.

Needless to say, our rep was nervous. But I had the full backing of our factory and our owners. This had gone on long enough.

The Meeting

man standing in front of people sitting beside table with laptop computers
Photo by Campaign Creators on Unsplash

It started with the usual pleasantries. Then I clearly enunciated the terms, telling them what we would do, what we would no longer do, and what would happen if they didn’t agree.

They were shocked. There was teeth sucking. There were “we need to talk to our managers”, and other things.

Regardless, I stayed in that conference room with our local agent, who was sweating profusely by this point, and I read a Science Fiction novel. There was no point in doing any other work until this was handled.

About 3 that afternoon, they returned, they agreed. They had one extra thing they wanted (they clearly had talked to a local merchant mask shop for this condition), and I agreed.

There was about 2 days more work, I tacked on two more days of training (programmability, and system use/maintenance) and we agreed.


Four days later, I had the signed acceptance, we got paid, and as a going away celebration, the engineers at the customer wanted to go out for drinks. We drank the night away, they became friends, and years later after my new company sold them a different instrument, we were back together. All was good.

A couple of other times in my career sales did a similar thing, no up front acceptance defined. But I learnt to pre-meet with customers to establish the end goal, and I never had such a problem again.

This experience taught me that whenever I interview people for a product management position, I ask them about a time that something went totally sideways, and how they dealt with it.

If you can’t recall a total shit storm, and you are an experienced product manager, then you might not be right for the role.

That is my first shitshow.