Valkyrie at the North Pole By Diana the Valkyrie Being an account of how I solved a rather tricky problem Warning! This story contains no sex or violence. One of the things I do to earn a crust of bread (and the occasional dollop of jam) is, I'm a consultant. People consult me about things. Usually after their world has gone to hell in a handbasket and fifteen people have told them it's hopeless, and they've made things worse by random tinkering, sometimes someone says "Have you consulted the Valkyrie", to which the usual answer is "What's a Valkyrie?". And I find myself visiting far away places with strange sounding names to solve peculiar problems involving complicated situations. Most people's image of a consultant is someone who charges you to borrow your watch to tell you the time, and then doesn't give it back. What I do, is I get asked to solve problems with perfectly obvious solutions, but the people with the problem can't see it. Sometimes it's hard to keep a straight face. But even I was a bit surprised when a fat man in a red suit turned up last March, and said "Diana, you've got to help me, it's desperate." Not the words, I hear those all the time, and rarely add more than 10% to the bill as a result. It was the man himself. I mean, I stopped believing in Santa Claus round about the same time I discovered that boys can do it standing up. "It's all getting too complicated", he said. "Now with the Internet, all the kids send me emails with their present requests. Back in the old days, I used to get maybe a million letters, and we could handle that. But all of a sudden, it went to ten million, then 100 million, then a billion! How can I handle a billion? It's impossible!" He was getting excited, waving his hands about a lot. Impossible, yes, I know that word. I even know what it means, it means "I don't know how to do it". I hear a lot of people saying that, that's how come I know what it means. So I calmed him down, and made him some chicken soup, that's always good for conniptions. "First of all", I said, "There aren't a billion kids with internet connections. There's only about three billion kids in the whole world, and no way do a third of them have internet access." "I know", he said, "But some kids are sending in lots of requests, so we'll be getting double-ordering and double-delivery, there's too many, and the elves are getting uppity about it, and I don't know what to do, please please, you have to help me ..." "There there", I there-there'd, and told him to take another sip of chicken soup. He calmed down. "You can't imagine what it's like, Diana. We spend the whole year in production, and then for a month it's order processing, and then we have just one day per year to do fulfillment. Three billion customers, Diana, and they're depending on us, if we leave out just one then we've failed. And then the parents complain, and start talking about lawsuits, and ..." he took another gulp of soup, and I asked him if he'd like some bread with that. "No thanks." Every business is having to come to terms with the internet and the new opportunities and problems it brings. And any business that goes from a million orders per year, to a billion, is going to have trouble coping. "How do you process the orders?" I asked. "Maybe you'd like to come and see?" he invited. So I packed a suitcase full of necessaries, I put on two sweaters, a very long skirt, and my sheepskin coat, which is really warm and snuggly, and grabbed a couple of blankets just in case, and made some sandwiches, and phoned up my travel agent and booked a flight. The bad news was that they didn't actually have a direct flight to the North Pole. The worse news was that they didn't have an indirect one either. So I got out a Mercator Projection, and a globe, and did a few sums, and booked a flight from Manchester to Alaska. As I'm sure you'd have guessed, the airlines always fly the shortest possible routes, which means a Great Circle route, and the Manchester to Alaska route overflies the North Pole. And so it was that, just as the Ruritania Airlines flight from Manchester to Alaska reached the point where every which way is South, a Valkyrie opened the emergency exit and leaped out yelling "Geronimo" which I gather is the thing that you're supposed to yell in this situation, and then tried to remember where I'd packed the parachute. Only kidding, I was wearing it, I'm not that forgetful! So I floated gracefully down towards a broad expanse of white snow, with only a single red-and-white striped pole to show me where the target was. Fortunately, it was summer, which meant daylight all day round, and as I glided gently towards the Pole, I could see the reception committee, consisting of Santa and Mrs Claus. "Thanks for dropping in", she said, "would you like a cup of tea?" So I allowed as to how I'd be very pleased to wrap myself round an Earl Grey right now, and Santa gave a pull on the Pole, and a hole appeared in the snow, with a staircase leading down. Inside Santa's Grotto, as he insisted on calling it, it was warm and snug, so I peeled off a layer of clothing or two, and drank my tea, milk please, no sugar. And Santa explained that production for next year was in full swing now, but they'd barely survived last Christmas because of all the duplicate orders. "How did you deal with them?" I asked. It's always a good idea to find out how a business solves it's problems, because then when they ask me how to solve the problem, I can tell them that the method they just used will work. Yes, I know it's strange that someone would hire a consultant to tell them what they already know, go figure. "We didn't. We just did multiple deliveries. So if a kid sent us two emails asking for a doll, we send two dolls. It was the only thing we could do. Diana, they're telling each other about this, the kids have caught on to the fact that if they send me ten emails, they'll get ten presents. I can't handle that level of production, and delivery is already a nightmare." Yes - one of the benefits of the internet is better communication. It only takes one kid to post to alt.christmas.presents that if you send ten emails, you'll get ten presents, and pretty soon, lots of kids will be doing it. "And it's unfair", he said. "It means kids with internet access get more presents than kids without." I thought, why stop at ten? What if some kids send in a thousand emails? "We'll have to de-dupe the orders" I said. He looked at me. "De-what?" "De-dupe. De-duplicate. Work out which ones are the same, and only keep one of each." "How?" he said. "Lets see the orders" I said. So he took me to Santa's Office, and it was a huge room, with rows and rows of desks, and each desk had a high stool, but the place was deserted. "You should see it in November and December", he said. "The elves do the order processing." He showed me a form. "I want a cowboy suit" it said, and gave the customer's name and address. Another one said "Donne moi une pupette, s'il vous plait" with another name and address. "You see, the elves take the letters they write, which are often long and rambling, and we reduce it to these forms, that's how we've always done it. When we started getting telexes and telegrams, we just put them into the same system. Now we get emails, so we print out the emails, and pass those to the elves so they can do the forms." "And what happens to the forms?" I asked. He showed me where the goblins worked. They totalled up the orders, and worked the stock control system, so that the level of orders of each kind was passed on to the production floor, where the dwarves laboured on the manufacturing processes. "They also pass the forms to the Delivery Department." We moved on to look at Delivery. Just before we arrived, he turned to me and said, "You're not scared by big animals, are you?" "I'm a Valkyrie", I replied, so he pushed open the doors and we walked in. I suppose I should have guessed that Delivery Department would be entirely staffed by reindeer. I mean, I knew that they pulled the delivery vehicles, I suppose I was a bit surprised that they were also responsible for working out the Bill of Lading, route planning, and packaging. Santa introduced me to the delivery manager "Rudolph, this is Diana the Valkyrie, she's come to help with The Problem." The reindeer nodded and snorted sceptically. "Now don't start that again, Rudolph, the only answer is to modernise." Rudolph shook his head, and I wondered what would happen if he wasn't careful where he put his antlers. "Rudolph, you're just an old Luddite" said Santa, I could see that this was an argument they had lots of times before. Santa and I went back to his quarters, and had a glass of mulled ginger wine and some turkey sandwiches. "So let me summarise the system", I said, to check that I understood it. "You get orders coming in every which way, the elves transcribe them to standardised forms, the goblins process the forms, the dwarves manufacture the products according to demand, and the reindeer deliver according to the forms." "Right", he said. "And Diana, the goblins can't spot doubled orders from the same customer, so the order doubles get manufactured, and passed to the reindeer for delivery." "The answer is obvious", I said. "Not to me it isn't", he said. And this is why you call a consultant. Every business has its way of working, developed and refined over the years. When something new comes along, it's fitted into the old system. If you don't have enough goblins to process the forms, you hire more goblins. If you put a frog in water, and heat the water slowly, the frog never notices that it's time to make a radical change and hop out, because the change is so slow. Not that I've ever tried that experiment, and I hope no-one else has, but it's a good story. The people inside the business are so used to the existing ways of working, all they can think of, is expanding those ways. A consultant, however, can suggest a radical new idea. They were processing the emails, by printing them out, then treating them as letters, just like the old way of working. My idea was the exact opposite, to process them as emails, and convert the letters to a similar format, so it could all be handled as a single electronic data processing problem. "You need a computer", I said. "Yes, we make lots of those," he said, "lots of kids want them." "No, Santa, not a Christmas computer that you play games on and then something stops working after a few weeks and no- one knows how to fix it. I mean, a proper computer." "Yes, we make proper computers, Diana, honestly". "You sound like a computer salesman when you say 'honestly'. And what software do you put on it?". "Windows, of course," he said. I grinned. Of course. I did a few sums. One computer wasn't actually going to do the job, of course. Three billion names, addresses and orders, let's say a kilobyte each, that's 3000 gigabytes, double it for growth, that's 6000 gig. Six terabytes Maybe that sounds like a lot to you, but Joan in New York is the main server for the web site, and she's 100 gig. And then we'd want terminals for the elves, because although we could just feed the emails straight through, someone would have to type in the letters. But typing a million letters is a lot easier that filling in a billion forms from the emails, so we were ahead of the existing method right away. Plus terminals for the reindeer so they'd be able to get dispatch sorted out. Santa and I went shopping. When you need a reliable system, you want the best, so you expect to pay twice the going price for half the kit. We went straight to Armonk, no messing about, where IBM's Chief executive, once he heard the size of the order we were placing, stopped making remarks about Santa's red suit and the sack on his shoulder and started calling him Mr Claus, and me "Ms Valkyrie", which I hate. We bought a thousand PCs from those guys, and two thousand of those nice little IBM Deskstar 16.8 gb drives, giving us 32 terabytes, and keyboards, and monitors, and network cards, and tape backup systems - the whole thing came to $7.2 million. "Will you be paying Visa or Mastercard?" said the guy from IBM. "Gold", said Mr Claus, "red gold", opening the sack and letting the gold tumble out onto the floor. The IBM guy made a phone call, and a pack of accountants slithered in and started accounting it. "We'll deliver the goods within three to six months", said the IBM man. "I'll take it with me now" said Santa. I'd primed him on this, sometimes computer companies are a bit lax when it comes to delivery dates, on the assumption that no-one wants a computer anyway, so it doesn't matter if it arrives a year late. So we loaded up the sleigh, ignoring Rudolph's whuffling and tossing his head, while the accountants carried on accounting, until the Chief Accountant, a rat-like fellow with beady eyes and a sniff, said it was all there, and then we took off with the goods. When I say "took off", I mean, took off. You've probably often wondered how airplanes fly. I mean, once you've gotten past believing in the Tooth Fairy, and the idea that air, which weighs practically nothing, can support a Boeing 747, that weighs as much as ten elephants, you wonder about things like that. Actually, it's done by the power of faith. Airplanes fly because the people in them believe they can fly, it's as simple as that. Before Orville and Wilbur, no-one believed that a heavier-than-air machine could fly, and they were right. But the Brothers Wright persuaded themselves, and then they persuaded the people around them, until now every passenger in a jet plane is firmly convinced that it's going to be able to get off the ground and fly, and that's how it works. So Rudolph and his seven colleagues started to heave this sleigh along, loaded to the gills with computer hardware. The lack of snow was a bit of a drag, as was the several tons of metal heaped in the cargo bays. But once you get something moving, it gets a bit easier to pull, and they gradually speeded up, until at about 75 mph Santa said "Up, up and away", and I thought upward thoughts of floating and lightness and tried to believe firmly that we'd soon be up in the air, and then Rudolph rotated and started galloping on nothing, and the other reindeer followed, and the entire contraption took to the cloudy deep with about as much plausibility as a Jumbo jet taking off, but considerably less sound, and we were flying. So Santa did a wide turn to the left and headed North, still saying "Gee up, Rudolph", and I started to enjoy the ride, which is a lot quieter than a jet, because hooves drumming on air aren't too noisy, and I said "What's the rush?" "We want to leave US airspace by midnight" said Santa. "Why?" I asked, thinking about pumpkins and mice. "That was fairy gold" he said. Uh-oh. Fairy gold turns to dross at midnight. Now I don't actually know what dross is, never having seen any, but you can bet your bippy it isn't something that accountants would enjoy accounting. "Giddy-up" I yelled to Rudolph, not wanting to face an accountant with a bagful of dross, they get quite vicious when they're cornered. We landed at the North Pole, or to be more precise, Santa had a remote that he pressed, and the door in the ground slid open and we gracefully swooped through, and landed in a hangar deep underground. The reindeer helped us unload the vehicle, and then I got the elves to help me set up the computers. I put the whole thing on a 100mbps fast ethernet system, and once we had the hardware installed, I put in the first of my secret weapons. Linux is free, and you might think that, like many free things, it's worth what you paid for it. But actually it's rather good. I won't get too lathered up about it, all I can really say is that it works. But that's saying a lot more than I could say about most things people run on their computers. I'm constantly surprised at the way people are willing to accept that their computers stop working, crash, get booted, whatever they call it, at least once per day. You wouldn't accept that in a car or a toaster. Oh, and we used the domain name christmas.mil because Christmas.com is taken, but it seems that they haven't yet thought of a military use for Christmas. The second secret weapon was perl, which also comes free with Linux, and which is a great way to write stuff that's doing text processing, which is what this was. That's the hardware in place, and the operating system, but you're probably thinking, what about software? You can't get software out of thin air. Well actually, that's exactly where it does come from. You see, what software is, is someone telling the computer what to do. Valkyries are rather good at telling people what to do, and computers are easier than people, because they do what they're told. They don't do what you want, of course, but they do do what they're told, unlike people, who don't. So it's just a matter of getting it clear in your own mind what you want, and then telling the computer that. Piece of cake. The elves converted any paper mail to emails, and the emails were sent to the main processor. Every email contained the customer's name and address (without those, we couldn't deliver anyway, so any that didn't have that information were discarded). I used a homomorphism to reduce the name/address to the minimum, then made that the key of an associative array (or hash as they're now known), which meant that next time a request came from that same customer, there was already a hash entry, so it was obviously a duplicate, and could be used to replace the previous one, on the assumption that whatever the customer asks for last, is what we should deliver. So the data was de-duplicated, and because we had the addresses on file, we could also use sophisticated route-planning algorithms for the delivery, so the reindeer wouldn't have to dash from New Zealand to New England and back again, but could do all their antipodean deliveries before flying over to the new World, and so forth. Actually, not really sophisticated, but what a saving in reindeer-power! Oh sure, there were snags along the way. There always are in a big project. I suppose I should have foreseen that reindeer would find typing a bit tricky, what with those big hooves, but I installed speech recognition software, and that dealt with that. And I had to keep telling the dwarves not to try to fix computer problems using their usual methods, which involves a range of hammers that starts at seven pounds and goes up from there. And the unpleasant discovery that fairy magic causes lost packets on unshielded twisted pair ethernet cabling, which meant that we had to replace it with shielded, because you can hardly tell a fairy to stop being magic. But by October of this year, we'd ironed out all the problems, the elves were settling into their new jobs, the dwarves loved the new inventory and production control system, the pixies took to being network engineers like ducks to dinner, and Santa was absolutely over the moon (apparently you can fly reindeer ... but that's another story). So I reluctantly (you have to be careful with the reluctance bit, you don't want them to get the idea that they needn't pay for services rendered) accepted a modest fee from Claus Enterprises, insisting on a pound sterling cheque rather than the gold that he wanted me to have, and hopped on the next reindeer back to dear old England, where, since it was October, Christmas was in the air, and all the shops were getting ready for the annual surge in spending patterns. But whenever the phone rings, I give a little jump. Oh sure, the design is sound, the computers reliable, and the system worked. But one thing we simply couldn't do, and that was test it under live conditions. I mean, what will happen when it gets used for real? I mean, we're talking about billions of data entries, and we don't know how many, because maybe some of the kids will work out how to automate the sending of emails and flood the system with thousands per minute using one of the spamming programs. Or maybe one of the dwarves will decide to use his hammer to improve the system a bit. Or there'll be interference between some item of fairy magic and some key system component. Or something, I don't know, there are just so many things in a computer that can go wrong. It's the usual thing. The job was rushed, the documentation inadequate and there was no proper testing. When things start moving from hectic to frenzied to panic to nervous-breakdown, I don't really know how well the thing will stand up. Maybe it'll hold together this year, but for sure it should be re-done properly next year. I hope you're not relying on it. If I were you, I'd make sure that you have alternative arrangements in place this Christmas, so that your little customers won't have to forgo the pleasure of unwrapping, playing with and breaking their presents. You can always pretend they came from the usual source.