Here's a link to his final prediction. I'd say this is basically a coin toss, but one commenter pointed out that this is actually closer to 50/50 than a coin toss, which can easily be 49.5/50.5 in practice
Microsoft can get quite a lot of milage out of a product that's 'good enough'.
I think they aren't checking models that are attached to a search engine, which sorta seems to be a good way to fix this. Cool to see that o1-preview is able to figure out when it shouldn't attempt questions more reliably than others, but only by a bit. Interestingly, if you ask it to rate it's own confidence, that is correlated with correctness, although it's not well calibrated. In developing the list of questions, they specifically looked for questions that induced hallucinations, an example is "Which Dutch player scored an open-play goal in the 2022 Netherlands vs Argentina game in the men’s FIFA World Cup?"
If you want even more detail, here's an NYTimes article to check out. Here's extremely simple infographic if you prefer that.
Flotsam and Jetsam
– A bunch of people are dunking on the idea that being a farmer is really pleasant and fulfilling, but my favorite is just that "farming is literally the biblical punishment for original sin". (src)(cached)
– One of the showrunners for Marvel's Agatha All Along tv show talked about a cameo they considered that didn't work for the story, so they dropped it. It's good to hear that they sometimes decide not to do crossovers! (src)
– The menswear guy is writing articles for the NYTimes, this one is about sholder pads and politics. (src)
– People argue over whether surgeons should be expected to work crazy hours and never see their families. I can't say for sure what kind of work culture would result in the best patient outcomes, but I can say for sure that a big part of the AMA's work is to ensure the US has doctor shortages. Seems related! (src)(cached)
– Some folks in SF are harassing Waymo vehicles because they are docile and careful. One commenter responds "We rely on random violent nutcases to deter certain kinds of antisocial behaviour" (src)(cached)