Why ‘open source’ AIs could be anything but, the derailment risks of long freight trains, and breeding better wheat

Benjamin Thompson

Hi, Benjamin from the Nature Podcast here. Slightly different show for you this week, we're going to be diving into the Nature Briefing and chatting about some of the stories that have appeared in it over the past couple of weeks and joining me to do so are two members of the podcast team, Nick Petrić Howe. Nick, how you doing?

Nick Petrić Howe

I'm very well thank you, Ben.

Benjamin Thompson

And Dan Fox. Dan, how's it going today?

Dan Fox

Hello, I'm good. Thanks. Ben,

Benjamin Thompson

So, three stories to talk about today. Nick, why don't you go first? You've got something about AI this week.

Nick Petrić Howe

That's right, I've been looking at a story about something called ‘open-washing’, and this is a story I was reading in Nature by our good colleague and friend, Lizzie Gibney. So, you guys, you know you’re science journalists, you’ve probably come across the term ‘greenwashing’. This is a similar sort of idea but instead of giving something more environmental clout than it deserves, this is about giving something, specifically AI, maybe more open clout than it deserves.

Benjamin Thompson

Right. I mean, I guess washing has negative connotations and how is it affecting the world of AI? And who is potentially doing this washing?

Nick Petrić Howe

Well, potentially a lot of people. So, this News article focuses on an analysis that has basically ranked different large language models — so, these are the models that power things like ChatGPT — and it's ranked them on how open they actually are. And it uses different criteria, such as how much of the training data people are able to see, whether people can understand the weights, which are the bits and bobs of the model that help it decide, like, how much weight to put on something, and that sort of thing. And, yeah, it seems like a lot of models that are sort of saying the word open aren't necessarily hitting all these criteria. So big players like Microsoft, but also like Meta and Google, they have said that things are open, whereas these researchers are saying, well, actually, you're not really meeting some of these criteria.

Dan Fox

So if those are the offenders, who's looking good? Are there any actually open open models?

Nick Petrić Howe

Well, the researchers highlight one standout model as being very open, and this is a model called BloomZ, which has been built by an international and largely academic collaboration. And as I said, the researchers point to this as an example that's one that's particularly open because you can see its code, you can see all the data that's gone in. It's got like a scientific paper around it. You can mess around with it. You can change things in it, with the application programming interface and all sorts of things like that. So, they say that this one's a real good example of it. And whether or not something is open is going to be very important as the EU's AI act comes into force. Because under the legislation, as its written at the moment, if an AI is open or open source, then there are less requirements that it will need to meet. So, the EU wants very extensive transparency requirements. If a model is deemed open by the EU, then you wouldn't have to do that, and there are other obligations as well that you may not have to meet if your model is open. But the problem is, and the reason that this is an issue is there isn't really a good agreement on what it is to be open in AI or open source in AI, and that's why the researchers have been trying to do this ranking.

Benjamin Thompson

And we've covered a lot on the show, and you've covered yourself Nick the– the black box that is so often associated with AI, and nobody really knows how their “brains”, in heavy inverted commas, actually work. So, you know, having an open system like this presumably would give researchers more– more insight into what's going on.

Nick Petrić Howe

Yeah, exactly because while you may never be able to know some aspects of what a model does, just because of the sort of complicated way in which they work and the way they sort of develop, like neurons coming together to make a network of weights and measures and all the rest of it. There are many things that we can know. We can know what a model was trained on, for instance. So, if a AI starts spouting, you know, hate speech or something like that, if you're able to see what it was trained on, you may be able to understand why it's doing that, and that could help you sort of improve it. And you know, if you're able to modify the model as well, which, if a model is like truly open source you would be able to do, then you can better tweak a model to be better for different circumstances and avoid things like that, and, yeah, more open the better, is the way that many researchers certainly view AI.

Benjamin Thompson

I mean, one could potentially argue that it's somewhat tricky for companies because they want to protect the products that they've developed while at the same time trying to meet whatever the definition of open is.

Nick Petrić Howe

Yeah. I mean, that is an argument, and companies may also want to protect themselves from litigation. If there's something that the model has been trained on that turns out to be copyrighted, they could be liable for that. But again, there are different ways in which things can be open. There can be other aspects of it that companies make open, and some companies have shown that it is possible to be open. Now from the company side, in the article, Lizzie reached out to a couple of the companies, and a Google spokesperson says that the company is precise about the language. It uses the term ‘open’ rather than ‘open source’ to describe its model, Gemma LLM, and it argues that existing open source-like concepts cannot always be directly applied to AI system. And Microsoft said, as well, they're being very precise, and they said they have made many things available. Meta did not respond to a request for a comment from Nature. So you know, there are definitely arguments to be had. And as I said, there's a not really a good definition. So one of the key things that needs to happen is researchers need to come together and actually develop a definition of what open sources, and particularly the EU, needs to define what that means. And there are concerns that as the EU tries to define this, that could be at risk of being lobbied against by different companies and that. So we'll have to see how this one develops.

Benjamin Thompson

I mean, it seems like this is a debate that will run and run for a while, especially as more players get into the game. But let's move on to our second story this week, and it well, it couldn't be more different. Dan, you've got a story about trains.

Dan Fox

Yeah. This is an article originally published in Scientific American about how longer freight trains drive up the risk of derailment. So, this is based on some research done in the US looking at these kind of enormous freight trains over a mile long the sort that we don't really see here in the UK. So, this study was published in a journal called Risk Analysis, and it showed that the odds of a derailment increase as a train gets longer. So, a 100-car train is 11% more likely to derail than two 50-car trains. And then if you carry that on, a 200-car train is 24% more likely than four 50-car trains to derail, and that's taking into account that it's less train. So even though that's one train versus four trains. Once you get to that sort of length, it's 24% more likely to derail. So, these are still relative risks, I mean, derailments are still quite rare comparatively, but as the freight train industry in the States looks to kind of bring down costs and be efficient, these longer trains are being looked at more and more, and this could become a safety issue quite quickly.

Nick Petrić Howe

So longer trains, more risk of derailing, is, I guess, the main point of this paper. But how have they worked this out? Is this based on, like, real life data, or is this sort of simulations of trains like, as they get longer does something wacky happen?

Dan Fox

So, they've used quite an interesting method, actually. So, it's from real life data, but the sort of data you would need to really build a sample in terms of how many trains of what lengths are running all around the country isn't publicly available, so they've used a method that's previously been used to study car accidents called “quasi-induced exposure”. And basically, while you can't get the data about what trains are running, the Federal Railroad Administration does record when there is an accident. And when they record an accident, they get data on how long the train was that had the accident, and where it was running and where it was in the country. So, using that data, they picked a different type of accident to a derailment. So, they picked what's called a “beat-the-train” type accident as a proxy. So that's when a car driver tries to kind of get out in front of the train at a crossing before the train comes across and gets hit by the train. So, using that data, they were able to build out this sample along with derailment data. So, the key thing being that the beat-the-train accidents are hopefully independent of train length. So, drivers who are kind of trying to get out in front of the train aren't necessarily looking at how long the train is, they're just trying to get out in front of it to save themselves some time. That allows them to use this data to build a model.

Benjamin Thompson

And this model then suggests that one very long is potentially riskier than four that sort of equate to the same length?

Dan Fox

Yeah, yeah. So the same number of cars, but if you put them all in one long run, that's statistically more likely to have a derailment than if you were to break those cars up into separate trains.

Nick Petrić Howe

And you know, you said, though it's hard to get some of the data on this, so do they have any sense as to why it is that these longer trains are at risk of derailment?

Dan Fox

Not really. So that's not what this paper is looking at. It's very much looking at the statistical likelihood of derailment rather than the reasons. And they do kind of do a bit of a literature review in their paper, looking at causes for derailment, which are sort of things you might expect, sort of different grades of rail and areas that are kind of checked more frequently being less likely to have derailments than areas that are less frequently used. But obviously that doesn't explain why these, specifically these longer trains, might be more prone to derailing. But in the article, there is a quote from a former locomotive engineer who conflates driving one of these very long freight trains to a slinky toy with the kind of couplings between each car. And they suggest that one problem that comes up in these very long trains is having a mixture of different cars with different materials being carried, so the weight distribution can be different in different parts of the train. And that makes safely driving this train that could be, you know, a mile and a half long along the track, very difficult.

Benjamin Thompson

And you mentioned there at the start then the trend seems to be that more of these larger trains are potentially coming in to service. I mean, what do you think this information will– will do for people who run the railways I suppose?

Dan Fox

So this specific paper, the authors said they, you know, they wanted to add this evidence into the discussion. And there is a discussion going on in the US around railway safety. There's currently a Railway Safety Act of 2023 being debated in the States, which, if enacted, would require the development of regulations regarding freight train length. Now, the authors of this paper that say, you know, they're not against long trains, and they, you know, point out the benefits in times of fuel consumption, they're just, you know, reporting this potential safety hazard. Having said that an assistant vice president at the American Association of Railroads has disputed the study's risk estimates because they say it fails to take into account different types of train or different car types, and make a good point that a 50-car train in the study could mean a train with incredibly long cars, or a train with very short cars, because they've used the number of cars rather than specific lengths. And so perhaps more research is needed, perhaps with a bigger data set.

Nick Petrić Howe

Well, it sounds like this will be useful information, as this railway act goes through all the sort of legislative hurdles, and hopefully will be less train accidents in the future. But for now, I want to know what you've got on the Briefing this week, Ben.

Benjamin Thompson

Well, I've got a story that I read about in Science, and it's based on a paper in Nature, and it's about wheat, okay, a staple crop of course, for many, many, many people around the world. And this is looking at efforts to improve it, to give it new traits. And these traits have come from an unexpected place. They may be new traits, but actually they're from old wheat,

Nick Petrić Howe

I see. So, is this more hybridization of wheat, sort of crossing them together, or is this genetic engineering to introduce these traits?

Benjamin Thompson

Well, this story seems to focus on the crossing, the hybridization of wheat, and that is super important in the story of this crop, right? So, current modern wheat was created during the 19th and 20th century, okay, through crossbreeding of a few key varieties, and it created, you know, a wheat that was really high yield, but it was kind of vulnerable to disease, you know, drought, things like this. And wheat is facing a lot of threats, you know, climate change, fungal infection, these sorts of things. So in the hunt to give this wheat new skills, as I say, the researchers went back into the past, and they were looking for genetic and phenotypic diversity from different types of wheat from different areas, okay. And these are known as landraces. And these landraces come from an antique collection. And many of these different sorts of wheat kind of disappeared a very, very long time ago, and they came from a collection that began in 1924 started by Arthur Ernest Watkins, here in the UK. He was studying wheat anatomy, and he amassed loads of different samples of grain from across the world, you know, 32 different countries, as I understand, 1000s of samples.

Nick Petrić Howe

Man loves his wheat.

Benjamin Thompson

Well, clearly he did right. And it turns out, not just him. Curators have kept this collection going, right, and sowing and collecting seeds every few years, with the exception of during World War Two, the article states when some of these land races were lost. And what's interesting is there's kind of a snapshot in time, right. These wheat land races were collected a long time ago, and the researchers behind this study really wanted to see what sort of made them tick, I guess.

Dan Fox

So, what were some of the exciting features of past wheat that the researchers found?

Benjamin Thompson

Well, they found a bunch of different things actually, and it was an absolutely Herculean effort. So the wheat genome is enormous, right. Lizzie and I talked on the podcast before about how odd some plants are. They just have enormous genomes, and apparently the wheat genome is 40 times larger than the rice genome. So the team behind the work did a bunch of sequencing, and the article says that they had to post a suitcase full of hard drives to their collaborators with all this genetic data in it. And in total, they did 827 historic land races and 208 modern varieties – a huge amount of work. And one of the researchers describes it as a ‘gold mine’ of kind of genetic data. But genetic data on its own is one thing, right? Having a sequence is one thing. And what the team had to do is to work out which of these land races could have desirable traits. And what they did was they crossed them with different wheats. They made loads of different breeding situations, and together, they created over 6000 unique populations of wheat, growing them in greenhouses and in fields in the UK and China. I mean, this is like a decade's worth of work. And then they measured the different traits and developed algorithms that could sort of link those traits back to this genetic sequence so they could see, okay, well, that gene or that area of the genome seems to be important in making them taller or making them less susceptible to heat, or whatever it is, and so they've gone through that. And I think, as I said, that it's kind of an interesting snapshot. And a lot of these samples came before the mass use of cheap fertiliser, right? So they were interested in how some of these land races would cope in low nitrogen environments, because presumably they must have grown before fertiliser was widely used. And it turns out, they have found a cluster of genes that looks to be related to nitrogen use. And previous work related to this had found a gene that seems to give resistance to this fungal disease called wheat blast. Now this is pretty bad, and it obviously can decimate wheat crops, and the paper says that breeding programmes involving this gene have already started happening around the world, which is kind of interesting.

Nick Petrić Howe

No, that's super interesting. And obviously, as you said, it's a staple crop. It's essential for a lot of people in the world. But I wonder, how does this transfer from this massive experiment to farms and then eventually people's plates?

Benjamin Thompson

I mean, that's a really good question. I think it is the long game. So, this is kind of really the start. It's kind of a toolkit, they describe it as. Now, there are other efforts involving trying to enhance the ability of wheat to grow in salty soil, for example, and these sorts of things, but it takes a while, right? So breeding in a desirable trait is one thing, but you need to make sure you don't breed in an undesirable trait at the same time, right? It's not just a kind of a one and done. So there'll be lots of crosses and lots of plant breeding going on to make sure that wheat with these new old traits, or old new traits, I suppose, are available. And in the article, someone says that this could take, you know, a decade or even longer for this kind of style of plant breeding to come up with the goods. So it could be a little while yet, but I think it really opens the door to researchers really getting in there and working out, as I say, what makes wheat tick and well, it's a ‘gold mine’, as one of the researchers is quoted as saying.

Dan Fox

Well, that story’s made me hungry for a sandwich, so I think that's time for lunch.

Benjamin Thompson

Agreed. I think that's a good place to end this week's Nature Podcast. And listeners, for more on those stories and where you can sign up to the Nature Briefing to get even more of them delivered directly to your inbox for free, head over to the show notes for some links. And all that remains to be said this week is Nick and Dan thank you so much for joining me.

Nick Petrić Howe

Thanks for having us.

Dan Fox

Thanks very much.