Jay (Narration): At the end of last year, the New York Times published a video report. 

 

Video Clip: When a 2,000-pound bomb detonates, it unleashes a blast wave and metal fragments thousands of feet in every direction. [non-English speech] Sometimes, 2,000-pound bombs leave giant craters in the earth, like this strike in south Gaza in November.  [non-English speech] [non-English speech]

​

Jay (Narration): The report had a shocking finding: during the first six weeks of the conflict in Gaza, through October and November, the Israeli military dropped at least 200 of one of its biggest and most destructive bombs, the 2,000 lb bomb, in areas of Southern Gaza that it had publicly designated safe for civilian evacuation. It was a stunning revelation at the time – and the story was made all the more interesting because of the way it was reported. Journalists at the NYT used artificial intelligence to gather data and make its assessment.

 

[Bot Beat music swells then settles]

 

Jay (Narration): This is Bot Beat, a new podcast that tracks how journalists are harnessing AI in their reporting. I’ll do quick chats with reporters and key you in on how exactly these emerging tools can affect your work. I’m your host, Jay Kemp, and I’m an independent journalist reporting on AI within my field. Today I’ll be chatting with Ishaan Jhaveri. He’s on the Visual Investigations team at the New York Times.  This isn’t an episode about the war in Gaza, but about the way new technologies changed how journalists approached a particular investigation. This kind of story would have been reported differently in the past, if reported at all – but I’ll let Ishaan explain why this finding matters, why new tech was needed here, and how they even used AI to get it done... 

 

Ishaan: Yeah. So I think the framing of our story was specifically the use of 2,000 pound bombs in areas that were designated safe. We felt that it was a choice of weapon that we felt was worth shedding light on – given the evacuation orders and given that these weapons were being used in South Gaza, which is where civilians in Gaza were asked to evacuate to.

 

Jay (Narration): I asked Ishaan what his team was looking for. In a way, it seemed pretty simple – bomb craters. Specifically, craters in Gaza’s sandy soil that matched the distinctive characteristics of a 2,000lb bomb.

 

Ishaan: Yeah, sure. So really, the, the idea to look for craters in satellite imagery came from the fact that, we, as the Visual Investigations team, often use satellite imagery in our reporting, and we had been using it in Gaza as well, and we had been seeing certain types of things like the proliferation of tents or, you know, razing of buildings, certain things that are very visible from satellite imagery. And so we knew that there were things that you could look for in satellite imagery that might translate to important findings for investigative work. And so that's like the backdrop behind which we decided to look for these craters in satellite imagery. Now, we wanted to try and make a very comprehensive statement about the use of these 2,000 pound bombs in the area south of Wadi Gaza, which is where the evacuation order for roughly the first six weeks of the war was telling people to evacuate to. We wanted to be sure that we had counted all of the craters in this area, and not just the ones that we could see. And, to do this manually would have been very time consuming. And so that's where the AI came in, where, we were able to train an AI to look for craters in satellite imagery. We trained it using craters from other parts of Gaza so that the training data would resemble the eventual context that we would run the algorithm in, as closely as possible.

 

Jay: And just to kind of slow down a bit and rephrase: what is actually the artificial intelligence that is being used here? What does that tool look like and what is it taught to do?

 

Ishaan: Sure, sure. So yeah. So the class of algorithms that allows you to look for objects in imagery and satellite imagery in any imagery are called object detection algorithms. The one that we used is called ResNet, and we accessed it through an interface built by a company called Picterra. And Picterra basically just gives you an interface where you can upload these heavy satellite images, these, large, you know, hundreds of megabyte satellite images. And it gives you a very clean interface to annotate instances of craters as training data and then also run the trained algorithm on the new satellite imagery, to get the final craters outputs that we used in the story.

​

[Video clip fades in]

 

Video: The A.I. tool detected over 1,600 possible craters. We manually reviewed each one to weed out the false positives, like shadows, water towers or bomb craters from a previous conflict. We measured the remaining craters to find ones that spanned roughly 40 feet across or more, which experts say are typically formed only by 2,000-pound bombs. Ultimately, we identified 208 of these craters in satellite imagery and drone footage, indicating 2,000-pound bombs posed a pervasive threat to civilians seeking safety across south Gaza. 

 

[Video clip fades out]

 

Jay: And you said you used over 100 different images.

 

Ishaan: Over 100 different instances of craters. Not over 100 different training images, but each instance of a crater that you annotate is almost treated as like a training image. So you could call it over 100 training images. But it came from a few satellite images, probably like 6 or 7 satellite images.

 

Jay: How did you figure out how to do this? Was this entirely your conception, and on a broader scale, how are you so comfortable with different kinds of, you know, online tools and algorithms?

 

Ishaan: Yeah. So, I have a background in computer science. I studied computer science at undergrad and then I did a masters, where I focused on computer vision and natural language processing. Essentially, if something has a clear enough visual signature, it is typically, an algorithm can typically find it in a large amount of imagery.

 

Jay: Yeah. So based on that, at this point, do you consider yourself more of a journalist or a computer scientist or both?

 

Ishaan: So to be perfectly honest with you, I think I started as a computer scientist. I've made the transition to journalism, relatively more recently compared to how much longer I feel like I've been comfortable with computer science. I think now I'm trying to develop the aptitude for actually posing investigative questions and looking for areas in which to investigate, which I'm still new at. And this is an area that I want to improve at. So I think I'm a computational journalist. But I, you know, I'm more comfortable and have more background on the computational side than on the journalism side.

  

Jay: Absolutely. And I think that's really interesting, I think because journalists are always looking for those needle in the haystack moments. That's why we do document reviews and things like that. So I understand that you have certain tools and access at the times that smaller publications may not have in general specifically for this story. Could other papers have done this?

 

Ishaan: Yeah. So I think, this was, you know, a team effort, but I think the finding that over 200 2,000 pound bombs were used in the area south of Wadi Gaza. That finding, you know, I think came from essentially our subscriptions to Planet, which is how we got satellite imagery and our subscription to Picterra, which is how we were able to use this tool that made the object detection process easier. And so to do that, really just required those two subscriptions and the knowledge of how to use, object detection and then the verification of the results, which, just me and or one of the other reporters were able to do it together.

  

Jay: And, you know, I imagine this is kind of unlocked. You know, this isn't the first time that you've used a tool like this perhaps – I'm familiar with some of your other work. So in general, what are the ways that you would like to continue to use object detection tools or other kinds of artificial intelligence to further journalism at The New York Times or outside of it?

 

Ishaan: Yeah. So I think, again, like, looking for needles in haystacks is a big one. So that could be scraping a bunch of court documents and looking- So one of the, first data journalism stories I ever worked on, back when I was at Gizmodo in 2018, we scraped a bunch of court documents, and we were looking for patterns of how this predatory auto loans company was looking for clients with low credit and then eventually suing their own clients. And that was like a large part of its business model. So that came from a scraping- building a tool that scrapes court cases and then, using algorithms to look to extract information computationally from those from the text of those court cases, to build a database of court cases and of the judgments in those court cases and the amounts that the judgments were for and stuff. Which allowed us to make this comment about the company's business model. So, yeah. And then like, you know, another example – I didn't have a huge role to play in this story, but in the story that several people on the Visual Investigations team worked on in 2022 on war crimes in Bucha, one of the things we were interested in doing is we had several thousand hours of CCTV footage, most of which was not relevant to our investigation, but some of which did show soldiers. And so, one of the things I worked on is a way of, extracting the frames or extracting the timestamps that, you know, soldiers uniforms or like certain types of patches or anything can be seen, so that we could take this gigantic trove of video and an algorithm could help us figure out which parts of it to look at so that we could feasibly use it – versus if we had to spend thousands of hours watching this, we probably feasibly couldn't use it.

 

Jay: So it kind of sounds like you're saying that these AI tools are opening up new story opportunities that we didn't have before.

 

Ishaan: Yeah, absolutely. I would say, you know, if you look at the 2,000 pound bomb story or like the example I gave you of looking for frames and thousands of hours of footage, it's not that an algorithm is replacing labor that would have been done by a human. It's that it's opening up new areas of discovery that maybe humans wouldn't have thought to spend time on. Because for a human at a newsroom to spend 1000 hours watching CCTV footage or, or spend, you know, innumerable hours combing through like a square inch at a time, parts of satellite imagery and counting craters like these are just things that newsrooms would probably decide that the people who might do them, their time is better spent talking to sources, you know, doing other kinds of reporting. So I don't think any of this is really replacing work that would have traditionally been done by human journalists. I think it's just giving human journalists tools to ask the types of questions that they didn't have before.

 

Jay: What advice do you have for other journalists who are just starting to think about experimenting with AI in their stories? So what advice would you have for other journalists who are just starting to think about experimenting with AI in their stories, who may be intimidated?

 

Ishaan: Yeah. So I would say, my advice is don't be daunted by the fact that you might not know how to code because coding is just a language. It's just a way for humans to write instructions and then computers to carry out those instructions. But today there's a lot of interfaces where you can upload imagery. You can draw boxes around the kinds of things that you're interested in looking for in those images, like I did with Picterra . And then just press train or press enter or whatever it is. So, you don't need to know how to code, I think, to be able to think about how AI could be useful for an investigation that you might want to do. So, don't let the fact that you don't know how to code stop you.

 

Jay: That's really great. And I just on a personal level, for my last question, I really want to ask you, what are you most proud of for this story?

 

Ishaan: I'm proud of the way that a group of people with different skill sets, with graphic skill sets, with computational skill sets, with storytelling skill sets. We're able to come together and, using insights from satellite imagery, using insights from previous work, using insights from contacts in the US military, come together and produce a rather substantial finding. I think that does hold power to account. And was cited, for example, in the International Court of Justice case. Just to give an example of the, the value and the impact of the, of the finding that we produced. So, yeah, I'm really proud of that.

[Bot Beat music swells, then settles]

 

Jay (Narration): I’ve been speaking with Ishaan Jhaveri of the Visual Investigation team at the New York Times. In this case, AI turned out to be a powerful tool for searching for truth in a war zone – to try and hold power to account. Next episode, I’ll be examining a not-so-flattering use of AI in journalism – and the repercussions that had for one prominent sports publication.

 

Maggie Harrison: But kind of once you zoom in a bit on the faces, there is that very uncanny valley feeling to them. You know, realistic from far away. But you zoom in and it doesn't quite look human.

 

Jay (Narration): A story about fake reporters – next time on Bot Beat. I’m Jay Kemp. I hope you’ll tune in.

 

[Music swells, then fades]

 

Ishaan: This story is really an example of “It Takes a Village.” There were so many people and I'm going to go through and name and thank them. Robin Stein and Hayley Willis. Michael. Reporters. Our video producers, Danielle Miller and Natalie Reneau. Aaron Byrd, who did the graphics in the web story. So the maps, the graphic, the visualizations of the impact of, the different types of bombs. And Julie Schaefer, who helped with the graphics for the print version of the story. Whitney Hurst and Dave Botti, our editors, Angela Roth, Eric Schmidt, Neil Collier, Yuzer Aliu, Christophe Cuttle, John Isman who contributed reporting. The people of Gaza, whose footage we relied on heavily to tell the story. Our translators, Omar Dewji, Lara Zoabi and Amir Pamuk. Andrew Sanden, who helped with print, and Rebecca Sooner, who helped with social media? And then finally, you know, Tara gave us a lot of resources, and then Marc Garlasco and Brian Castner, who are our military experts, who we relied on a lot for expert testimony.