Transcript for A.I.’s Data Wall + a Surprise Privacy Bill + What Happened to the TikTok Ban?

I got a delightful piece of listener feedback which was an email that I got from someone named Molly who told me that she apparently has a farm and that she had named her Rooster Kevin after me. She said he's a sarama, which is the smallest breed of chicken. He's very cute and has a nice disposition. And whenever he's crowing, she yells, shut up Kevin. So thank you Molly for informing me about Kevin the rooster and I would love to meet him someday.

I'm Kevin Bruce and tech colleges in the New York Times. I'm Casey Newn from Platformer.

So Casey, it has been another big week in the world of AI, a GPT-4 got a new update from OpenAI as did Google's Gemini.

And there are some new other new tools out including this new AI music tool that I've been playing around at this morning. But I don't want to talk about the new stuff that's coming out of the AI world this week because there is something much bigger and juicier that we have to talk about, which is this AI data race. Yes. There has been some really interesting news over the past few weeks that I think we should talk about because it's something that you and I have been discussing with just a lot of excitement.

Yeah, so before we get into this discussion, we should just make a blanket disclosure slash caveat. The New York Times is suing open AI and Microsoft over their use of copyrighted materials to train their models. This is all obviously very relevant to that ongoing legal battle. So just like we are not speaking on behalf of the New York Times, even though this is in New York Times podcast. So this story, there's a lot in here, but it is basically the story of three companies, open AI, Google, and meta, racing to get as much data as they can from any sources that they can get their hands on. There was an article in the Wall Street Journal that came out last week about the lengths that some of these companies are going to to find new sources of data. And then, uh, over the weekend, there was a blockbuster story from my colleagues at the New York Times, Kate Metz, Cecilia Kong, Sheriff Frankl, Stuart Thompson, and Nico Grant called how tech giants cut corners to harvest data for AI. And this was sort of the story that I and lots of other people had been waiting for for the better part of a year.

So the story says that at Open AI, One of the things that they have been doing to train their models is scraping YouTube videos. The story says that an OpenAI team transcribe more than a million hours of YouTube videos to feed into GPT-4 and the Greg Brockman OpenAI's president personally helped scrape some of these videos and feed them into GPT-4.

So this is controversial because YouTube, which is owned by Google, has a policy against using their videos to build external products and to use any kind of automated scraping tools or bots to kind of pull information out of YouTube in an automated way.

So right, when people at Google raised this, apparently one of the discussions that they had internally was, well, we can't exactly take OpenAI to task for doing this because we are also scraping YouTube videos to train RAI model.

Yeah, the story also says that last year Google broadened its terms of service that we in a way that would allow the company to tap into publicly available Google Docs restaurant reviews on Google Maps and other sort of user generated content to feed into their AI models. And this is apparently something that they needed to change their policies in order to be able to do. So Google has said that these changes to Google's privacy policies were made for clarity and they said they didn't start training on our digital types of data based on this language.

So that's OpenAI and Google. The story by my colleagues also dives into what's happening at meta, which I would say has been a little bit of a laggard in the AI race, but is really scrambling to catch up. And part of the reason that they are sort of lagging behind OpenAI and Google is because they just don't have as much good high quality data to use to train their models. Because if you're Google, you've got the entire internet that you've, you know, indexed and crawled for your search engine, you've got YouTube and, you know, Google docs and things like that. When people are posting on Facebook and Instagram, they're not giving you the kinds of sort of book length or long essays that have been carefully fact checked by experts. They're not posting like high quality training data. They're posting memes. They're posting things about their cousins wedding or their bar mitzvah.

Right, so my colleagues dug up that Meta's vice president of generative AI at one point told executives that his team had used up almost every available English language book essay poem and news article on the internet and he told his colleagues that they basically couldn't match chatGBT unless they had more data. So they start sort of meeting inside Meta to try to brainstorm ways to get around this problem. One of the ways that they propose possibly doing this is buying Simon and Schuster, the publisher to sort of get access to their whole back catalog of books, which they can then presumably feed into their language models to train them. They also talked about, you know, basically licensing, paying up to $10 a book for the rights to new titles. And at the same time that they were doing these, this brainstorming, they were also kind of debating amongst themselves whether it was legal or ethical to train models this way on copyright or data without the permission of the original authors. And it sounds like they decided, yeah, it's fine, just do it. Yeah, they basically were like, look, you know, Are we going to get sued over this possibly? Are we maybe getting into some legal gray areas that will have to be litigated? Yes, but they basically decide according to this article that there's such an industry precedent around training large language models on copyrighted data and claiming fair use that they are just going to plow ahead anyway.

Yeah. So, there's obviously a legal question here. And this is something that, you know, then New York Times is actively litigating right now and that many other publishers are starting to litigate as well over whether this kind of mass unauthorized use of copyrighted data to train large language models is in fact legal and that will be decided by the courts we don't know yet that is a big sort of question mark hanging over the industry right now the large AI companies are all basically saying versions of the same thing in response to some of these concerns about how they collect and use copyrighted data to train their models they say you know we train our models on publicly available data which basically means look we're not like breaking into computers or hacking into anything to get this data. It's all out there on the internet and they believe that it is protected under fair use.

So what was your reaction to this article?

And I think there's a question here about whether they could have done this any other way. I mean, one of the questions that the article raises is, like, is it even possible to build something like a chat GPT without using copyrighted information for free? Like if you had to go out in license every article, every book that you wanted to feed into your large language model, would that even be economically feasible? And the answer appears to be, no, you just, you couldn't do it.

Yeah. Yeah, so I think this article and others that have come out like it about kind of the behind the scenes process here are really illuminating because they show why these companies have been so reluctant to talk about this. I mean, one thing we've talked about in the show before is a couple of years ago. AI companies would gladly tell you where they got their data to train their models. You know, I remember having conversations with people at the big AI labs, you know, back in the kind of GP3 days, and they would tell you, oh, yeah, we scrape red it. You know, we go on to Wikipedia. We grab these these data sets that are that are out there and publicly available. Now, if you ask them, they basically short-circuit and they start looking for the exits because they have been told repeatedly by their lawyers, do not say anything about where we get the information.

Yeah, I think these companies obviously want their models to keep improving and sort of one up in each other. And they don't really have a lot of ideas about how to do that except to get more data to get more compute to train bigger and bigger models. Because so far, at least for the past four or five years, that's all you've needed. is more data and more compute equals better model. And so if they are in fact running out of data, it raises the possibility of whether we are kind of reaching an asymptote. They will have scraped all of the high quality text sources, all of the academic papers, all of the, you know, they'll have transcribed all of the YouTube videos and podcasts, all of the news articles and Wikipedia articles and legal articles. They'll just have they'll have exhausted the supply. And at that point, the question is where do they go from there? And this is a very active area of discussion and argument inside the AI industry.

Yeah, my AI cannot be literally God T-shirt is.

Maybe the lawyers that open the eye were just asking chat GPT all along.

I think you're out of something there. I think these language models are sandbagging. I like that you did that exercise. I do not think that these models are going to tell you how to improve themselves so that they become God, but I'm glad you asked. Yeah, it was worth a shot. There is also another wrinkle to the story that we should talk about, which is that there is a way that these, some of these AI companies are starting to believe that they can get themselves around the data wall, which is through the use of synthetic data.

So the case that you'll hear from people in the AI industry is basically look, you know, yes, to get these language models to be even reasonably good, we had to ingest all of this data from the internet. But now, or very soon, they will be so good at creating kind of human caliber data that we can actually use that data, the outputs from the model to train the next generation of the model. So instead of going, you know, and getting, you know, a million hours of YouTube videos, you could just ask a model to generate a million hours worth of transcripts of something like a YouTube video. And it would give you data that was for all intents and purposes, just as good as the kind of real copyrighted stuff that's out there on the internet.

So it's an area of active debate inside the industry. I've talked to some people who say, yeah, synthetic data. It's getting really good and pretty soon we're not going to need any of this kind of original human created data because we can just ask the model to create a training set and then train the next generation of model on that training set. And there are other people inside the industry who say, you actually can't do that because, you know, we've talked about in the show about some studies that have found that if you feed AI generated data into an AI model, it does tend to degrade the performance of that AI model. The Wall Street Journal compared it to the computer science version of Inbreeding, which I love. and it tends to weaken the performance of the model over time. And they also, I've talked to people in the industry who say, look, synthetic data just isn't going to be good enough. It can teach us stuff that we already know because it is essentially derivative from all of the human data that was fed into the model, but it can't produce new things. It can't tell us things that we maybe don't know come up with new breakthroughs, new discoveries, new ideas, because if it's not, Part of that corpus of human data, if it's just the sort of extruded outputs of some other AI model, it's never going to be as good as the stuff that humans are coming up with from scratch.

How I wouldn't find a tumor. I'd go to the doctor, but I think they call those things scans.

So yes, we do see that synthetic data is getting better. These language models may be able to make some kinds of intelligence gains using only or mostly synthetic data. And the lawyers at these AI companies are all pushing them to use more and more synthetic data because it's not copyrighted. You actually can't copyright synthetic data under our current copyright regime. And it lowers the risk that these companies are going to get sued by people who publish information on the internet. But there are other people then who say, well, wait a minute, if you needed to ingest a bunch of copyrighted material to create the model that was capable of making the synthetic data that wasn't based on copyrighted material, you're all you're basically doing is laundering the copyright. And so people, you know, that I've talked to also think like synthetic data is not going to solve these companies legal challenges because in order to be able to create the synthetic data, they had to ingest a bunch of copyrighted information.

Oh, I totally disagree. I think it is still going on. I think there are still companies that are training models using copyrighted information and they will probably continue to do so unless and until a court says you're not allowed to do this.

So, Casey, maybe there are people out there who think, you know, these guys, they are hopelessly biased when it comes to this issue, you know, the New York Times is suing over the use of copyright or material in generative AI programs made by open AI and Microsoft. What is the strongest version of the argument that you have heard from the AI companies about why they believe they are not only legally allowed to train their models this way on all this data without the permission of the people who made the content. But why they believe it is actually ethical to do.

Yeah, and I should say, I think one of the most valuable things about articles like this is that they establish what in law is sometimes called men's rare, which is the guilty mind, right? It's the fact that as these companies were collecting data this way, they did have people inside these companies saying, wait a minute. This feels a little too close to the line for my comfort. Uh, we may get ourselves into trouble doing this. And it just shows that every time those issues came up at one of these big companies, the response was, well, other people are doing it. So we're going to do it too.

So it does show not only that there is this intense race for data among all the big AI giants and anxiety about this idea of the data wall that they may soon hit when they run out of human creator data, but it shows just how controversial this practice has been even inside the companies that are doing it.

Okay, so do you have any messages to send to the AI scrapers that are ingesting the transcript of this podcast right now?

Yes, please be benevolent. We love you.

You will talk about data privacy and how a new bill might actually have a chance of getting it under control.

Okay, so we just talked about data and AI and how AI companies are harvesting and using material to train their language models. But I want to talk about something else that has been in the news this week, which is this data privacy bill.

No, come on. I've been picking my nose this whole time. So you have been talking specifically to me about the need for a federal data privacy bill in the United States for many years now. I feel like this is the thing that you sort of like lean on hard whenever anyone talks about data privacy. You're like, why don't we have a freaking bill already?

So something that happened this week that caught my attention is that we actually might have a federal data privacy bill. Yeah, this is crazy. Yes. So if you say things enough, they manifest. That's what I learned from reading the secret. And so it seems like you have personally manifested and attempt to pass a comprehensive federal data privacy bill in these United States.

Yeah, so we have been waiting for Congress to pass a bill like this for more than 20 years. There's near universal bipartisan agreement that the federal government needs to do something about data privacy in this country. Many other countries have already passed very. Most countries have passed data privacy bills. And we've had lots of proposals for addressing data collection by tech companies and other companies. But up until now, none of these bills have gained significant bipartisan support.

Yeah, so this bill was proposed this week by two co-sponsors, both from Washington State, Kathy McMorris Rogers, who's a Republican congresswoman in the chair of the House Committee on the Energy and Commerce in Maria Cantwell, who's a Democratic senator. She's the chair of the Senate Committee on Commerce, Science and Transportation. They unveiled this big sort of comprehensive federal privacy bill, and it appears to have a shot of actually passing.

So to talk about this proposed bill, what's in it, how compares to other privacy laws and whether it actually could become a law in this country, we've invited on someone who we consider to be one of the leading experts on data privacy, Trevor Hughes. Trevor is the president and CEO of the International Association of Privacy Professionals. It's a group that does training and provides education for privacy professionals around the world.

And Trevor has been following this bill closely, and he's here to talk to us all about it. Let's bring in Trevor Hughes. Trevor Hughes, welcome to Hartford. Hi, it's great to be here. Great to have you. So just give us the kind of your 30 second overview of what is in this bill, the American Privacy Rights Act.

Some of the biggest data privacy scandals and stories that we talk about in this show, things like GM's sale of customer driving data to companies like Lexus Nexus, which we talked about with Kashmir a few weeks ago, are not done by what we would consider tech companies. but something that we are all learning as a society is that everything is a tech company and a lot of things that we may not think of as collecting our data actually are. So would this law primarily apply to tech companies, per se, or is there some sort of other broader definition of a data collection company that would apply here? Would it apply to something like a car maker?

Why is this happening now? I mean lawmakers in the US have been trying to enact some sort of federal data privacy law for years. There have been other similar bills that were proposed and then killed or never made it out of committee. So what is it about this moment that has made at least some lawmakers more willing to consider going forward with something like this?

So basically, let me just clarify what I just heard from you and you tell me if I got wrong. So you're saying that part of the reason that we're at this moment now, where the US, which has lagged behind many other nations in creating national data privacy laws, part of the reason that we're here at this juncture where it may actually be viable is that there is a bunch of other stuff that lawmakers want to do that kind of requires as a prerequisite having some kind of national data privacy law. If you want to ban TikTok or make it stop exporting user data, if you want to address kids online safety, you have to have sort of a bedrock of privacy law on the books before you can do any of that. Is that what I'm hearing?

Right, preemption being kind of the ability of the federal law to kind of supersede and supplant these individual state laws, some which already exist and private right of action, being like individuals can sue companies for misusing their data. I want to ask Trevor about how this compares to data privacy laws in other countries. In Europe, GDPR, we know went into effects six years ago and there have been some studies about sort of what the effect of GDPR has been on companies that operate in Europe. One study found that the main effect that GDPR had was just making small and medium sized businesses a slightly less profitable. And that basically, if you're a big company, if you're Apple, if you're Google, if you're meta, if you have tons of money to sort of hire all the people that you need to comply with these regulations, they're not seeing a ton of hit to their business from things like GDPR. Do you think that this American version of a privacy law might do essentially the same thing might sort of take a little profit out of the pockets of kind of mid-sized companies and that the big companies would essentially sail through unscathed?

Because they're already having to comply with all these other countries. Come privacy.

I'm just curious for the average person who is listening to this and thinking, well, you know, maybe I have spent some time in Europe and I use the internet there and the only difference that I see is that I have to click through a bunch of like, allow cookie pop ups. That's sort of the only tangible difference that I feel and maybe they're wondering to themselves like, well, what are the stakes of this for me? How will my experience of using the internet or using internet connected appliances in my house. How will any of that be different if this bill passes? What's your answer to that?

Well, when it comes to the APRA or appra as a calling it, I am a little less apprehensive after talking to you today, Trevor.

All right, Trevor, we'll let you go. Thanks so much. Thanks, Trevor.

We'll check in on the TikTok ban that we talked about a few weeks ago and why it hasn't happened yet.

It is, and I know this because I've spent a lot of time on TikTok over the past few days looking up various things related to WrestleMania and why everyone keeps talking about it.

Yes, and we just talked about this privacy bill that is now sort of being proposed in Congress and some of the various hopes out there about whether it might actually pass. The TikTok one I will say is a case where I was too optimistic. I thought that lawmakers were sort of rounding the corner that they were heading toward actually passing this bill to force bite dance. The Chinese owner of TikTok to sell it. And it's been about a month. since we first started talking about that. And so far, TikTok is still on my phone. I think it's still on your phone. And it still exists.

Right, because President Biden had already signaled that he would sign this bill if it arrived on his desk and sort of the only big hurdle remaining was to get the Senate to pass the same bill at the House had already passed. So what happened in the Senate?

What are some of the explanations you've seen and heard for why this bill seemed to lose momentum after having so much of it in the House?

Right. The one change that I have seen to the content on my TikTok app when I open it in recent weeks is that there's just a lot of pro TikTok marketing that the company itself is putting on two people's TikTok apps. So like I opened my TikTok the other day and I got this video that was basically from a TikTok creator sponsored by TikTok explaining why millions of small businesses depend on TikTok and why it would be a super bad thing if it went away. Do we think that TikTok's efforts to kind of mobilize its user base through the actual TikTok app had it effect?

I mean, my assumption was that it was some combination of, you know, aggressive lobbying, the kind of Trump factor that you mentioned, and also just kind of the fact that Congress can't really seem to focus on anything for longer than a few days. Like there's a kind of shiny object problem here, which is that something comes up like TikTok. And everyone gets really excited about it and they build some momentum and all the politicians want to go on cable news and talk about why they're so excited to do this thing. And then they kind of, you know, something else happens, you know, the eclipse or something happens in Ukraine.

But there's just kind of, there seems to be a lack of perseverance when it comes to actually moving these bills to the point where they might actually become lost.

We'll have you listen to the entire thing beginning to end. I will say it's in the top quartile. No, obviously I didn't know. That actually surprises me because I thought that after form president Trump came out against the TikTok ban, my assumption was that most Republicans were gonna run screaming from this bill because they didn't want to do anything that could put them at odds with him. Obviously he is sort of the standard bearer of the party. And so it is surprising to me that Mitch McConnell broke from former president Trump here. What do you have any sort of inside about why that might be.

So given that it now appears to have lost some momentum, this attempt to ban or force the sale of TikTok, but that maybe it's coming back because Mitch McConnell wants to focus on it. Where do you handicap the odds of a TikTok ban at this point in time?

Yeah, so I think there are a couple ways of analyzing the position that TikTok is in. One is by sort of parsing the statements of lawmakers who are actually in a position to pass this bill and do something about it. The other is by looking at TikTok itself and by dance itself and how they are reacting and what the mood is over there. So what can you tell me about by dance and how they are proceeding at this point? Are they acting like a band of TikTok or a force sale as imminent?

No.

This is huge news.

And it also raises a very thrilling possibility, which is that teens might actually be able to justify spending hours a day on TikTok by their parents by saying, Mom, I'm paying my way through college. This isn't mindless entertainment.

You skipped my favorite one of these. Did you do the TikTok or by dance rather has a homework cheating app? I just learned about this this week. I feel like every week, I learn about some new piece of like the bite dance, you know, cinematic universe. And this week for the first time, I learned about the existence of an app called Goth, G-A-U-T-H, not Goth, like the face I had. So this is an app that bite dance has created that basically lets you point your phone at a homework problem, like a math problem or a science problem or an essay prompt, and it just uses AI to like solve your homework for you. And they are marketing this aggressively to teens. Which is just my favorite. piece of this because if you are as a parent of a teenager, we're not already pissed off at TikTok for like common during hours and hours of your kids screen time a day. Now they're like, we're gonna give you another reason to hate us. Here's an app that lets you cheat on your homework.

I have no idea. Honestly, I have lost faith in my ability to accurately forecast what is happening to TikTok and by dance because there was a point where I thought it was sort of a done deal that we were this bill, this attempt to force a sale of TikTok, had so much momentum behind it from so many different parts of the political universe that I thought this is maybe the one thing that Congress can agree on right now and actually go forward and get it done. And then it just, it seemed to stall out. And so I just have a very little ability to, you know, maybe I could spend a day making calls to lobbyists on K Street and lawmakers and sort of piece together some more accurate forecast, but I just do not know. look, I think there are still good reasons to want to support a forced sale of TikTok by bite dance. Like you and I made the case several weeks ago for why we were sort of leaning toward being none of those underlying reasons have changed. And in fact, like the first thing I thought When I opened up my TikTok app and I saw all of these ads, these like pro TikTok ads that were actually put there by TikTok the company. I thought, well, this is a demonstration of the power that this company has to kind of shape the views of Americans. And I do worry about that and none of that reasoning has changed, but I guess, man, I am just so cynical about Congress and their ability to get anything done before being distracted by the next shiny object. So I guess if I had to handy cap and I would put it in, I don't know, maybe 50 50 at this point, but that's a total guess.

Have you seen the thing on TikTok? Wait, I'm going to test your knowledge. Test me. The TikTok universe. Because one particular thing about TikTok is like, you know, everyone's TikTok is so individual that I never know like what anyone else is seeing. So I have years could be totally different than mine. Does the phrase, what's up, brother? Me or anything to you? Sadly, it does not. Okay. We can cut this. There's a thing going around that TikTok where you're supposed to go up to your boyfriend or your husband and go, what's up, brother? And then what do they do? I'm sad that I know this because they did a deep dive on this last night. You're supposed to go. Tuesday, Tuesday. Special teams, special players, special plays.

The reverse mysteries. I only take. I never give. These are apparently, and I know this because I was so mystified by this TikTok trend that I had to go all the way down the rabbit hole. These are lines, these are catch phrases from a popular Twitch streamer known as sketch that apparently men, especially men who are into sports, no very well.

You're only interested in fake sports. That's right. All right. Well, in conclusion, TikTok is a land of contrast.

Before we go, thank you to everyone who has sent in so many amazing anecdotes and stories about how you are using generative AI at work both successfully and very unsuccessfully. Keep in common, we are putting together an episode where we will go through some of our favorites and talk about them with you and we just love hearing from you.