2022 will be known as the year when a lot of AI technology became accessible to hackers and developers. With the recent explosion of capabilities as shown by Chat GPT and Stable Diffusion, it is clear that AI applications to our daily lives are going to explode in 2023. This could be one of the most exciting technological advances since the iPhone.

What’s harder to do is to make sense of the companies best positioned to take advantage of this breaking dam, what the market looks like and where it is likely headed in 2023. With core expertise in applying AI to search, using existing AI technologies, and training our own models, here is our analysis of the industry landscape and our predictions for what is to come.

Let’s start off by categorizing the AI players into a 5 classes.

These include Anthropic, OpenAI, Cohere, and AI21. All of these players will all have instruction following models that are more or less substitutes for each other. Their businesses will all be based on an API model. Their APIs will all converge to (a) generations (b) embeddings (c) fine tuning (d) classification. They will all attempt to recruit AI frontend startups (see below). Inspired by the likes of Stripe, they will use an “index YCombinator” business model, and outsource GTM to AI frontend startups. The moat for these companies will be the quality of their model,  their ability to recruit loyal customers who use their APIs.




These are companies like Jasper and Copy.AI that are vertical specific “prompt engineering layers” on top of the foundation model players. To a first cut, all these companies will look like a designer, 2 full stack engineers, 1 BE engineer, 1 applied ML engineer and a GTM team. These companies will use APIs from the foundation model players to stitch together end user solutions. Their strength will be user focus, and a relentless focus on GTM. Most of YCombinator’s 2023 vintage will be companies of this form. A large number of these companies will be marketing tech companies, and the market will be flooded with different styles of textual and visual marketing artifacts because the cost of generation of tailored messages is phenomenally lower than it used to be. These companies will look a lot like a dressed-up version of the enterprise SaaS companies from the last decade (PLG, consumer grade UX), with an “AI intelligence layer”.




These will include the original CoPilot for programmers (GitHub/MSFT), “LLM for Chatbots” (Character.ai, Inflection), “LLM for RPA” (Adept), “LLM for search” (Neeva) and many others being incubated as we speak. These will also include OG LLM companies like Cresta (enterprise chatbots) and Lilt (translation). All these companies will make a bet that task-specific LLM tech + deep understanding of user + product is how the market is won. They will need big capital raises to pull off their mission.



 These are classic shovel providers in a gold-rush and will include the labeling companies (Scale.AI, SurgeHQ, Snorkel), training infra companies (MosaicML, Stronger Compute), inference infra companies (Goose.AI), and many others.



Big compute clouds like GCP, AWS, Azure and Oracle


These companies will all wake up to the business opportunity that LLMs provide and compete very aggressively to go up the value chain beyond being just “GPU compute providers”. Azure threatens to have run away with this game already.

Over the next year, we expect to see multiple battles, consolidation, and splintering across these categories. Here are our top predictions of what 2023 looks like:


Compute clouds and foundation model players will collide. The compute clouds will all move towards making “LLM AI capabilities” a feature of their platforms. This will cause the first and last categories of companies to converge. Microsoft already has its OpenAI investment. Google will join the rungs of the foundation model providers by exposing an LLM API via GCP. Amazon and Oracle will buy out one of the independent foundation model companies, or roll out their own models. It’s an open question if there will be any Snowflake and Databricks-style independent companies at the end of 2023.



The CoPilot for X”companies will find themselves competing with AI frontend companies that use foundation model APIs to beat them to market. We’ll see this play out in search, RPA, programming and other categories. The jury will be still out on whether you need to invest in deep tech to succeed in these verticals, or can build a frontend business on top of the pure foundation API players. We think it’s the former.



The tooling companies will find the going a bit hard. The largest of the foundation models will turn out to be great for producing labeled data, often better than humans. The AI labeling companies will find themselves competing with the large models they helped bootstrap. The infra companies will find the open source tooling continues to get better and the GPU compute availability situation becomes easier, eroding their advantage. Both these classes of companies will react by “moving up the value chain” to solve for business use cases and find themselves in competition with the “AI frontend companies” and the “CoPilot for X” companies.



OpenAI will try to be both an API provider and a “CoPilot for X” company for many use cases. This will lead to channel conflict and friction with its partner companies. We will start to see beginnings of rent seeking behavior.




And finally, traditional enterprise companies like SalesForce and ServiceNow and UIPath will find themselves at the end of an onslaught the likes of which they haven’t seen before. They will be threatened by one of the “Copilot for X” or “AI frontend” companies who will promise automation to disrupt their existing business model. They will react by acquiring “CoPilot for X” companies.

With core expertise in applying AI to search, using existing AI technologies, and training our own models, here is our analysis of the industry landscape and our 2023 predictions for what is to come.

2023 AI Players and Predictions

Unbeknownst to many, one of the biggest obstacles to competing in search is a lack of crawl neutrality. The only way to build an independent search engine and the chance to fairly compete against Big Tech is to first efficiently and effectively crawl the Internet. However, the web is an actively hostile environment for upstart search engine crawlers, with most websites only allowing Google’s crawler and discriminating against other search engine crawlers like 

This critically important, yet often overlooked, issue has an enormous impact on preventing upstart search engines like Neeva from providing users with real alternatives, further reducing competition in search. Similar to net neutrality, today we need an approach to crawl neutrality. Without a change in policy and behavior, competitors in search will remain fighting with one hand tied behind our backs.

Let’s start from the beginning. Building a comprehensive index of the web is a prerequisite to competing in search. In other words, the first step to building the 

 is “downloading the Internet” via Neeva’s crawler, called Neevabot.

Here is where the trouble begins. For the most part, websites only allow Google and Bing’s crawlers unfettered access while discriminating against other crawlers like Neeva’s. These sites either disallow everything else in their robots.txt files, or (more commonly) don’t say anything in robots.txt, but return errors instead of content to other crawlers. The intent may be to filter out malicious actors, but the consequence is throwing the baby out with the bathwater. And you can’t serve up search results if you can’t crawl the web.

This forces startups to spend inordinate amounts of time and resources coming up with workarounds. For example, Neeva implements a policy of “crawling a site so long as the robots.txt allows GoogleBot and does not specifically disallow Neevabot.” Even after a workaround like this, portions of the web that contain useful search results remain inaccessible to many search engines.

As a second example, many websites will often allow a non-Google crawler via robots.txt and block it in other ways, either by throwing various kinds of errors (503s, 429s, …) or rate throttling. To crawl these sites, one has to deploy workarounds like “obfuscate by crawling using a bank of proxy IPs that rotate periodically.” Legitimate search engines like Neeva are loath to deploy adversarial workarounds like this.

These roadblocks are often intended at malicious bots, but have the effect of stifling legitimate search competition. At Neeva, we put a lot of effort into building a well behaved crawler that respects rate limits, and crawls at the minimum rate needed to build a great search engine. Meanwhile, Google has carte blanche. It crawls the web 50B pages per day. It visits every page on the web once every three days, and taxes network bandwidth on all websites. This is the monopolist’s tax on the Internet.

For the lucky crawlers among us, a set of well wishers, webmasters and well meaning publishers can help get your bot whitelisted. Thanks to them, Neeva’s crawl now runs at hundreds of millions of pages a day, on track to hit billions of pages a day soon. Even so, this still requires identifying the right individuals in these companies that you can talk to, emailing and cold calling, and hoping for goodwill from webmasters on webmaster aliases that are typically ignored. A temporary fix that is not scalable.

Gaining permission to crawl shouldn’t be about who you know. There should be an equal playing field for anyone competing and following the rules. Google is a monopoly in search. Websites and webmasters are faced with an impossible choice. Either let Google crawl them, or don’t show up prominently in Google results. As a result, Google’s search monopoly causes the Internet at large to reinforce the monopoly by giving Googlebot preferential access.

The internet should not be allowed to discriminate between search engine crawlers based on who they are. Neeva’s crawler is capable of crawling the web at the speed and depth that Google does. There are no technical limitations, just anti-competitive market forces making it harder to fairly compete. And if it’s too much additional work for webmasters to distinguish bad bots that slow down their websites from legitimate search engines, then those with free rein like GoogleBot should be required to share their data with responsible actors.

Regulators and policymakers need to step in if they care for competition in search. The market needs crawl neutrality, similar to net neutrality.

Op-ed first appeared in Fast Company by Vivek Raghunathan Co-founder, Neeva, and Asim Shankar is the Chief Technology Officer, Neeva.

—controls nearly all of the world’s access to information on the internet. Their monopoly in search means for billions of people, their gateway to knowledge, to products, and their exploration of the web is in the hands of one company. Most agree, this lack of competition in search is bad for individuals, communities and democracy.

The web is hostile to upstart search engine crawlers, and most websites only allow Google’s crawler.

[Op-Ed] With Google dominating search, the internet needs crawl neutrality

We are delighted to welcome Steven Shure to the Neeva team. Since Neeva became publicly available in June 2021, over half a million people have visited the site. We’ve been thrilled at the response and the number of people who have signed up for Neeva. Steven’s arrival marks a critical step in the Neeva journey, and a commitment to building the business of Neeva at the same standard at which we’ve been building Neeva search itself since day one.  As the architect of Amazon Prime’s early and extraordinary growth and product evolution, and with three decades of experience overseeing widely-loved, consumer-paid subscription businesses, Steven is uniquely qualified to help the business of Neeva realize its full potential.

In the past months, we have made material investment in building out Neeva’s own web search stack to provide users the best possible search experience, private, and free of ads. Our stack is built on top of tens of billions of web pages we have selected from the hundreds of billions of URLs that our crawlers have discovered. With Steven’s arrival, the coming months will see similar investment and progress in how we take Neeva to market as well.

From the start, at Neeva we have believed in excellence. From the Neeva board to brilliant and broadly respected technologists, like Udi Manber and Darin Fisher, we have entrusted Neeva to 

 who both share a deep passion for Neeva’s mission and themselves have a history of building extraordinary products and businesses. We are thrilled to add Steven, a leader with a similar track record in subscription-based products and businesses to the team. At Neeva, Steven will serve as Chief Business Officer, overseeing all business operations.

Steven joined Amazon in 2006 as the first Vice President of the just-launched Amazon Prime, overseeing all aspects of that business including product, engineering, and marketing. He helped build the service from the ground up, including its launch across Europe and in Japan and the incorporation of key benefits like streaming video. In Steven’s seven years overseeing Prime, its membership grew from fewer than 100,000 to more than 10 million. Steven added Amazon’s Worldwide Consumer Marketing organization to his responsibilities in 2008, overseeing all performance marketing for the company for 11 years, and administering a budget of over $2B annually. In addition to Prime and Marketing, Steven held senior leadership positions in Supply Chain Technology and Amazon Web Services, managing teams from as small as ten to greater than ten thousand. Prior to Amazon, he spent two decades at Time Inc. overseeing direct-to-consumer marketing and subscription services for magazine brands such as People, Sports Illustrated and Entertainment Weekly.

Steven’s ex-colleague, Christopher North, Managing Partner, L Catterton, and former Managing Director, Amazon UK says “During the decade we worked closely together at Amazon, Steve built an outstanding team, collectively driving large-scale innovation across global marketing and Amazon Prime that helped power growth from $11B to $136B in 2006-2016 alone. His thought leadership, domain expertise, and humble style made him one of the great all-time Amazonians.”

Says Steven, “I am beyond excited to join the Neeva team. Since my start in the magazine industry, I’ve always had a deep belief in the power of customer-supported, subscription-based products, and have seen firsthand how those products (and customers) are inevitably compromised when they become dependent on advertising revenue. Above all, it is my belief in Neeva’s mission, and the extraordinary team already in place, that led me to want to join. I remember people asking me years ago why I thought anyone would ever want to subscribe to a shipping service. I hope and expect that in the not too distant future, the question ‘why would anyone want to pay for a search engine?’ will, in hindsight, sound just as ridiculous.”

We’re continuing on our journey of building the world’s first ad-free, private subscription search engine that puts the customer first, and we’re thrilled to have Steven help make that a reality. We’re excited about what’s ahead and if you share a similar passion come 

A longtime Amazon VP, Steven played a pivotal role in building and running Amazon Prime and oversaw Amazon’s Worldwide Consumer Marketing organization for over a decade.


We are delighted to welcome Steven Shure to the Neeva team. Steven played a pivotal role in building and running Amazon Prime and oversaw Amazon’s Worldwide Consumer Marketing organization for over a decade. At Neeva, Steven will serve as Chief Business Officer, overseeing all business operations.

Vivek Raghunathan

2023 AI Players and Predictions

[Op-Ed] With Google dominating search, the internet needs crawl neutrality

Neeva Continues Growth, Welcomes Steven Shure as Chief Business Officer