Journal articles and conference proceedings

The Office of the Chief Economist (OCE) publishes research findings from its team of economists and their collaborators in peer-reviewed economic journals and in law journals.

The artificial intelligence patent dataset (AIPD) 2023 update

Nicholas A. Pairolero, Alexander V. Giczy, Gerard Torres, Tisa Islam Erana, Mark A. Finlayson, and Andrew A. Toole
Read the published article
Read an unpublished version

Read the abstract.

The 2023 update to the Artificial Intelligence Patent Dataset (AIPD) extends the original AIPD to all United States Patent and Trademark Office (USPTO) patent documents (i.e., patents and pre-grant publications, or PGPubs) published through 2023, while incorporating an improved patent landscaping methodology to identify artificial intelligence (AI) within patents and PGPubs. This new approach substitutes BERT for Patents for the Word2Vec embeddings used previously, and uses active learning to incorporate additional training data closer to the “decision boundary” between AI and not-AI to help improve predictions. We show that this new approach achieves substantially better performance than the original methodology on a set of patent documents where the two methods disagreed—on this set, the AIPD 2023 achieved precision of 68.18% and recall of 78.95%, while the original AIPD achieved 50% and 21.05%, respectively. To help researchers, practitioners, and policymakers better understand the determinants and impacts of AI invention, we have made the AIPD 2023 publicly available on the USPTO’s economic research web page.

Competitive dynamism in nascent markets: trademarks and superstar firm entry in the metaverse

Wendy A. Bradley and Julian Kolev
Industry and Innovation, 1-41, November 2024
Read the published article
Read the working paper

Read the abstract.

Does entry by a superstar firm change the competitive dynamics of nascent industries? Using USPTO trademark data and a quasi-experimental design, we analyse the introduction of new products in the metaverse. We develop a theoretical framework of heterogeneous strategies for market entry under uncertainty, contrasting digital vs. physical industries. Empirically, we show that firm size has a strong positive correlation with metaverse participation, and this effect is amplified for international, physical-goods firms and reversed for born-digital firms. Following Facebook’s entry in 2014, we document a 10–12% decline in metaverse product introductions, particularly among smaller and international physical-goods firms. This decline coincides with an increase in the international scope of metaverse trademarks, suggesting that superstar firm entry into the metaverse reduced the quantity but increased the geographic scope of product introductions. Our analysis highlights intangible assets as key drivers of firms’ competitive advantage in the global, digital economy, with implications for antitrust regulation.

Procrastination or incomplete data? An analysis of USPTO examiner search activity

Charles A.W. de Grazia, Alexander V. Giczy, and Nicholas A. Pairolero
Research Policy, Volume 53, Issue 7, September 2024
Read the published article

Read the abstract.

Frakes and Wasserman (2020) finds evidence that United States Patent and Trademark Office (USPTO) patent examiners disproportionately receive credit for examination activities at the end of production periods and interprets this behavior as systemic examiner procrastination. Using newly assembled micro data on the timing of USPTO examiner activities, our results show examiner work effort more closely resembles a uniform distribution over the production period, indicating consistent workflow and not procrastination. The assembled data better track the precise timing of examiner work activities than the data used in Frakes and Wasserman (2020), explaining the differences in our results. While Frakes and Wasserman (2020) conduct a thorough analysis of the markers and consequences of “end-loading”, its use of an inadequate proxy for examination activity (i.e., Office action count dates) leads to incorrect conclusions about the examination process (i.e., widespread examiner procrastination).

Patent eligibility after Alice: Evidence from USPTO patent examination

Jesse Frumkin, Nicholas A. Pairolero, Asrat Tesfayesus, and Andrew A. Toole
Journal of Economics & Management Strategy, 1–22.
Read the published article

Read the abstract

In a series of decisions over the last decade, the Supreme Court of the United States altered the classes of inventions that are eligible for patent protection—a body of law called subject matter eligibility. One of the more contentious of these decisions, Alice Corp. versus CLS Bank International (Alice), questioned the patentability of a broad class of inventions involving abstract ideas, particularly in digital technologies. Exploiting a quasinatural experiment, we find that the Alice decision reduced favorable patent eligibility decisions by 31% and significantly and persistently increased legal uncertainty in patent examination by 26% for a broad set of technologies. Our analysis quantifies how legal decisions can limit patent protection and highlights the need for further research on how greater legal uncertainty affects upstream investments supporting invention and downstream innovations fueling growth.

Alexander V. Giczy, Nicholas A. Pairolero, and Andrew A. Toole
Nature Biotechnology, Volume 42, January 2024, pg. 26-29
Read the published article
Listen to AUTM on the Air podcast ("Behind the Numbers: Insights from U.S. Patenting on Diagnosing COVID-19")

More can be done to diversify the AI innovation ecosystem, especially in the emerging field of biotechnology AI, where women’s participation is associated with substantial value.

Read the abstract

Although recent research has begun to identify the determinants and impact of women’s patenting in general, very little is known about the involvement of women in the AI innovation ecosystem. This is because until now, reliable identifiers of both the gender of inventors and which patents contain AI were unavailable to researchers. The dataset used in our research overcomes both of these limitations. We show that women’s participation in patenting—both in AI and other technologies—is growing and associated with more diverse teams and patents with higher economic value. We also find that more can be done to diversify the AI innovation ecosystem, especially in the university sector and within the emerging field of biotechnology AI, an area where women’s participation is associated with substantial value.

Wendy A. Bradley and Julian Kolev
Research Policy, Volume 52, Issue 3, April 2023
Read the published article
Read the working paper

Read the abstract

Despite digital piracy's well-documented impact on firm revenue, the relationship between piracy and firm innovation, including the creation of new intellectual property (IP) rights, is not well-understood. To fill this gap, this paper estimates the impact of piracy on innovation through a quasi-experimental design and explores the mechanisms driving this relationship using data on software firms. Leveraging a 2001 technological shock that suddenly enabled rising software piracy, we find increases in subsequent R&D spending, copyrights, trademarks, and patents for large, incumbent software firms. Furthermore, firms with large patent portfolios appear to disproportionately increase copyrights and trademarks following the piracy shock. After considering alternatives, our analysis suggests that impacted firms perceive piracy as a form of product-market competition that causes them to increase innovation and balance their IP portfolios.

Closing the Gender Gap in Patenting: Evidence from a Randomized Control Trial at the USPTO

Nicholas A. Pairolero, Andrew A. Toole, Peter-Anthony Pappas, Charles A.W. deGrazia, Mike H.M. Teodorescu
Academy of Management Proceedings , vol 2022, no 1, July 2022
Read the article

Read the abstract.

Using a randomized control trial at the United States Patent and Trademark Office, we identify the heterogeneous causal impacts across gender and technologies of increased patent examination assistance on the probability of obtaining patent rights. Women applicants were 11 percentage points more likely than men to benefit from this assistance, and the benefits were largest for new inventors and in areas of technology where women had the worst relative outcomes. Our results suggest that a portion of the gender patenting gap could be eliminated through the provision of additional resources and assistance during patent prosecution.

Challenges facing independent inventors: Do patents help?

Charles A.W. deGrazia, Nicholas A. Pairolero, Peter-Anthony Pappas, Andrew A. Toole, Mike H.M. Teodorescu
Academy of Management Proceedings, vol 2022, no 1, July 2022
Read the article

Read the abstract.

Independent inventors, those inventors who are not affiliated with a formal organization or institution when filing for a patent, have made outstanding contributions to economic growth and prosperity in the United States. While a key source of invention, independent inventors face several challenges related to successful commercialization. This paper extends the literature on the potential benefits of patent protection to independent inventors. We find that patents have significant positive effects on the use of legal representation on subsequent patent applications, participation in markets for technology through patent sales, and in certain cases, becoming connected to first-time and established firms. These results demonstrate that patenting lowers the barriers to successful innovation and commercialization for independent inventors.

Identifying artificial intelligence (AI) invention: A novel AI patent dataset

Alexander V. Giczy, Nicholas A. Pairolero, and Andrew Toole
The Journal of Technology Transfer, 2021
Read the published article
Read the working paper

Abstract

Artificial intelligence (AI) is an area of increasing scholarly and policy interest. To help researchers, policymakers, and the public, this paper describes a novel dataset identifying AI in over 13.2 million patents and pre-grant publications (PGPubs). The dataset, called the Artificial Intelligence Patent Dataset (AIPD), was constructed using machine learning models for each of eight AI component technologies covering areas such as natural language processing, AI hardware, and machine learning. The AIPD contains two data files, one identifying the patents and PGPubs predicted to contain AI and a second file containing the patent documents used to train the machine learning classification models. We also present several evaluation metrics based on manual review by patent examiners with focused expertise in AI, and show that our machine learning approach achieves state-of-the-art performance across existing alternatives in the literature. We believe releasing this dataset will strengthen policy formulation, encourage additional empirical work, and provide researchers with a common base for building empirical knowledge on the determinants and impacts of AI invention.”

Examination incentives, learning, and patent office outcomes: The use of examiner’s amendments at the USPTO

Charles A.W. deGrazia, Nicholas A. Pairolero, and Mike H.M. Teodorescu
Research Policy, vol 50, issue 10, December 2021
Read the published article
Read the working paper

Abstract

We investigate how U.S. Patent and Trademark Office (USPTO) patent examiner experience and seniority-based incentives affect the innovation ecosystem. First, we show that examiners respond to production incentives and demonstrate learning by increasing the use of examiner’s amendments in both experience and seniority, a mechanism not previously studied. Second, this examination procedure directly benefits innovators and firms by significantly reducing prosecution processing time without impacting patent quality. Finally, after considering examiner’s amendments, the negative relationship between examiner characteristics and patent examination quality found in the previous literature does not persist at first action, a decision point that allows for the clear measurement of examiner behavior. Our results demonstrate a need for reformulated policy recommendations related to the structure of examination at the USPTO.

The Promise of Machine Learning for Patent Landscaping

Andrew A. Toole, Nicholas A. Pairolero, James Q. Forman, and Alexander V. Giczy
Santa Clara High Technology Law Journal, vol 36, no 4, May 2020
Read the published article
Read the working paper

Read the abstract.

Patent landscaping involves the identification of patents in a specific technology area to understand the business, economic, and policy implications of technological change. Traditionally, patent landscapes were constructed using keyword and classification queries, a labor-intensive process that produced results limited to the scope of the query. In this paper, we discuss the advantages and disadvantages of using machine learning to produce patent landscapes. Machine learning leverages traditional queries to construct the data necessary to train machine learning models, and the models allow the resultant landscapes to extend more broadly into areas of technology not expected a priori. The models, however, are “black boxes” that limit transparency regarding their underlying reasoning. To illustrate these points, we summarize two landscapes we recently conducted, one in mineral mining and another in artificial intelligence.

Innovation activities and business cycles: are trademarks a leading indicator?

Charles A. W. deGrazia, Amanda Myers, and Andrew A. Toole
Industry and Innovation, vol 27, issue 1-2, 2020
Read the published article
Read the working paper

Read the abstract.

Despite the widespread use of economic information to anticipate changes in business conditions, innovation metrics are not considered to be leading indicators. We argue that aggregate trademark data reflect firm-level choices that can help predict business cycles. In addition to establishing the conceptual basis for considering trademarks, our statistical evaluations, using turning point analysis and a novel machine learning method, find that trademark filings for product and service offerings in commercial use outperform many of the conventional leading indicators. Our work suggests that including trademark metrics in composite indexes could improve recession forecasting performance.

Previous journal articles

2019

The Dynamic Relationship Between Investments in Brand Equity and Firm Profitability: Evidence Using Trademark Registrations

Dirk Crass, Dirk Czarnitzki, and Andrew A. Toole
International Journal of the Economics of Business, vol 26, no 1, March 2019
2019 Best Paper Prize from the International Journal of the Economics of Business
Read the article

Read the abstract.

Most marketing practitioners and scholars agree that marketing assets such as brand equity significantly contribute to a firm’s financial performance. In this paper, we model brand equity as an unobservable stock that results from up to 30 years of past brand-related investment flows. Using firm-specific trademarks as investment proxies, our results show a significant long-run impact on financial performance. The dynamic profile of brand-related investments has an inverted-U shape that reaches its peak after 11 years. On average, it takes four years before brand-related investments show a positive return, and investments older than 19 years show no significant impact. For the median trademarking firm, brand equity contributes €265,000 to annual profits.

Shorter Patent Pendency Without Sacrificing Quality: The Use of Examiner’s Amendments at the USPTO

Charles deGrazia, Nicholas Pairolero, and Mike Teodorescu
Academy of Management Proceedings, vol 2019, no 1, August 2019
Read the article

Read the abstract.

Prior research argues that USPTO first-action allowance rates increase with examiner seniority and experience, suggesting lower patent quality. However, we show that the increased use of examiner's amendments account for this prior empirical finding. Further, the mechanism reduces patent pendency by up to fifty percent while having no impact on patent quality, and therefore likely benefits innovators and firms. Our analysis suggests that the policy prescriptions in the literature regarding modifying examiner time allocations should be reconsidered. In particular, rather than re-configuring time allocations for every examination promotion level, researchers and stakeholders should focus on the variation in outcomes between junior and senior examiners and on increasing training for examiner's amendment use as a solution for patent grant delay.

Patent Examination Quality and Litigation: Is There a Link?

Alan C. Marco and Richard D. Miller
International Journal of the Economics of Business, vol 26, no 1, February 2019
Read the article

Read the abstract.

In this study, we use carefully constructed matched samples of litigated and non-litigated patents to investigate the characteristics that predict litigation. We define different control groups based on filing characteristics and value correlates (or both), and test the extent to which examination characteristics predict litigation. This paper is the first to use detailed examination characteristics to understand the resulting patent rights. By controlling (at least in part) for patent value, the estimation strategy has implications for the degree to which patent examination characteristics are correlated with uncertainty. We find that some examination characteristics predict litigation, but that the bulk of the predictive power in the model comes from filing characteristics.

Embracing Invention Similarity for the Measurement of Vertically Overlapping Claims

Charles A. W. deGrazia, Jesse P. Frumkin and Nicholas A. Pairolero
Economics of Innovation and New Technology, 2019
Read the article

Read the abstract.

Clear and well-defined patent rights can incentivize innovation by granting monopoly rights to the inventor for a limited period of time in exchange for public disclosure of the invention. However, with cumulative innovation, when a product draws from intellectual property held across multiple firms (including fragmented intellectual property or patent thickets), contracting failures may lead to suboptimal economic outcomes. However, an alternative theory, developed by a variety of scholars, contends that patent thickets have a more ambiguous effect. Researchers have developed several measures to gauge the extent and impact of cumulative innovation and the various channels of patent thickets. This paper contends that mis-measurement may contribute to the incoherence and overall lack of consensus within the patent thickets literature. Specifically, the literature is missing a precise measure of vertically overlapping claims. We propose a new measure of vertically overlapping claims that incorporates invention similarity to more precisely identify inventive overlap. The measure defined in this paper will enable more accurate measurement, and allow for novel economic research on cumulative innovation, fragmentation in intellectual property, and patent thickets within and across all patent jurisdictions.

2011-2018

Patent transactions in the marketplace: Lessons from the USPTO Patent Assignment Dataset
Journal of Economics & Management Strategy
Special Issue: Innovation Economics III: Patents, Trademarks, and Technology Standards Dataset
Volume 27, Issue 3, Fall 2018, Pages 343-371
Stuart J. H. Graham, Alan C. Marco, and Amanda F. Myers

Research-ready dataset of patent assignments and other transactions recorded with the USPTO creates new opportunities for original research, particularly relating to patent collateralization and the markets for technology and innovation.
Read the article

Business dynamics of innovating firms: Linking U.S. patents with administrative data on workers and firms
Journal of Economics & Management Strategy
Special Issue: Innovation Economics III: Patents, Trademarks, and Technology Standards Dataset
Volume 27, Issue 3, Fall 2018, Pages 372-402
Stuart J. H. Graham, Cheryl Grim, Tariqul Islam, Alan C. Marco, and Javier Miranda

Patenting firms, particularly young patenting firms, disproportionally contribute jobs to the U.S. economy, based on analysis of a new longitudinal database on inventors and patent‐owning firms.
Read the article

Monetizing marks: Insights from the USPTO Trademark Assignment Dataset
Journal of Economics & Management Strategy
Special Issue: Innovation Economics III: Patents, Trademarks, and Technology Standards Dataset
Volume 27, Issue 3, Fall 2018, Pages 403-432
Stuart J. H. Graham, Alan C. Marco, and Amanda F. Myers

Research-ready dataset of trademark assignments and other transactions recorded with the USPTO opens new avenues for research on the market for brands and trademark collateralization.
Read the article

The USPTO Patent Examination Research Dataset: A window on patent processing
Journal of Economics & Management Strategy
Special Issue: Innovation Economics III: Patents, Trademarks, and Technology Standards Dataset
Volume 27, Issue 3, Fall 2018, Pages 554-578
Stuart J.H. Graham, Alan C. Marco, and Richard Miller

Research-ready dataset of patent application prosecution records enables original and in-depth empirical research on the process of obtaining a patent from the USPTO.
Read the article

The Law and Economics of Concealing Health and Safety Information
Daniel E. Ingberman and Asrat Tesfayesus
Journal of Law, Economics, & Policy, Volume 13, Number 2, Spring 2017, Pages 181
Read the Article

Liberalization Agreement in the GATT/WTO and the Terms-of-trade Externality Theory: Evidence from Three Developing Countries
Asrat Tesfayesus
Review of International Economics, Volume 24, Issue 5, November 2016, Pages 1000-1022
Read the Article

Knowledge Creates Markets: The influence of entrepreneurial support and patent rights on academic entrepreneurship
Dirk Czarnitzki, Thorsten Doherr, Katrin Hussinger, Paula Schliessler, and Andrew A. Toole
European Economic Review, Volume 86, July 2016, Pages 131–146
Read the Article

More than BRIC-a-Brac: Testing Chinese Exceptionalism in Patenting Behavior Using Comparative Empirical Analysis
Jay Kesan, Alan Marco, and Richard Miller
Michigan Telecommunications and Technology Law Review, Volume 22, No. 53, 2015
Read the Article

What is the Probability of Receiving a US Patent?
Michael Carley, Deepak Hegde, and Alan Marco
The Yale Journal of Law & Technology, Volume 17, 2015, Pages 203-223
Read the Article

Using the Classroom to Bring Big Data to Statistical Agencies
Ron Jarmin, Julia Lane, and Alan Marco, with Ian Foster
AMSTATNEWS, The Membership Magazine of the American Statistical Association, 1 November 2014
Read the Article

The USPTO economics research agenda
Stuart Graham and Galen Hancock
The Journal of Technology Transfer, June 2014, Volume 39, Issue 3, pp 335-344
Read the Article

The USPTO Trademark Case Files Dataset: Descriptions, Lessons, and Insights
Stuart Graham, Galen Hancock, Alan Marco, and Amanda Fila Myers
Journal of Economics and Management Strategy, Volume 22, Issue 4, Winter 2013, Pages 669-705.
Read the Article

Of Smart Phone Wars and Software Patents
Stuart Graham and Saurabh Vishnubhakat
Journal of Economic Perspectives, Volume 27, Number 1, Winter 2012, Pages 67-86.
Read the Article

The Case for Standard Measures of Patent Quality
David J. Kappos and Stuart Graham
MIT Sloan Management Review, Volume 53, Issue 3, Spring 2012.
Read the Article

Journal articles and conference proceedings

The artificial intelligence patent dataset (AIPD) 2023 update

Competitive dynamism in nascent markets: trademarks and superstar firm entry in the metaverse

Procrastination or incomplete data? An analysis of USPTO examiner search activity

Patent eligibility after Alice: Evidence from USPTO patent examination

Discovering value: women’s participation in university and commercial AI invention

How does digital piracy affect innovation? Evidence from software firms

Closing the Gender Gap in Patenting: Evidence from a Randomized Control Trial at the USPTO

Challenges facing independent inventors: Do patents help?

Identifying artificial intelligence (AI) invention: A novel AI patent dataset

Examination incentives, learning, and patent office outcomes: The use of examiner’s amendments at the USPTO

The Promise of Machine Learning for Patent Landscaping

Innovation activities and business cycles: are trademarks a leading indicator?

Previous journal articles

The Dynamic Relationship Between Investments in Brand Equity and Firm Profitability: Evidence Using Trademark Registrations

Shorter Patent Pendency Without Sacrificing Quality: The Use of Examiner’s Amendments at the USPTO

Patent Examination Quality and Litigation: Is There a Link?

Embracing Invention Similarity for the Measurement of Vertically Overlapping Claims