See my earlier article on demand estimation for background.

There are two broad approaches to estimating aggregate-level demand: product-space approaches and characteristics-space approaches. Product-space approaches, such as Price Invariant Generalised Logarithmic models and Almost Ideal Demand Systems, treat individual products as the unit of analysis and endeavour to estimate demand functions using restrictions from economic theory. However, these approaches need to estimate N^2 elasticities (considering both own-price and cross-price elasticities), where N is the number of products, hence, even for a modest market with (e.g.) 20 products, over 400 parameters need to be estimated. This is computationally difficult, leading to the curse of dimensionality, and poses a key downside to product-space approaches, along with the detraction that such methods do not allow for the counterfactual estimation of new products being introduced.

An alternative approach treats products as a bundle of characteristics and then estimates demand functions based on these features of products. This improves on product-space approaches as it (a) reduces the number of parameters to be estimated, in most cases, and (b) allows for the estimation of the effect of product introduction, where new products are just different combinations of characteristics which have already been modelled.

The general approach taken to estimate demand in characteristics-space relies on the discrete choice framework. Under such a framework, we posit that the individual (i) utility (u), from a given product (j) in market t, is a function of the product’s price (p), observed product characteristics (x), unobserved (to the econometrician) product characteristics (ζ), preference heterogeneity (μ, which may itself be a function of observed demographic characteristics as well as unobserved preferences), and an error term (ε) assumed to be distributed iid with type 1 extreme value:

u_{i,j,t} = \alpha p_{j,t}+x^{'}_{j,t}\beta +\zeta_{j,t}+\mu_{i,j,t}+\epsilon_{i,j,t}

We can use the Berry-Levinsohn-Pakes (BLP) estimation technique, and BLP contraction, to estimate this equation, as there is no closed-form analytical expression which solves this non-linear equation. Alternatively, we can ignore preference heterogeneity (set μ=0; this causes unrealistic estimates for price-elasticities but allows us to better explain the underlying endogeneity present in the system) and use our assumption about the distribution of the error term to express the market share (s) of a product as the following function:

s_{j,t} = \frac{exp( \alpha p_{j,t}+x^{'}_{j,t}\beta +\zeta_{j,t})}{1+\sum{exp( \alpha p_{k,t}+x^{'}_{k,t}\beta +\zeta_{k,t})}}

Next, we can use Berry’s trick to express the log of the ratio of market share of each product compared with the market share of the outside option, which is a linear equation which can be estimated using OLS:

log(\frac{s_{j,t}}{s_{0,t}}) = \alpha p_{j,t} + x^{'}_{j,t} \beta + \zeta_{j,t}

However, estimation using OLS overlooks the inherent endogeneity in this linear equation. Price is likely correlated with unobserved quality: firms which produce higher-quality or higher-branded products ought to charge a higher price for their brand. This will lead to the estimate on alpha being biased towards zero.

To mitigate this endogeneity concern and estimate unbiased coefficients on our estimators, we can include product and market fixed effects to eliminate some bias. Nonetheless this is unlikely to completely eliminate the bias, as zeta will typically vary by product and market. Instead, we can turn to the instrumental variables approach to mitigate endogeneity concerns. The key to an IV strategy is to find a variable (instrument) which is correlated with the endogenous variable of price (relevance criterion), but which is not correlated itself with the unobserved quality term (exclusion criterion). Note that the relevance and exclusion criteria need to hold across markets (t) and products (j), meaning our instrument needs to have sufficient variation across both markets and products. Furthermore, note that we can test, in linear applications, for the relevance criterion by estimating an equation with prices as the dependent variable and the instrument(s) as the independent variable(s) checking that the regression F-stat is sufficiently high (Stock and Yogo, 2005). In the non-linear case of demand estimation, testing for relevance is more tricky. One potential approach is to use Salanie and Wolak’s (2019) linear-in-parameters approximation to the BLP inverse share function and use this to test the F-statistic in the first stage regression.

The demand estimation literature has primarily focused on 4 types of instruments to deal with this endogeneity problem. We will discuss each instrument in turn and highlight their strengths and weaknesses.

The first instrument we will discuss is the supply-side cost shifter. Fundamentally, the price of a product depends on the equilibrium of demand and supply. On the supply side, the input costs of a product affect the price the firm is willing to supply the product at. This might include labour costs, raw materials, capital costs, taxes, tariffs and a built-in mark-up. Supply shocks, such as poor weather, natural disasters, pandemics and exogenous increases in taxes or tariffs can cause changes to these input prices and are known as cost shifters. The effect of the cost shifter is to increase the product price as, under certain assumptions about pass-through, firms will pass on higher costs directly to consumers. Consequently, we can expect the relevance condition to be met: cost shifters and price are expected to be strongly correlated. At the same time, if the cost-shifter is truly exogenous (so long as these cost shifters are independent of unobserved demand characteristics) then we satisfy the exogeneity condition required for the instrument.

The cost shifter can thus be a useful instrument and is likely to meet the exogeneity criterion. However, we require the instrument to have sufficient variation across markets and products for it to meet the relevance criterion, which might not be the case for cost-shifters, which are likely to be the same value across markets and for a large number of (substitute) products, particularly within brand. For instance, if a raw material is used in the same ratio across products and in all markets, then there will be no variation in the instrument across products and markets. We would therefore need to find a cost shifter which is present in all products for it to provide useful variation. This might rule out some forms of cost shifters, e.g. input costs which are not used in all products, or tariffs, if they do not apply to all markets (at different rates).

Furthermore, it can be difficult to obtain (proprietary) data on underlying marginal costs for products, or the ratio of each input as a proportion of total costs, with the exception of tax and tariff costs. Finally, the rate of cost-shock pass-through depends on market structure: in monopoly markets, pass-through is typically lower than in more competitive markets (Genakos and Pagliero, 2022)[1], which would result in low relevance for cost-shocks as an instrument.

A second type of instrument is the contemporaneous price of the same good in neighbouring markets (Hausman instrument). In practice, this method has been used by Nevo (2001) to study breakfast cereal. This follows a similar logic to the cost shifter instrument discussed above, assuming that even if we cannot observe cost shifters, variation in production costs for the same product is likely to be present and explain variation in prices that the same producer sets across markets. In other words, a price increase in a neighbouring market can signal an increase in the producer’s costs which also increase the price in focal market. We therefore require that cost shocks to be (somewhat) correlated across markets for the relevance criteria to hold but that demand shocks are not correlated across market, to ensure exogeneity holds. However, with imperfect competition, prices reflect not only costs but also demand elasticities and it could be the case that demand elasticities are correlated across markets and therefore we are picking up this endogenous demand shock rather than the exogenous cost shock we are hoping for.

This often means that Hausman instruments need to be carefully justified to ensure the exogeneity condition holds in reality. One example of a violation of exogeneity could be if there is an unobserved advertising campaign which occurs across markets.

The third type of instrument we will discuss is known as the BLP instrument. This captures the observed product characteristics of rivals’ products. These characteristics affect the market share for a given product, due to competition, and they also affect prices through consumer’s willingness to pay for certain characteristics. In other words, rivals’ product characteristics determine the degree of competition between the focal product and those of the rivals and thus the mark-up which can be applied to the focal product (hence, linking the instrument with the price of the focal product).

Again, the key assumption is that rivals’ product characteristics are uncorrelated with unobserved demand characteristics. For instance, if firms decide the location of their products in characteristic space before they observe the consumer’s valuation of the unobserved product characteristics, then this assumption will hold. However, if firms are able to observe demand shocks and easily alter product characteristics to adjust to such shocks, then the assumption will fall apart and exogeneity no longer remains, making this a poor instrument. A further problem is that these instruments become weak if the number of products in the market is relatively large (Armstrong, 2016). To avoid these weak instruments, Gandhi and Houde (2020) adjust these BLP instruments to generate “differentiation instruments”. These instruments use the distance between the focal product and rival product characteristics to generate the instrument, rather than just the rival product characteristics.

The final form of instrument we discuss to deal with endogeneity between price and unobserved demand characteristics is the Waldfogel instrument. These instruments use demographic characteristics of nearby markets to act as exogenous shifters of equilibrium markups. Fundamentally, the strength of these instruments depend on the assumption that prices are uniformly set at a high-level geographic area (i.e. at a higher level than a single market). It then follows that “one’s neighbours influence the types of products and prices one is offered” (Berry and Haile, 2021). For example, with uniform pricing, a market with a given distribution of income will be more likely to have high prices if it is surrounded by high income markets than low income markets (Berry and Haile, 2021). It must be the case that we can control for the demographic characteristic which is the basis of the uniform pricing to ensure that it is not correlated with unobserved demand characteristics.

Clearly, the strength of these instruments comes from the assumption of uniform pricing across certain (but not all) markets. Therefore, this instrument may not be useful to study demand for products only available at the national level where the products are uniformly priced (e.g. supermarket-branded products where the majority of UK supermarkets employ uniform pricing[2]). The evidence on whether national chains charge geographically uniform prices in the US is mixed (e.g. Butters et al. 2022). Moreover, the firm-level choice to price products uniformly over a certain geographic area is in itself endogenous, which could cause issues for the exogeneity condition when using this instrument.

In conclusion, there are a number of instruments that can be employed to overcome underlying exogeneity between price and unobserved demand characteristics. The relative strengths of each particular instrument depend on their ability to be both relevant (highly correlated with price) and exogeneous (uncorrelated with unobserved demand characteristics). Their ability to meet these criteria will vary in different circumstances, depending on the particular market and the exact variable under consideration.

[1] However, this conclusion is sensitive to the assumption that firms’ marginal cost curves are constant. With convex costs, market power can raise pass-through (Ritz, 2024).

[2] https://www.bbc.co.uk/news/business-43979167

References

Ackerberg, D., Benkard, C.L., Berry, S. and Pakes, A., 2007. Econometric tools for analyzing market outcomes. Handbook of econometrics, 6, pp.4171-4276.

Armstrong, T.B., 2016. Large market asymptotics for differentiated product demand estimators with economic models of supply. Econometrica, 84(5), pp.1961-1980.

Berry, S.T. and Haile, P.A., 2021. Foundations of demand estimation. In Handbook of industrial organization (Vol. 4, No. 1, pp. 1-62). Elsevier.

Gandhi, A. and Houde, J.F., 2020. Measuring Firm Conduct in Differentiated Products Industries. Unpublished. Available at: https://jfhoude.econ.wisc.edu/wp-content/uploads/sites/769/2020/12/GH_conduct_v1. pdf.

Gandhi, A. and Nevo, A., 2021. Empirical models of demand and supply in differentiated products industries. In Handbook of industrial organization (Vol. 4, No. 1, pp. 63-139). Elsevier.

Genakos, C. and Pagliero, M., 2022. Competition and pass-through: evidence from isolated markets. American Economic Journal: Applied Economics, 14(4), pp.35-57.

Nevo, A., 2000. A practitioner’s guide to estimation of random‐coefficients logit models of demand. Journal of economics & management strategy, 9(4), pp.513-548.

Nevo, A., 2001. Measuring market power in the ready‐to‐eat cereal industry. Econometrica, 69(2), pp.307-342.

Ritz, R.A., 2024. Does competition increase pass‐through?. The RAND Journal of Economics.

Salanié, B. and Wolak, F.A., 2019. Fast,” robust”, and approximately correct: estimating mixed demand systems (No. w25726). National Bureau of Economic Research.

Stock, J.H. and Yogo, M., 2002. Testing for weak instruments in linear IV regression.