Time-series econometrics:

* Time Series Analysis, by Hamilton: classic text on time series econometrics
* Econometric Analysis, by Greene: classic text on theoretical econometrics

Financial time series and modeling:

* Analysis of Financial Time Series, by Tsay: standard applied time series text for financial econometrics
* Market Models: A Guide to Financial Data Analysis, by Alexander: excellent introduction to financial modeling and forecast
* Asset Price Dynamics, Volatility, and Prediction, by Taylor: classic text on financial modeling and forecast

Filtering and wavelets:

* Wavelet Methods for Time Series Analysis, by Percival and Walden: standard theoretical text on wavelets
* An Introduction to Wavelets and Other Filtering Methods in Finance and Economics, by Gençay, Selçuk, and Whitcher: applied filtering and wavelets for finance and economics

Volatility, correlation, and dispersion:

* Volatility and Correlation, by Rebonato: excellent coverage of volatility and correlation
* Dynamic Hedging, by Taleb: dated practitioner on the real-life realities of hedging
* Volatility Trading, by Sinclair: volatility arbitrage by a retail practitioner
* Volatility Surface, by Gatheral: theoretical coverage of vol models, by well-known researcher
* Options as a Strategic Investment, by McMillan: classic introductory text on derivative hedging and volatility trading
* Option Volatility & Pricing, by Natenberg: dated practitioner introduction to volatility trading

Portfolio theory and financial engineering:

* Modern Portfolio Theory and Investment Analysis, by Elton et al.: standard text on modern portfolio theory
* Options, Futures and Other Derivatives, by Hull: standard reference for introductory financial engineering
* Active Portfolio Management, by Grinold & Kahn: standard introduction to quantitative portfolio management by the BGI guys who invented it
* Principles of Financial Engineering, by Neftci: intermediate financial engineering

Machine and statistical learning:

* Artificial Intelligence: A Modern Approach, by Russell and Norvig: standard introductory AI text
* The Elements of Statistical Learning, by Hastie, Tibshirani, and Friedman: standard intermediate statistical learning text
* Pattern Recognition and Machine Learning, by Bishop: intermediate classification and learning text
* Pattern Classification, by Duda: standard introductory classification text
* Genetic Algorithms in Search, Optimization, and Machine Learning, by Goldberg: standard GA introductory text

High frequency & market microstructure:

* Trading and Exchanges: Market Microstructure for Practitioners, by Harris: practitioner introduction to stylized financial microstructure effects
* An Introduction to High-Frequency Finance, by Dacorogna et al.: theoretical and dated practitioner introduction to HF, with emphasis on FX
* Empirical Market Microstructure, by Hasbrouck: intermediate equity market microstructure, with coverage of standard theoreitcal models
* Microstructure Approach to Exchange Rates, by Lyons: intermediate FX market microstructure
* Market Microstructure Theory, by O’Hara: classic introduction to microstructure theory; now dated
* Optimal Trading Strategies, by Kissell and Glantz: practitioner introduction to market impact and optimal execution

Extending beyond retail, the seminal algorithmic trading literature revolves around market impact and optimal execution (along the lines of Kissell and Glantz, above). This tradition owes much to Almgren, who is currently Adjunct at Courant, and several collaborators. Representative papers include:

* Optimal execution of portfolio transactions, by Almgren and Chriss
* Optimal control of liquidation costs, by Bertsimas and Lo
* Optimal Trading Strategy and Supply/Demand Dynamics, by Obizhaeva and Wang
* Bayesian Adaptive Trading with a Daily Cycle, by Almgren and Lorenz
* Understanding the Profit and Loss Distribution of Trading Algorithms, by Kissell

See Chris Donnan’s recent post on seminal algorithmic trading papers for further details on this research tradition.