Smart Beta (or whatever you want to call it) is the hottest new trend on Wall Street. On the whole I think it is a great idea as there is no rule that market cap weighted indices are the best way to go. However, whenever you see a flood of new products most are probably going to be more hype than anything else, so how do you evaluate this stuff?
Past performance doesn't predict future results, this applies to Smart Beta also. No new strategy is going to be launched without a backtest showing that the strategy beats its relevant index. There are some things you would want to pay attention to on any backtested returns:
1. Are the returns too good? There are two ways to backtest, you can take a premise that should work going forward and see how it would have worked in the past---the right way. Or, you can use your knowledge about what happened in the past to construct an awesome backtest---the wrong way. If the returns are too good there is a chance the sponsor used his 20/20 hindsight to come up with the strategy.
2. What are the drawdowns vs. the benchmark. Past performance is fairly meaningless for the future but past risk has some predicative ability for future risk. Take a look at the drawdowns and calculate the MAR ratio (average annual return/maximum drawdown). The strategy might have outperformed the benchmark but did it have a better MAR?
3. Do the past returns match the strategy? For example if it is a lower volatility strategy what were the backtested results in 2008? Should have been better than the benchmark. If it is a high return strategy it should have done better than the benchmark in rally years.
Obviously what a strategy will do in the future is much more important than what it did in the past. You can't predict this but you can determine if the premise makes sense. For example, there has been a lot written about factor tilts. Small cap stocks have shown outperformance vs. the S&P 500. If we assume that the market is up more often than it is down and when we are in a "risk on" type of environment then riskier stuff should outperform, then it makes sense that small caps will outperform large caps (albeit with more risk). Value is another factor that has shown outperformance. Going forward it makes sense that if you buy solid stocks when they are undervalued then you should perform better over time (albeit with some underperformance during speculative bubbles).
On the other hand, there has also been research showing that low volatility stocks outperform. This one doesn't make as much sense to me as lower volatility stocks shouldn't do as well during a bull market and I doubt that they can protect that much during a 2008 type of scenario.