There is no optimization for AI crawlers for our documentation at the moment, sometime when asking AI assistant about certain features, it won't yield the correct results due to legacy docs reference. In some cases, asking for doc explicitly will point to older version documentation.

To mitigate this, we should at least add a robot.txt to cover:

1. Only allow "latest" documentation.
2. Disallow all legacy versions(1.x, 2.x) being indexed.