The caravans I have encountered so far have all had 35-40 units. If you have any fewer, it may be difficult for you to defeat them. It helps to have less units than the caravans or you will be chasing them across the world.
If you have too many, you won’t be able to catch the caravans. You must have around 30-40 mid tier units.To be able to achieve the strategy I’m going to give you, you must fulfill these prerequisites. They have a great amount of trading goods that can be sold in mass for big stonks, they have higher tier troops that sell well as prisoners, they drop a large amount of horses that increase your speed and weight cap in their loot, and they also have a better chance of giving you higher tier gear. What I have found is that caravans in Bannerlord are also even more profitable than Warband.
Because of that, there are also an ungodly amount of caravans. In Bannerlord, there are many factions to join and conquer.