Launch concept from SSTO ideas

Would it be worth it to make the smallest possible spacecraft that can take a couple of astronauts to orbit?

The question comes from the fact that cargo gets added to the spacecraft, and that invariably makes the rocket big.  Considering that the payload fraction tends to be only 1%, then every pound adds 100 pounds to the wet mass on the launch pad.

If you keep everything as small as possible, a rocket's size can be minimized.  Secondly, if you can make use of what's available, like the atmosphere, you can increase payload fraction.  Take Skylon as an example.  The payload fraction there is 23 pounds for every pound to orbit.  Could that be improved upon?

What about an engine with airbreathing capability that gets dropped from a plane, and then proceeds to orbit?  It can be a dual capability type engine like the SABRE, which will go to conventional mode at sufficient velocity.

Dropping from the plane removes the need for landing gear.  The capsule will splash down, or land on the ground.  The rocket taking the capsule to space can be reusable, returning to base after achieving orbital velocity.  It would be preferable to return to base as a glider, like the shuttle.  It would do a once around the earth, then land.  The capsule would be released.  It would have to circularize its orbit on its own.

Such a configuration wouldn't be SSTO.  It could possible be fully reusable.  That's the goal.  It seems to me that to do that may be simplified if the payload mass is kept to a minimum.