C++ cares so little about ABI that I'm actually amazed it doesn't reorder members in a struct to avoid padding, but apart from C++'s class is is so ill-designed that breaks library ABI's unnecessarily. Anything that appears in a library's header files should be a promise. All changes to a header file must be backwards compatible, and if breaking change is introduced, it should have been unavoidable and the library's major version number shall be increased. Breaking changes include both ABI (compilation and linking details) and API (usage details). But this is not how C++ is designed.
In C++, the only difference between struct and class is that members are private by default in class but public by default in struct. This in it self is bad because it hints to the developer that private members shall go at the top, whereas in reality they should go on the bottom. Although it is a bad idea to use struct for classes, this means the problems with class also applies to struct.
To implement virtual functions, C++ adds to the top of the class a function-pointer-pointer, called the virtual table, that is, a pointer to an array of function-pointers. This means that if you have a class without any virtual functions but later decide to add a virtual function, the offset of each member in the class class will be shifted to make room for the virtual table, and thereby breaking the ABI.
In a well-designed language ABI specification, the virtual table should be placed at the position the first virtual function is specified. Alternatively, the virtual table could be skipped altogether and simply insert the function pointers where the virtual functions are specified, just as member functions, which C developers are used to but which also exist in (and are still useful in) C++.
In C, when adding private members to a struct, sometimes it is just added and the regular way but with a comment that it is for internal use. But private exist for a reason: because it it may change in the future version so you don't want the user accessing them as their code could break at any time. So in C, when you really need to hide a member, you add a pointer to a struct which is not defined in the public header files. In C++, does is not how it's done, instead private members are stored precisely where specified, just like public members. This means that adding a new private member, change the size of the class, which breaks the ABI as both new and static and automatic storage depend on the size of the class, even new finds the size of the class in the application code (otherwise the library developer would need to specify in some translation unit that new should be supported). The change also affects subclasses.
In my opinion the difference between class and struct should not be the default visibility, but rather that class should always have a virtual table (provided that solution is used in the first place) and a pointer to the memory for the private members.
A problem with the solution using a pointer to private members is that it's added a level of indirection, and an additional memory allocation. Another problem, is that it disables static and automatic storage, allowing only heap storage. The later problem is easily solved by adding an extent const size_t that specifies the size of the private members, but the user would have to add it to a translation unit. The former problem however is not as easy to solve as long as subclasses are supported (well it could be solved partially, using the same trick is used for shared object files to change addresses at link time). But without subclesses it could be specified that the size of the class is only known by functions with private access.