Bootstrap is something I've been working with for over a decade. When I first started using it, I didn't quite get it. After a few attempts at building website themes from scratch on my own it started to make more sense.
Developing a complete, cohesive and consistent theme is a big job! If you just look at the number of components that Bootstrap offers you can get an appreciation for the volume of work involved in building all of your HTML and CSS from scratch.
This article outlines more of a core philosophy on how you and your team should use and customize Bootstrap. This isn't a detailed technical walkthrough (although one of those may come at a later time). Even without the technical details, this article should give you a solid foundation for building your site or application with the Bootstrap framework.
Take Your Time to Learn the Framework
This might be obvious advice but in order to approach customization the right way you need to be familiar with the framework. Try to use it out of the box first — get accustomed to the different components and classes, and rely more on them than any custom CSS.
Once you've done this for a while (at least a month of active practice is a good amount in my opinion) then you'll have a much more solid foundation as you approach the problem of customization.
The principles that I'm going to outline here will probably be appreciated more after you've gained some experience with Bootstrap, so come back once you've had some practice and see if you agree with me.
Write Custom CSS Only as a Last Resort
This is one principle that serves as a real foundation for my whole philosophy with Bootstrap. If you always fall back to writing your own custom CSS, you'll severely inhibit your ability to learn the framework deeply and thoroughly.
I've worked with people in the past who really seemed to enjoy writing their own custom CSS. The way I see it, you can access just about any CSS property with a Bootstrap class, and if you learn which ones to use in various situations, you'll be much better off in the long run.
Customize from the Inside Out
Inevitably, every site needs some custom CSS regardless of which platform or framework it uses. When you do need custom classes, then it's time to start using the Bootstrap source code.
Although this will require a bit more work upfront, it'll become less work in the long run and will produce an optimally sized bundle of final CSS for you to deploy on your site.
Don't Change Existing Functionality
This is another key principle of mine that is easy to violate — especially for those who might be recently starting out with the framework or those who have never customized Bootstrap by using the source code before.
Let's say you want to customize the .table class in Bootstrap. Let's say you're doing a project for ACME Corporation. I would create a custom class for the table, .acme.table, so that when you want the original version you just use the table class, and if you want your custom version then you add your company prefix.
By keeping the original framework intact you make it easier to fall back to the original components in cases where you may still need them. It's a nice, unobtrusive way to add custom functionality to Bootstrap.
If you truly want to customize every instance of a component, that should most often be accomplished by customizing variables in your project. With the latest versions of Bootstrap (v5 at the time of this writing, although v4 was very good in this regard too) they have variables for just about everything, down to a very granular level.
Use the Mixins Provided
Once you've grown familiar with adding your own classes and customizing some of the variables in the platform, you can begin to use the mixins that are provided.
To really use the mixins effectively I recommend just studying the source code of the existing components that the framework offers.
The vast majority of these components are taking advantage of the mixins that come along with the platform. Buttons, for example, are generated by running all of the theme colors through a set of mixins to produce the final code (while only having to write and maintain that core logic in one place).
When you master the mixins that come with Bootstrap you are really opening your opportunities to customize everything to your needs. Getting to this level will allow you to create highly customized, yet still highly cohesive and consistent, themes quickly and easily.
Wrapping Up
I hope that these principles will help guide you in your journey with Bootstrap and its customization. By sticking to the philosophy I have outlined here you'll be able to customize Bootstrap to your needs down to every last detail, and you'll be able to do so in a sustainable way that won't end up being a maintenance nighmare.
This philosophy has gotten me through several major version changes with Bootstrap, which is probably something I should've mentioned earlier. In any case, good luck with your projects and let me know how this framework works for you or what other thoughts you have on the process!